Image processing method, model training method and related device

Abstract

An image processing method is applied to the technical field of artificial intelligence. According to the method, images in a continuously collected image sequence are divided into reference frames and non-reference frames, and for the non-reference frames in the image sequence, semantic features of the reference frames before the non-reference frames are multiplexed to predict the semantic features of the non-reference frames, and the semantic features of the non-reference frames are not re-extracted any more. And an image segmentation result of the non-reference frame is obtained based on the semantic feature prediction of the non-reference frame. According to the scheme, the similarity between the reference frame and the non-reference frame is effectively utilized, the semantic feature extraction process for the non-reference frame is reduced by multiplexing the semantic features of the reference frame, the calculation amount of the image segmentation process for the non-reference frame can be effectively reduced, and the application scene of the image processing method is expanded.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, a model training method and related devices. Background Art

[0002] Image segmentation is an important branch of computer vision and is widely used in all walks of life. Image segmentation refers to dividing an image into several required image blocks based on its intrinsic features (such as texture, brightness or topological structure, etc.). Each image block has similar features, and different image blocks have significantly different features. With the increasing popularity of deep learning technology, the methods used to achieve image segmentation have gradually changed from traditional methods to methods based on deep learning.

[0003] At present, in deep learning, the U-net structure is usually used to achieve image segmentation. The U-net structure actually divides the network into two parts: an expansion path with features from large to small and a contraction path with features from small to large. In the expansion path, the input image is convolved and downsampled layer by layer from left to right to obtain the global semantic information of each object; in the contraction path, the spatial details of the image are restored by performing upsampling and the desired image segmentation result is finally obtained by splicing the previous semantic information.

[0004] However, the current deep learning-based image processing methods have a large amount of computation. In some scenarios where real-time segmentation of continuously acquired images is required, such as video calls or video conferences, devices with limited computing resources often find it difficult to continuously perform image segmentation on each acquired image, which limits the application of image processing methods. Summary of the invention

[0005] The present application provides an image processing method that can effectively reduce the amount of computation in the image segmentation process and expand the application scenarios of the image processing method.

[0006] In a first aspect, the present application provides an image processing method for segmenting images in an image sequence. In the method, a first image and a second image in the image sequence are first acquired, the first image may be a reference frame in the image sequence, the second image may be a non-reference frame in the image sequence, and the second image is located after the first image in the image sequence, and the second image and the first image meet a preset similarity requirement. The image sequence includes a plurality of continuously acquired images, which can be essentially understood as a video.

[0007] Then, the first image is processed by a semantic feature extraction network to obtain a semantic feature of the first image. The semantic feature of the first image is essentially an image feature stored in a three-dimensional matrix form, which can be used to predict the image segmentation result of the first image.

[0008] Secondly, the semantic features of the first image and the second image are processed by the denoising network to predict the semantic features of the second image, and the semantic features of the second image are used to obtain the image segmentation result of the second image. Since the second image as the non-reference frame has a high similarity with the first image as the reference frame, the denoising network predicts the semantic features of the non-reference frame based on the semantic features of the reference frame and the image content of the non-reference frame itself by reusing the semantic features of the reference frame, thereby avoiding re-extraction of the semantic features of the non-reference frame.

[0009] Finally, semantic segmentation processing may be performed on the semantic features of the second image, that is, segmentation of each part in the image may be achieved based on the semantic information of each part contained in the semantic features of the second image, thereby obtaining an image segmentation result of the second image.

[0010] In this solution, when the first image and the second image in the image sequence are obtained, for the second image located after the first image in the image sequence, the semantic features of the first image are reused to predict the semantic features of the second image, and the semantic features of the second image are no longer extracted again, and then the image segmentation result of the second image is predicted based on the semantic features of the second image. Since the similarity between images in the image sequence is effectively utilized in this solution, the semantic features of the previous image are reused to reduce the semantic feature extraction process for the subsequent image, which can effectively reduce the amount of calculation for the image segmentation process for some images and expand the application scenarios of the image processing method.

[0011] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold. That is, it can be determined whether each image in the image sequence is a reference frame or a non-reference frame by the similarity between the images, thereby ensuring that the similarity between the reference frame and the non-reference frame is not less than the first threshold. In this way, while ensuring that the first image and the second image have a high similarity, the accuracy of reusing the semantic features of the first image to predict the semantic features of the second image can be guaranteed.

[0012] In general, by determining the reference frame and the non-reference frame based on the similarity between images, it is possible to ensure that the non-reference frame has a high similarity with the adjacent reference frame, thereby ensuring the accuracy of the semantic features of the non-reference frame predicted based on the semantic features of the reference frame.

[0013] In a possible implementation, in the image sequence, the number of images between the first image and the second image is less than the second threshold. That is, a fixed number of images may be used as an interval, and the first image of each interval may be used as a reference frame, that is, the number of non-reference frames between two adjacent reference frames is fixed. In this way, the number of images between the first image as the reference frame and the second image as the non-reference frame is less than the fixed threshold (i.e., the second threshold).

[0014] In this scheme, by limiting the number of images between the first image and the second image in the image sequence, the continuity of the first image and the second image can be guaranteed, thereby ensuring that the first image and the second image have a high degree of similarity, and ensuring the accuracy of reusing the semantic features of the first image to predict the semantic features of the second image.

[0015] In a possible implementation, processing the first image through a semantic feature extraction network may specifically include: performing a first number of feature extraction processes on the first image through the semantic feature extraction network.

[0016] Processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image may specifically include: performing a second number of feature extraction processing on the second image to obtain the original features of the second image. The second number is smaller than the first number, such as the first number is 5 and the second number is 4. For example, first performing a feature extraction on the second image through a convolutional network to obtain a feature map with a size of 1 / 2 of the second image, and then downsampling the feature map using a bilinear interpolation method to obtain the original features with the same size as the semantic features of the first image. Then, processing the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0017] In this solution, by reducing the number of feature extraction processes performed on the second image and then using a denoising network to fuse the features of the second image and the semantic features of the first image, the amount of computation required to extract the semantic features of the second image can be effectively reduced, thereby improving image processing efficiency.

[0018] In a possible implementation, the semantic features of the first image and the original features of the second image are processed through a denoising network to predict the semantic features of the second image, including: splicing the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and inputting the spliced ​​features into the denoising network to predict the semantic features of the second image.

[0019] In a possible implementation, after obtaining the semantic features of the first image, semantic segmentation processing can be performed on the semantic features of the first image to obtain an image segmentation result of the first image. The image segmentation result of the first image is predicted by a first semantic segmentation network, and the image segmentation result of the second image is predicted by a second semantic segmentation network. That is, for a reference frame, the first semantic segmentation network is used to process the semantic features of the reference frame to obtain the image segmentation result of the reference frame; for a non-reference frame, the second semantic segmentation network is used to process the semantic features of the non-reference frame to obtain the image segmentation result of the non-reference frame.

[0020] In this scheme, by using different semantic segmentation networks to process the semantic features of the reference frame and the semantic features of the non-reference frame respectively, the deviation of the predicted semantic features of the non-reference frame can be well compensated, ensuring that the image segmentation effect is finally achieved based on the predicted semantic features.

[0021] In a possible implementation, the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters. That is, during the training process, the training process for the reference frame and the non-reference frame is separate and independent, and by independently training the first semantic segmentation network for processing the reference frame and the second semantic segmentation network for processing the non-reference frame, it can be effectively ensured that the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

[0022] In this solution, by using the same network structure as the first semantic segmentation network and the second semantic segmentation network, and training the first semantic segmentation network and the second semantic segmentation network respectively, so that the first semantic segmentation network and the second semantic segmentation network have different weight parameters, the network structure can be reused to reduce the network design process while ensuring the accuracy of the two semantic segmentation networks in performing image segmentation tasks.

[0023] In one possible implementation, the denoising network may include a convolutional neural network or an attention network.

[0024] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0025] The second aspect of the present application provides a model training method, including: obtaining semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; inputting the second image into the semantic feature extraction network to obtain target semantic features; updating the denoising network based on a loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0026] In this scheme, a semantic feature extraction network for processing reference frames is pre-trained. For the denoising network used to predict the semantic features of non-reference frames, the semantic features extracted by the semantic feature extraction network and the semantic features predicted by the denoising network are used to construct a loss function, thereby realizing the training of the denoising network. Finally, a denoising network can be obtained which reduces the computational amount of semantic feature extraction for non-reference frames by reusing the semantic features of the reference frames.

[0027] In a possible implementation manner, the similarity between the first image and the second image is not less than a first threshold.

[0028] In a possible implementation manner, in the image sequence, the number of images spaced between the first image and the second image is less than a second threshold.

[0029] In a possible implementation, the method also includes: performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determining a difference value between the image segmentation result of the second image and a true segmentation result of the second image, wherein the true segmentation result of the second image is pre-annotated, for example, manually pre-annotated; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0030] In this scheme, the loss function for training the denoising network is constructed from two aspects. On the one hand, it is based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network. On the other hand, it is based on the image segmentation results and the true segmentation results finally output by the semantic segmentation network. This enables the denoising network to take into account the final image segmentation results during the training process, ensuring that accurate image segmentation results can be obtained after the model is trained.

[0031] In a possible implementation, the method further includes: updating the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0032] In a possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image through a semantic feature extraction network. The semantic features of the first image and the second image are processed through a denoising network to predict the semantic features of the second image, including: performing a second-order feature extraction process on the second image to obtain the original features of the second image, the second order being less than the first order; and processing the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0033] In a possible implementation, the semantic features of the first image and the original features of the second image are processed through a denoising network to predict the semantic features of the second image, including: splicing the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and inputting the spliced ​​features into the denoising network to predict the semantic features of the second image.

[0034] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0035] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0036] The third aspect of the present application provides an image processing device, including: an acquisition module, used to acquire a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image; a processing module, used to process the first image through a semantic feature extraction network to obtain the semantic features of the first image, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; the processing module is also used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; the processing module is also used to input the second image into the semantic feature extraction network to obtain the target semantic features; the processing module is also used to update the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0037] In a possible implementation manner, the similarity between the first image and the second image is not less than a first threshold.

[0038] In a possible implementation manner, in the image sequence, the number of images spaced between the first image and the second image is less than a second threshold.

[0039] In one possible implementation, the processing module is specifically used to: perform a first number of feature extraction processing on the first image through a semantic feature extraction network; perform a second number of feature extraction processing on the second image to obtain the original features of the second image, the second number being smaller than the first number; process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0040] In a possible implementation, the processing module is specifically used to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0041] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0042] In one possible implementation, the processing module is also used to: process the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; process the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; wherein the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

[0043] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0044] In a fourth aspect, the present application provides a model training device, comprising: an acquisition module, used to acquire semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; a processing module, used to process the semantic features of the first image and the second image through a denoising network, and predict the semantic features of the second image; the processing module is also used to input the second image into the semantic feature extraction network to obtain target semantic features; the processing module is also used to update the denoising network based on a loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0045] In a possible implementation manner, the similarity between the first image and the second image is not less than a first threshold.

[0046] In a possible implementation manner, in the image sequence, the number of images spaced between the first image and the second image is less than a second threshold.

[0047] In one possible implementation, the processing module is further used to: perform semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determine a difference value between the image segmentation result of the second image and a true segmentation result of the second image, wherein the true segmentation result of the second image is pre-annotated; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0048] In a possible implementation, the processing module is further used to: update the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0049] In a possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image through a semantic feature extraction network; the processing module is specifically used to: perform a second-order feature extraction process on the second image to obtain the original features of the second image, the second order is smaller than the first order; process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0050] In a possible implementation, the processing module is specifically used to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0051] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0052] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0053] In a fifth aspect, the present application provides an image processing device, comprising: a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the image processing device executes a method as in any one of the implementation modes in the first aspect.

[0054] The sixth aspect of the present application provides a model training device, which includes: a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes a method as implemented in any one of the second aspects.

[0055] The seventh aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes a method as implemented in any one of the first aspect or the second aspect.

[0056] An eighth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method as implemented in any one of the first or second aspects.

[0057] In a ninth aspect, the present application provides a chip, comprising one or more processors, wherein some or all of the processors are used to read and execute a computer program stored in a memory to execute the method in any one of the implementations of the first or second aspect.

[0058] Optionally, the chip includes a memory, and the memory is connected to the processor through a circuit or wire. Optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing results through the communication interface. The communication interface can be an input and output interface. The method provided in the present application can be implemented by one chip, or by multiple chips in collaboration.

[0059] Among them, the technical effects brought about by any design method in the second to ninth aspects can refer to the technical effects brought about by different implementation methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of using a U-shaped network structure to implement image segmentation provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of a system architecture 400 provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the structure of an electronic device 101 provided in an embodiment of the present application;

[0065] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of a process for performing image segmentation on an image acquired in real time provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of processing a reference frame and a non-reference frame through different networks provided in an embodiment of the present application;

[0068] Fig. 9 A flowchart of a model training method provided in an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a system architecture for executing an image processing method provided in an embodiment of the present application;

[0070] Fig.11 A schematic diagram of performing portrait segmentation processing on a reference frame and a non-reference frame provided in an embodiment of the present application;

[0071] Fig.12 A schematic diagram of performing portrait segmentation processing on a reference frame provided in an embodiment of the present application;

[0072] Fig.13 A schematic diagram of a process for processing a non-reference frame provided in an embodiment of the present application;

[0073] Fig.14 A schematic diagram for comparing a reference frame and a non-reference frame provided in an embodiment of the present application;

[0074] Fig.15 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0075] Fig.16 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0076] Fig.17 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0077] Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0078] Fig.19 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those of ordinary skill in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0080] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0081] To facilitate understanding, some technical terms involved in the embodiments of the present application are first introduced below.

[0082] (1) Neural Network

[0083] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0084]

[0085] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0086] (2) Deep Neural Networks

[0087] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neural unit in the i-th layer must be connected to any neural unit in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also very large. The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neural unit in the second layer to the 2nd neural unit in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neural unit in the L-1th layer to the jth neural unit in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a prediction model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0088] (3) Convolutional Neural Network (CNN)

[0089] A convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network includes a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or feature map. A convolutional layer refers to a neural unit layer in a convolutional neural network that performs convolution processing on the input signal (for example, the first convolutional layer and the second convolutional layer in this embodiment). In the convolutional layer of a convolutional neural network, a neural unit can be connected only to some neural units in adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangular shape. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. That means that the image information learned in a certain part can also be used in another part. So for all positions on the image, we can use the same learning to obtain image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0090] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0091] Specifically, Figure 1 As shown, the convolutional neural network 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0092] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in the present application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0093] Convolutional layer / pooling layer 120. Convolutional layer: Figure 2 The convolution layer / pooling layer 120 shown may include layers 121-126 as shown in the examples. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0094] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0095] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0096] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0097] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 1 The layers 121-126 illustrated in 120 may be a convolutional layer followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers.

[0098] Neural network layer 130: After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As before, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 1 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, text classification, speech classification, etc.

[0099] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function value similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 140 to 110 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0100] It should be noted that if Figure 1 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network prediction models, such as Figure 2 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0101] Generally, convolution processing can be represented by Conv.

[0102] (4) Residual Network (ResNet)

[0103] The residual network is an improved convolutional neural network that is easy to optimize and can improve accuracy by increasing the depth. The residual blocks inside the residual network use skip connections to alleviate the gradient vanishing problem caused by increasing the depth of the deep neural network.

[0104] (5) Encoder

[0105] In the embodiment of the present application, the encoder is essentially a neural network model that can convert data such as text or images into vectors in a coding space (i.e., convert text or images into text features or image features). In the encoder-decoder structure, the encoder actually performs feature extraction on the input data to obtain the features of the input data.

[0106] (6) Decoder

[0107] In the embodiment of the present application, the decoder is essentially a neural network model that can convert the features output by the encoder into output. For example, in an image segmentation task, the encoder performs feature extraction on the input image to obtain image features; the decoder processes the image features to obtain image segmentation results.

[0108] (7) Attention Network

[0109] Attention network is a network model that uses attention mechanism to improve the model training speed. At present, the typical attention network includes Transformer network. The model using attention mechanism can assign different weights to each part of the input sequence, so as to extract more important feature information from the input sequence, so that the model can finally obtain more accurate output.

[0110] In deep learning, the attention mechanism can be implemented by a weight vector that describes importance: when predicting or inferring an element, the weight vector is used to determine the correlation between the element and other elements. For example, for a pixel in an image or a word in a sentence, the attention vector can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors can be used as an approximation of the target value.

[0111] The attention mechanism in deep learning simulates the attention mechanism of the human brain. For example, when humans look at a painting, although the human eye can see the whole picture of the whole painting, when humans observe it in depth and carefully, the eyes actually focus on only a part of the pattern in the whole painting. At this time, the human brain mainly focuses on this small pattern. In other words, when humans observe an image carefully, the human brain does not pay equal attention to the whole image, but has a certain weight distinction. This is the core idea of ​​the attention mechanism.

[0112] Simply put, the human visual processing system tends to selectively focus on certain parts of the image and ignore other irrelevant information, which helps the human brain perceive. Similarly, in the attention mechanism of deep learning, in some problems involving language, speech or vision, some parts of the input may be more relevant than other parts. Therefore, through the attention mechanism in the attention model, the attention model can perform different processing on different parts of the input data, so that the attention model only dynamically focuses on the data related to the task.

[0113] (8) Loss function

[0114] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function (that is, the loss function value), the greater the difference, so the training of the neural network becomes a process of minimizing the loss function value as much as possible.

[0115] (9) Back propagation algorithm

[0116] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial prediction model during the training process, so that the error loss of the prediction model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial prediction model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the prediction model, such as the weight matrix.

[0117] (10) Gradient descent

[0118] Gradient descent is a first-order optimization algorithm that is often used in machine learning to recursively approximate the minimum deviation prediction model. To use gradient descent to find the local minimum of a function, it is necessary to iteratively search for points at a specified step distance in the opposite direction of the gradient (or approximate gradient) corresponding to the current point on the function. Gradient descent is one of the most commonly used methods for solving the prediction model parameters of machine learning algorithms, that is, unconstrained optimization problems.

[0119] Specifically, when solving the minimum value of the loss function, we can use the gradient descent method to iterate step by step to obtain the minimized loss function value and prediction model parameter value. Conversely, if we need to solve the maximum value of the loss function, we need to use the gradient ascent method to iterate.

[0120] (11) Reference frame

[0121] Usually, frames are used to describe video information, and the reference frame is the frame that best represents the video information. The so-called reference frame refers to the frame in which the object in the scene moves abnormally, and the other frames between the abnormal frames keep the scene normal, where the scene refers to a set of several shots with related content.

[0122] Simply put, when people observe the objective world, they are often most interested in abnormal events and gain a lot of information from abnormal changes. For an object, abnormality means that the object's state of motion has changed significantly, such as from stillness to motion, from motion to stillness, the direction of motion has changed, or the speed of motion has changed significantly. Similarly, when people watch video scenes, they focus on the changes in the scene, and the changes in the video scene are a reflection of the significant changes in the state of motion of the objects in the scene. Scene changes also include scene switching, which can be considered as the sudden movement of an object in the original scene to an infinitely far place, while the object in the new scene moves from an infinitely far place, and the object's state of motion changes dramatically.

[0123] (12) Normalized exponential function (Softmax function)

[0124] The Softmax function is a generalization of the logic function. The Softmax function can compress a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that the range of each element is between (0,1) and the sum of all elements is 1.

[0125] (13) Batch Normalization (BN)

[0126] Normalization is a way to simplify calculations, that is, to transform a dimensioned expression into a dimensionless expression, a scalar. In simple terms, normalization is a dimensionless processing method that converts the absolute value of a physical system into a relative value relationship.

[0127] (14) Linear rectification function (ReLU)

[0128] The linear rectification function, also known as the rectified linear unit, is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0129] (15) Squeeze-and-Excitation (SE) module

[0130] Generally, the SE module consists of two parts, namely the Squeeze part and the Excitation part. For the Squeeze part, the dimension of the original feature map is H*W*C, and the work of the Squeeze part is to compress H*W*C into 1*1*C, which is equivalent to compressing H and W into one dimension, so that this one-dimensional parameter obtains the global vision of the previous H*W and has a wider perception area.

[0131] For the activation part, after obtaining the feature of the size of 1*1*C output by the compression part, a fully connected layer is added to predict the importance of each channel. After obtaining the importance of different channels, it is applied (stimulated) to the corresponding channels of the previous original feature map, and then subsequent operations are performed.

[0132] In general, the SE module essentially calculates the similarity in the channel dimension and enhances the information interaction between multiple channels.

[0133] In deep learning, a U-net is usually used to implement image segmentation. Figure 3 , Figure 3 A schematic diagram of using a U-shaped network structure to implement image segmentation is provided in an embodiment of the present application. Figure 3 As shown in the figure, the U-shaped network structure actually divides the network into two parts: an expansion path with features from large to small and a contraction path with features from small to large. In the expansion path, the input image is convolved and downsampled layer by layer from left to right to obtain the global semantic information of each object; in the contraction path, the spatial details of the image are restored by performing upsampling and the desired image segmentation result is finally obtained by splicing the previous semantic information.

[0134] Since the image features obtained by downsampling are spliced ​​each time during the process of predicting the image segmentation results through upsampling, the U-shaped network structure framework has a huge amount of computation. Therefore, in some scenarios where real-time segmentation of continuously acquired images is required, such as video calls or video conferences, devices with limited computing resources often find it difficult to continuously perform image segmentation on each acquired image, resulting in limited application of image processing methods.

[0135] In view of this, an embodiment of the present application provides an image processing method, which divides the images in a continuously acquired image sequence into reference frames and non-reference frames. For the non-reference frames in the image sequence, the semantic features of the reference frame located before the non-reference frame are reused to predict the semantic features of the non-reference frame, and the semantic features of the non-reference frame are no longer extracted again, and then the image segmentation result of the non-reference frame is predicted based on the semantic features of the non-reference frame. Since the similarity between the reference frame and the non-reference frame is effectively utilized in this scheme, the semantic features of the reference frame are reused to reduce the semantic feature extraction process for the non-reference frame, which can effectively reduce the computational amount of the image segmentation process for the non-reference frame and expand the application scenarios of the image processing method.

[0136] For ease of understanding, the following will introduce the application scenarios and system architecture of the image processing method provided in the embodiments of the present application.

[0137] In a possible implementation, the image processing method provided by the embodiment of the present application can be applied in video calls, video conferences, or video live broadcast scenarios. In video calls, video conferences, or video live broadcast scenarios, users usually collect their own videos through terminal devices such as smart phones, tablet computers, laptops, or personal computers, and send them to other users. In addition, in these scenarios, users may need the terminal device to distinguish their portraits from the background in order to perform specific processing on the portraits or backgrounds, such as beautifying the portraits or replacing the backgrounds. In this way, after the terminal device collects the video, it can perform image segmentation on the images in the video based on the image processing method provided by the embodiment of the present application to determine the portraits and backgrounds in the image, and then perform specific processing on the portraits or backgrounds.

[0138] In another possible implementation, the image processing method provided in the embodiment of the present application can be applied in an autonomous driving scenario. In an autonomous driving scenario, the autonomous driving vehicle will collect images on the road in real time while driving, and identify objects in the collected images to determine the driving strategy of the autonomous driving vehicle. In this case, the autonomous driving vehicle can use the image processing method provided in the embodiment of the present application to perform image segmentation on the images collected in real time, thereby obtaining the image segmentation results corresponding to each image, so that the autonomous driving vehicle can decide the next driving strategy based on the image segmentation results.

[0139] In another possible implementation, the image processing method provided in the embodiment of the present application can be applied in the field of security or traffic safety. In the field of security, security cameras often collect images in a specific area in real time, and identify objects or pedestrians in the collected images to ensure whether the specific area is safe. Similarly, in the field of traffic safety, traffic cameras often collect images on traffic roads in real time, and identify vehicles or pedestrians in the collected images to determine whether the vehicles or pedestrians comply with traffic rules. Based on this, both security cameras and traffic cameras can use the image processing method provided in the embodiment of the present application to perform image segmentation on the images collected in real time, thereby obtaining image segmentation results corresponding to each image, so as to determine whether the security area is safe or whether the vehicles or pedestrians comply with traffic rules.

[0140] In another possible implementation, the image processing method provided by the embodiment of the present application can also be applied in the field of smart home. In the field of smart home, some smart home appliances (such as smart TVs, sweeping robots, cameras or smart speakers) often collect images in the user's home in real time, and need to perform image segmentation on the collected images to continue to perform the next task based on the image segmentation results. For example, the sweeping robot can determine the cleanable area and the area to avoid obstacles based on the image segmentation results; the smart TV can determine to start or shut down based on the image segmentation results; the smart speaker can adjust the volume in all directions based on the image segmentation results, etc. Therefore, smart home appliances in the field of smart home can also use the image processing method provided by the embodiment of the present application to implement image segmentation on the collected images, so as to perform the next task based on the obtained image segmentation results.

[0141] In general, the image processing method provided in the embodiments of the present application can be applied to scenarios where image segmentation needs to be performed on continuously acquired images, and is not limited to the scenarios exemplified above.

[0142] See also Figure 4 , Figure 4A schematic diagram of a system architecture 400 provided in an embodiment of the present application. Figure 4 As shown, in the system architecture 400, the execution device 410 can be implemented by one or more servers, and optionally, cooperate with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 410 can be arranged on a physical site, or distributed on multiple physical sites. The execution device 410 can use the data in the data storage system 420, or call the program code in the data storage system 420 to implement the training method of the classification model provided in the embodiment of the present application, and then obtain the model.

[0143] Users can operate their respective user devices (e.g., local device 401 and local device 402) to interact with execution device 410. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0144] Each user's local device can interact with the execution device 410 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0145] In one implementation, the execution device 410 is used to implement the training method of the classification model provided in the embodiment of the present application, and send the obtained model to the local device 401 and the local device 402 through the communication network, so that the local device 401 and the local device 402 can deploy and run the model, thereby realizing the image processing method provided in the embodiment of the present application.

[0146] In another implementation, one or more aspects of the execution device 410 can be implemented by each local device. For example, the local device 401 can provide local data or feedback calculation results to the execution device 410, or execute the image processing method and model training method provided in the embodiments of the present application.

[0147] It should be noted that all functions of the execution device 410 may also be implemented by the local device. For example, the local device 401 implements the functions of the execution device 410 and provides services to its own user, or provides services to the user of the local device 402.

[0148] In general, the image processing method and model training method provided in the embodiments of the present application can be applied to electronic devices, such as the above-mentioned execution device 410, local device 401 or local device 402. Exemplarily, the electronic device can be, for example, a server, a wireless electronic device in industrial control, a smart phone (mobile phone), a personal computer (personal computer, PC), a laptop computer, a tablet computer and other devices. Exemplarily, in order to take the image processing method applied to terminal devices such as laptops or smart phones and the model training method applied to servers as an example, the method provided in the embodiments of the present application is described in detail.

[0149] See also Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device 101 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 101 includes a processor 103, which is coupled to a system bus 105. The processor 103 may be one or more processors, each of which may include one or more processor cores. A display adapter (video adapter) 107, which may drive a display 109, which is coupled to the system bus 105. The system bus 105 is coupled to an input / output (I / O) bus via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with a variety of I / O devices, such as an input device 117 (such as a touch screen, etc.), an external memory 121 (for example, a hard disk, a floppy disk, an optical disk or a USB flash drive), a multimedia interface, etc.). A transceiver 123 (which may send and / or receive radio communication signals), a camera 155 (which may capture static and dynamic digital video images), and an external USB port 125. Optionally, the interface connected to the I / O interface 115 may be a USB interface.

[0150] The processor 103 may be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an ASIC.

[0151] The electronic device 101 can communicate with the software deployment server 149 through the network interface 129. Exemplarily, the network interface 129 is a hardware network interface, such as a network card. The network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, the network 127 can also be a wireless network, such as a WiFi network, a cellular network, etc.

[0152] The hard disk drive interface 131 is coupled to the system bus 105. The hard disk drive interface is connected to the hard disk drive 133. The internal memory 135 is coupled to the system bus 105. The data running in the internal memory 135 may include an operating system (OS) 137 of the electronic device 101, an application 143, and a scheduler.

[0153] The operating system includes a shell 139 and a kernel 141. Shell 139 is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input to the operating system, and processing various operating system output results.

[0154] The kernel 141 consists of those parts of the operating system that manage memory, files, peripherals, and system resources. The kernel 141 interacts directly with the hardware. The operating system kernel usually runs processes and provides communication between processes, provides CPU time slice management, interrupts, memory management, IO management, etc.

[0155] See also Figure 6 , Figure 6 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. Figure 6 As shown, the image processing method includes the following steps 601-604.

[0156] Step 601: Acquire a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is located after the first image.

[0157] Specifically, the image processing method provided in this embodiment can be applied to perform image segmentation on images in a video. A video is essentially an image sequence, which is composed of a plurality of continuously acquired images. Moreover, for each image in the image sequence, the image can be divided into a reference frame or a non-reference frame. Among them, a reference frame may refer to an image in which the motion state of a person or object in the image sequence changes significantly. A non-reference frame refers to other images in the image sequence except the reference frame. Generally speaking, in an image sequence, one or more non-reference frames are usually included between two adjacent reference frames, so the difference between two adjacent reference frames is relatively large, while the difference between a non-reference frame and an adjacent reference frame is relatively small.

[0158] In this embodiment, the first image is a reference frame, and the second image is a non-reference frame. In addition, the reference frame and the non-reference frame meet a preset similarity requirement, that is, the first image and the second image meet a preset similarity requirement. The following will introduce how to determine the reference frame and the non-reference frame to ensure that the reference frame and the non-reference frame meet the preset similarity requirement.

[0159] In a possible implementation, whether each image in an image sequence is a reference frame may be determined by the similarity between images. Specifically, for an image sequence, the first image in the image sequence may be regarded as a reference frame, and for any image after the first image in the image sequence, the similarity between the image and the nearest reference frame located before the image may be determined. If the similarity between the image and the nearest reference frame is greater than or equal to a first threshold, the image may be determined as a non-reference frame; if the similarity between the image and the nearest reference frame is less than the first threshold, the image may be determined as a new reference frame. The similarity between the two images may be measured by cosine similarity or structural similarity index measure (SSIM), which is not specifically limited in this embodiment. The first threshold may be, for example, a value such as 90, 92, or 95%, which may be adjusted according to the actual scenario, which is not specifically limited in this embodiment.

[0160] That is to say, for the first image and the second image, since the first image is a reference frame and the second image is a non-reference frame located after the first image, the similarity between the first image and the second image is not less than the first threshold value. The similarity between the first image and the previous reference frame of the first image in the image sequence (i.e., the reference frame adjacent to the first image) is less than the first threshold value.

[0161] In another possible implementation, a fixed number of images may be used as an interval, and the first image of each interval may be used as a reference frame, that is, the number of non-reference frames between two adjacent reference frames is fixed. For example, assuming that 5 consecutive images are used as an interval, and the first image of each interval is used as a reference frame, then there are 4 non-reference frames between two adjacent reference frames. For example, assuming that the image sequence includes 20 images, the 1st to 5th images are the first interval, the 6th to 10th images are the second interval, the 11th to 15th images are the third interval, and the 16th to 20th images are the fourth interval; and the 1st image, the 6th image, the 11th image, and the 16th image are reference frames, and the 2nd to 5th images, the 7th to 10th images, the 12th to 15th images, and the 17th to 20th images are non-reference frames.

[0162] That is, in the image sequence, the number of images between the first image and the second image is less than the second threshold. In other words, a fixed number of images may be used as an interval, and the first image of each interval may be used as a reference frame, that is, the number of non-reference frames between two adjacent reference frames is fixed. In this way, the number of images between the first image as the reference frame and the second image as the non-reference frame is less than the fixed threshold (i.e., the second threshold).

[0163] In general, in this embodiment, the reference frame is a representative image in the image sequence, including rich information, and the difference between two adjacent reference frames is relatively large, while the difference between a non-reference frame and an adjacent reference frame is relatively small.

[0164] Step 602: Process the first image through a semantic feature extraction network to obtain semantic features of the first image. The semantic features of the first image are used to predict an image segmentation result of the first image.

[0165] In this embodiment, for the first image as the reference frame, the first image can be input into the semantic feature extraction network to perform image feature extraction to obtain the semantic features of the first image. The semantic features of the first image are the semantic features of the first image, which are essentially image features stored in the form of a three-dimensional matrix and can be used to predict the image segmentation result of the first image.

[0166] Specifically, the semantic feature extraction network can be a neural network for performing semantic estimation, which can extract image features based on the semantic information of each object in the image, so that the extracted semantic features can roughly indicate the position information of each object in the image, so as to facilitate the subsequent image segmentation. Among them, the semantic feature extraction network can be, for example, a convolutional neural network or a Transformer network, and this embodiment does not specifically limit the structure of the semantic feature extraction network.

[0167] Step 603: Process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image.

[0168] Since the difference between the non-reference frame and the adjacent reference frame is small, the present embodiment can reuse the semantic features of the reference frame, and predict the semantic features of the non-reference frame based on the semantic features of the reference frame and the image content of the non-reference frame itself, thereby avoiding re-extraction of the semantic features of the non-reference frame and reducing the large amount of computation caused by executing the extraction of semantic features.

[0169] Specifically, the denoising network is a neural network, such as a convolutional neural network, whose input may include the semantic features of the reference frame and the non-reference frame itself, and can eliminate the difference between the semantic features of the reference frame and the semantic features of the non-reference frame based on the non-reference frame itself, and then predict the semantic features of the non-reference frame. Therefore, in this embodiment, by using the denoising network to process the semantic features of the first image and the second image input to the denoising network, the semantic features of the second image (i.e., the semantic features of the second image) can be predicted by the denoising network. The semantic features of the second image are essentially also a kind of image features stored in the form of a three-dimensional matrix, which can be used to predict the image segmentation result of the second image.

[0170] Optionally, processing the first image through a semantic feature extraction network may specifically include: performing a first number of feature extraction processes on the first image through the semantic feature extraction network.

[0171] In addition, the semantic features of the first image and the second image are processed by the denoising network to predict the semantic features of the second image, which may specifically include: performing a second number of feature extraction processing on the second image to obtain the original features of the second image. The second number is smaller than the first number, such as the first number is 5 and the second number is 4. For example, a feature extraction is first performed on the second image through a convolutional network to obtain a feature map with a size of 1 / 2 of the second image, and then the feature map is downsampled using a bilinear interpolation method to obtain the original features with the same size as the semantic features of the first image. Then, the semantic features of the first image and the original features of the second image are spliced ​​to obtain spliced ​​features; and then the spliced ​​features are input into the denoising network to predict the semantic features of the second image.

[0172] In this way, by reducing the number of feature extraction processes performed on the second image and then using the denoising network to fuse the features of the second image and the semantic features of the first image, the amount of computation required to extract the semantic features of the second image can be effectively reduced, thereby improving image processing efficiency.

[0173] Step 604: perform semantic segmentation processing on the semantic features of the second image to obtain an image segmentation result of the second image.

[0174] After obtaining the semantic features of the second image corresponding to the first image, semantic segmentation processing can be performed on the semantic features of the second image, that is, the segmentation of each part in the image is achieved based on the semantic information of each part contained in the semantic features of the second image, thereby obtaining the image segmentation result of the second image.

[0175] In this solution, the images in the continuously acquired image sequence are divided into reference frames and non-reference frames. For the non-reference frames in the image sequence, the semantic features of the reference frame located before the non-reference frame are reused to predict the semantic features of the non-reference frame, and the semantic features of the non-reference frame are no longer extracted again, and then the image segmentation result of the non-reference frame is predicted based on the semantic features of the non-reference frame. Since the similarity between the reference frame and the non-reference frame is effectively utilized in this solution, the semantic features of the reference frame are reused to reduce the semantic feature extraction process for the non-reference frame, which can effectively reduce the computational amount of the image segmentation process for the non-reference frame and expand the application scenarios of the image processing method.

[0176] Optionally, the above steps 601-604 introduce the process of performing image segmentation on the second image as a non-reference frame. In practical applications, it is often necessary to perform image segmentation on each image in the entire image sequence. Therefore, before performing image segmentation on the second image, it is actually necessary to perform image segmentation on the first image.

[0177] Exemplarily, before the above-mentioned step 603, after obtaining the semantic features of the first image, semantic segmentation processing can be performed on the semantic features of the first image to obtain an image segmentation result of the first image. That is, in the process of performing image segmentation on each image in the image sequence, semantic features of the first image that is at the front of the image sequence and serves as a reference frame can be first extracted, and image segmentation can be achieved based on the semantic features of the extracted first image. Then, by reusing the semantic features of the first image corresponding to the first image, the semantic features of the second image that is at the back of the image sequence and serves as a non-reference frame are predicted, thereby achieving image segmentation of the second image. That is, the semantic features of the front reference frame in the image sequence are used to predict the semantic features of the back non-reference frame, thereby simplifying the image segmentation process of the non-reference frame.

[0178] The image segmentation result of the first image may be predicted by the first semantic segmentation network, and the image segmentation result of the second image may be predicted by the second semantic segmentation network. That is, for the reference frame, the first semantic segmentation network is used to process the semantic features of the reference frame to obtain the image segmentation result of the reference frame; for the non-reference frame, the second semantic segmentation network is used to process the semantic features of the non-reference frame to obtain the image segmentation result of the non-reference frame.

[0179] Since the first image is a reference frame, the semantic features of the first image corresponding to the first image are extracted through a semantic feature extraction network, and the semantic features of the second image corresponding to the second image as a non-reference frame are predicted based on the semantic features of the first image. Therefore, the acquisition methods of the semantic features corresponding to the two images are different, and the semantic features of the second image predicted for the second image may deviate from the actual semantic features of the second image. Therefore, in this embodiment, different semantic segmentation networks can be used to process the semantic features of the reference frame and the semantic features of the non-reference frame respectively, so as to make up for the deviation of the predicted semantic features of the non-reference frame, and ensure that the image segmentation effect is finally achieved based on the predicted semantic features.

[0180] Optionally, the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters. For example, the first semantic segmentation network and the second semantic segmentation network are both neural network structures. Since the weight parameters of the first semantic segmentation network and the second semantic segmentation network are determined during the training process, the first semantic segmentation network and the second semantic segmentation network having different weight parameters actually represent that the first semantic segmentation network and the second semantic segmentation network are trained separately during the training process. That is, during the training process, the training process for the reference frame and the non-reference frame is separate and independent. By independently training the first semantic segmentation network for processing the reference frame and the second semantic segmentation network for processing the non-reference frame, the performance of the first semantic segmentation network and the second semantic segmentation network can be effectively guaranteed.

[0181] In this solution, by using the same network structure as the first semantic segmentation network and the second semantic segmentation network, and training the first semantic segmentation network and the second semantic segmentation network respectively, so that the first semantic segmentation network and the second semantic segmentation network have different weight parameters, the network structure can be reused to reduce the network design process while ensuring the accuracy of the two semantic segmentation networks in performing image segmentation tasks.

[0182] In general, in this embodiment, for the reference frame and the non-reference frame in the image sequence, two different branches can be used to process the reference frame and the non-reference frame respectively. In addition, in the process of processing the non-reference frame, the semantic features of the reference frame are reused, thereby reducing the amount of calculation for processing the non-reference frame.

[0183] For example, see Figure 7 and Figure 8 , Figure 7 A schematic diagram of a process for performing image segmentation on an image acquired in real time provided in an embodiment of the present application; Figure 8 A schematic diagram of processing a reference frame and a non-reference frame through different networks provided in an embodiment of the present application. Figure 7 As shown, the process of performing image segmentation on an image acquired in real time includes the following steps 701-706.

[0184] Step 701, obtaining images collected in real time.

[0185] Since the image segmentation process is performed on the images in the video in this embodiment, and the video is composed of multiple images collected continuously in real time, during the execution of the method in this embodiment, the images collected in real time may be continuously acquired, and the image segmentation process may be performed on the acquired images.

[0186] Step 702, determining whether the acquired image is a reference frame.

[0187] For an image acquired in real time, it may be determined whether the acquired image is a reference frame to determine a subsequent image segmentation processing method for the image.

[0188] In a possible implementation, whether the image is a reference frame can be determined by the similarity between the image and the previous reference frame. Specifically, a reference frame closest to the image is first obtained, and then the similarity between the image and the reference frame is calculated. If the similarity between the image and the most recent reference frame is greater than or equal to a first threshold, the image can be determined as a non-reference frame; if the similarity between the image and the most recent reference frame is less than the first threshold, the image can be determined as a new reference frame.

[0189] In another possible implementation, whether the image is a reference frame may be determined by the number of images between the image and the previous reference frame. Specifically, a reference frame closest to the image is first obtained, and if the number of images between the image and the reference frame is less than a specific threshold (e.g., 4), the image is determined to be a non-reference frame; if the number of images between the image and the reference frame is not less than a specific threshold, the image is determined to be a reference frame.

[0190] Step 703: If the acquired image is a reference frame, semantic feature extraction is performed on the image through a semantic feature extraction network to obtain semantic features of the image.

[0191] like Figure 8 As shown, for the reference frame, in this embodiment, the image is input into a semantic feature extraction network, and the semantic feature extraction network is used to extract the semantic features of the image.

[0192] Step 704: Perform semantic segmentation processing on the semantic features of the image through the first semantic segmentation network to obtain an image segmentation result.

[0193] For the reference frame, after the semantic features of the reference frame are extracted by the semantic feature extraction network, the semantic features of the reference frame are further processed by the first semantic segmentation network to obtain the image segmentation result of the reference frame.

[0194] Step 705: If the acquired image is a non-reference frame, the semantic features of the image are predicted through a denoising network based on the semantic features of the reference frame closest to the image in the image sequence and the image.

[0195] If the acquired image is a non-reference frame, the semantic features of the image are no longer extracted through the semantic feature extraction network, but the denoising network is used to predict the semantic features of the image. The input of the denoising network includes the semantic features of the non-reference frame (i.e., the image itself) and the previous reference frame closest to the image (i.e., the features extracted by the semantic feature extraction network), and the output of the denoising network is the predicted semantic features of the non-reference frame.

[0196] Step 706: Perform semantic segmentation processing on the semantic features of the image through the second semantic segmentation network to obtain an image segmentation result.

[0197] For an image that is a non-reference frame, after the semantic features of the non-reference frame are predicted by the denoising network, a semantic segmentation process may be performed on the semantic features of the non-reference frame by the second semantic segmentation network to obtain an image segmentation result of the non-reference frame.

[0198] Optionally, in this embodiment, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait. That is, the image processing method provided in this embodiment can be applied to scenes such as video calls, video conferences, or live video broadcasts, and is used to segment the portrait and background of each image in the video, and then replace the background based on the segmented portrait and background.

[0199] In this scheme, in scenarios such as video calls, video conferences, or live video broadcasts, the content of different images in the video usually does not change much, and the main change is the posture of the person, that is, the similarity between the reference frame and the non-reference frame is large. Therefore, an image processing method based on feature reuse is used to achieve portrait segmentation in the video, which can effectively reduce the amount of calculation while ensuring the effect of portrait segmentation.

[0200] The above introduces the image processing method provided by the embodiment of the present application. The following will introduce the model training method provided by the embodiment of the present application to facilitate understanding of the training method of the network mentioned in the above image processing method.

[0201] It is understandable that, since different networks are used to process the reference frame and the non-reference frame in the above-mentioned image processing method, during the model training process, different networks for processing the reference frame and the non-reference frame may be trained independently to ensure that the trained network is suitable for executing the image processing method.

[0202] See also Fig. 9 , Fig. 9 A flow chart of a model training method provided in an embodiment of the present application. Fig. 9 As shown, the model training method includes the following steps 901-905.

[0203] Step 901: Acquire a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is located after the first image.

[0204] In this embodiment, the first image and the second image are training data in a training set, which are used to train a denoising network. Moreover, the first image and the second image are also images in an image sequence, so the first image and the second image have a high similarity.

[0205] Step 902, processing the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict an image segmentation result of the first image, and the semantic feature extraction network is a trained network.

[0206] In this embodiment, since the network for processing the reference frame and the network for processing the non-reference frame are trained independently, and the network for processing the reference frame is trained first and then the network for processing the non-reference frame is trained, when the denoising network for processing the non-reference frame is trained, the semantic feature extraction network for processing the reference frame is a trained network.

[0207] In addition, in some embodiments, when the first image has been processed by a semantic feature extraction network, the semantic features of the first image and the second image may be directly obtained without performing semantic feature extraction processing on the first image.

[0208] Step 903: Process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image.

[0209] After the semantic features of the first image are extracted using the trained semantic feature extraction network, the semantic features of the first image and the second image can be further processed using the denoising network to be trained to obtain the semantic features of the second image output by the denoising network, wherein the semantic features of the second image are the semantic features of the second image predicted by the denoising network.

[0210] Step 904: input the second image into a semantic feature extraction network to obtain target semantic features.

[0211] Since the denoising network is used to predict the semantic features of the second image in this embodiment, in order to guide the denoising network to predict accurate semantic features, the second image is input into the semantic feature extraction network in this embodiment to obtain the target semantic features. The target semantic features are the semantic features obtained through the complete semantic feature extraction process, that is, the actual semantic features of the second image.

[0212] In addition, the structures of the denoising network and the semantic feature extraction network in this embodiment are similar to the structures of the denoising network and the semantic feature extraction network mentioned in the above-mentioned image processing method. Please refer to the contents introduced in the above-mentioned image processing method for details, and no further details will be given here.

[0213] Step 905: Based on the loss function value, the denoising network is updated to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

[0214] After obtaining the semantic features of the second image predicted by the denoising network for the second image and the target semantic features extracted by the semantic feature extraction network for the second image, a loss function can be constructed based on the semantic features of the second image and the target semantic features, and then the loss function value is obtained and the denoising network is updated based on the loss function value. The loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features, and can represent the difference between the semantic features of the second image and the target semantic features.

[0215] In the process of training the denoising network, the training goal is to reduce the loss function value as much as possible so that the semantic features predicted by the denoising network are as close as possible to the semantic features extracted by the semantic feature network. In the process of training the denoising network based on the loss function value, the back propagation algorithm and the gradient descent method can be used to gradually update the weight parameters in the denoising network, thereby realizing the training of the denoising network.

[0216] It can be understood that the above steps 901-905 introduce a round of iterative update process in the process of training the denoising network. In practical applications, the denoising network can be subjected to multiple rounds of iterative updates, and the termination condition of the iterative update can be that the iterative update discussion of the denoising network reaches a preset number of rounds or the loss function value is not greater than a preset threshold.

[0217] In this scheme, a semantic feature extraction network for processing reference frames is pre-trained. For the denoising network used to predict the semantic features of non-reference frames, the semantic features extracted by the semantic feature extraction network and the semantic features predicted by the denoising network are used to construct a loss function, thereby realizing the training of the denoising network. Finally, a denoising network can be obtained which reduces the computational amount of semantic feature extraction for non-reference frames by reusing the semantic features of the reference frames.

[0218] Optionally, since the above-mentioned denoising network is only used to predict the semantic features of non-reference frames, the predicted semantic features are ultimately used to predict the image segmentation results. Therefore, the training goal is not only to make the semantic features predicted by the denoising network as close as possible to the semantic features extracted by the semantic feature extraction network, but also to make the final image segmentation results as close as possible to the real segmentation results. Based on this, the loss function can be constructed from two aspects, and then the update of the denoising network can be constrained from two aspects.

[0219] Specifically, after obtaining the semantic features of the second image predicted by the denoising network, a semantic segmentation process is performed on the semantic features of the second image through the semantic segmentation network to obtain an image segmentation result of the second image. Then, a difference value between the image segmentation result of the second image and a true segmentation result of the second image is determined. The true segmentation result of the second image may be a segmentation result obtained by manual labeling in advance, representing the segmentation result of the second image output by the desired network.

[0220] In the process of training the denoising network, the loss function value used to train the denoising network is obtained based on the difference between the image segmentation result of the second image and the true segmentation result of the second image and the distance between the semantic feature of the second image and the target semantic feature. In other words, the loss function is composed of two parts, one part is obtained based on the image segmentation result output by the semantic segmentation network and the true segmentation result, and the other part is obtained based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network.

[0221] In this scheme, the loss function for training the denoising network is constructed from two aspects. On the one hand, it is based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network. On the other hand, it is based on the image segmentation results and the true segmentation results finally output by the semantic segmentation network. This enables the denoising network to take into account the final image segmentation results during the training process, ensuring that accurate image segmentation results can be obtained after the model is trained.

[0222] Optionally, after obtaining the loss function value based on the above method, the semantic segmentation network used to predict the image segmentation result can also be updated based on the loss function value to obtain an updated semantic segmentation network.

[0223] That is to say, the denoising network and semantic segmentation network used to process non-reference frames are trained together based on the same loss function, thereby ensuring that the denoising network and semantic segmentation network finally trained can work together to output accurate image segmentation results.

[0224] The above introduces the process of training the denoising network and the semantic segmentation network for processing non-reference frames. Before training the denoising network and the semantic segmentation network, the semantic feature network and the semantic segmentation network for processing reference frames are also trained. The semantic feature network and the semantic segmentation network for processing reference frames can be trained using the traditional training method (i.e., constructing the loss function based on the image segmentation result finally output by the network and the real segmentation result), which will not be repeated here.

[0225] To facilitate understanding, the image processing method and model training method provided in this embodiment will be described in detail below with reference to specific examples.

[0226] Taking the application of the image processing method in scenarios such as screen recording, video calls, video conferencing or live video streaming as an example, the terminal device used by the user (such as a smart phone, laptop computer or tablet computer) uses the image processing method provided in this embodiment to perform portrait segmentation on the images in the image sequence in the process of continuously acquiring an image sequence, thereby obtaining a portrait segmentation result, so as to further beautify the portrait or replace the background based on the portrait segmentation result.

[0227] For example, taking the video conferencing software on a laptop as an example, please refer to Fig.10 , Fig.10 A schematic diagram of a system architecture for executing an image processing method provided in an embodiment of the present application. Fig.10 As shown in the figure, the conference software on the laptop obtains the original image through the image acquisition module. When the original image is in YUV image format, the software converts the original image into RGB image format and performs a certain degree of scaling to meet the algorithm input requirements. In the RGB image format, each pixel of the image is composed of the three primary colors of red, green, and blue, and the overall image is a matrix of length*width*3.

[0228] Then, the conference software sends the processed RGB image to the portrait segmentation module, which uses image processing methods to process the RGB image to obtain a mask. The mask is a two-dimensional matrix, where the value of the matrix element is 1, indicating that the pixel corresponding to the position belongs to the portrait, and the value of the matrix element is 0, indicating that the pixel corresponding to the position belongs to the background. After obtaining the mask, the portrait segmentation module passes the mask to the conference software for portrait extraction and background replacement. Finally, the conference software uploads the processed image to the user interface for display.

[0229] When the portrait segmentation module processes the RGB image, the portrait segmentation module first distinguishes whether the RGB image is a reference frame or a non-reference frame. For the reference frame, the portrait segmentation module uses a semantic feature extraction network to perform complete semantic feature extraction, and further performs portrait detail prediction based on the first semantic segmentation network, and finally obtains the portrait segmentation result (i.e., the above-mentioned mask). For the non-reference frame, the portrait segmentation module reuses the semantic features of the reference frame, predicts the semantic features of the non-reference frame through the denoising network, and further performs portrait detail prediction based on the second semantic segmentation network, and finally obtains the portrait segmentation result.

[0230] For example, see Fig.11 , Fig.11 This is a schematic diagram of performing portrait segmentation processing on a reference frame and a non-reference frame provided in an embodiment of the present application. Fig.11 As shown, for an image sequence to be processed for portrait segmentation, the image sequence includes I0, I1, I2, I3, I4 ... I k And so on. Among them, I0, I3, I k is the reference frame, and I1, I2, and I4 are non-reference frames.

[0231] For the reference frame, the portrait segmentation result is mainly obtained through two steps: semantic estimation and detail prediction. Semantic estimation refers to using a semantic feature extraction network to extract the semantic features of the reference frame, and detail prediction refers to using the first semantic segmentation network to process the semantic features of the reference frame to predict the portrait segmentation result.

[0232] For non-reference frames, after downsampling the non-reference frames, the semantic features of the reference frames before the non-reference frames and the non-reference frames themselves are used to predict the semantic features of the non-reference frames (i.e. Fig.11 After obtaining the semantic features of the non-reference frame, the portrait details are predicted based on the second semantic segmentation network, and finally the portrait segmentation result of the non-reference frame is obtained.

[0233] like Fig.11 As shown, for non-reference frames, after being processed by the method provided in this embodiment, accurate portrait segmentation results can be obtained without executing a complete semantic feature extraction process, thus saving a lot of calculation.

[0234] During the model training process, the network for processing the reference frame and the network for processing the non-reference frame are trained separately and independently. The network for processing the reference frame includes a semantic feature extraction network that performs semantic estimation and a first semantic segmentation network that performs detail prediction. The network for processing the non-reference frame includes a denoising network that performs a denoising process and a second semantic segmentation network that performs detail prediction.

[0235] For the semantic feature extraction network used to process the reference frame, the semantic feature extraction network is mainly used to extract the semantic features of the reference frame to obtain the rough outline information of the portrait. The semantic feature extraction network can adopt an encoder-decoder structure. Specifically, the semantic feature extraction network can include a lightweight residual network (MobileNetV2), a feature pyramid module (Pyramid Pooling Module, PPM) and two MobileV2 separated convolutions connected in sequence.

[0236] Among them, MobileNetV2 can be regarded as a feature extractor, the input is the H*W*3 RGB image matrix, and the output is the h*w*320 feature matrix. Among them, H and W are the height and width of the original image, and h and w are the height and width of the feature map. Fig.12 As shown, in the feature map output by MobileNetV2, h = H / 32, w = W / 32.

[0237] PPM is similar to a plug-in in the network, which can improve the network effect. PPM is usually placed after the feature extractor MobilenetV2, with an input feature matrix of h*w*320 and an output feature map of h*w*320.

[0238] In the semantic feature extraction network, the input image is processed by MobileV2 to obtain 320 feature maps with a size of 1 / 32 of the original image. These feature maps are then processed by the PPM structure to obtain fused features. After the fused features are upsampled, they are processed by two MobileV2 separation convolutions to output 128 semantic features with a size of 1 / 16 of the original image. Among them, the semantic features are processed by two branches: one branch outputs the detailed prediction in the next stage; the other branch is processed by the Sigmoid activation function in the training stage to obtain a rough mask s p , making this stage more focused on estimating the rough mask s p , the loss function is designed as follows:

[0239]

[0240] in, represents the semantic estimation loss function, G() represents Gaussian blur, α g is the target mask (i.e., the real portrait segmentation result), and ||·||2 represents the mean square error. p is a rough mask with fuzzy edges, while the target mask α g The edge of the target mask is clear, so g Apply Gaussian blur to compare the rough mask p and the target mask α g .

[0241] For the first semantic segmentation network that receives the output of the semantic feature extraction network as input, the first semantic segmentation network is used to further predict details based on the semantic features obtained through semantic estimation, thereby restoring relatively fine portrait edges and details. Compared with the semantic feature extraction network, the first semantic segmentation network has fewer convolution modules and a maximum number of channels of 128 (the maximum number of channels of the semantic feature extraction network is 320).

[0242] Specifically, the first semantic segmentation network repeats the "2x upsampling and separable convolution processing" 4 times on the semantic features output by the semantic feature extraction network to obtain the final decoding features, and then passes through the Sigmoid output layer to obtain the final mask. Among them, the intermediate features extracted from the semantic estimation are spliced ​​in the form of U-NET jump connection in the 1 / 2 size feature map part. Separable convolution processing refers to dividing the original ordinary convolution into two steps: 1. Channel-by-channel convolution: perform k*k convolution on each channel of the original feature map (a total of c1) to obtain h*w*c1; 2. Point-by-point convolution: perform 1x1 convolution on the result of the channel-by-channel convolution, change the number of channels to c2, and obtain h*w*c2.

[0243] For example, using α d To represent the final output of the detail prediction part, the loss function of this part is as follows:

[0244]

[0245] in, represents the detail prediction loss function, The first term is the absolute value loss, α d is the mask output by the first semantic segmentation network, α g is the target mask (i.e. the real portrait segmentation result), ∈ is a constant, The second term is the marginal loss, Represents the Sobel edge detection operator to enhance the image edge effect.

[0246] Therefore, combining the semantic estimation loss function and detail prediction loss function introduced above, it can be obtained that the reference frame loss function for training the semantic feature extraction network and the first semantic segmentation network for processing the reference frame can be composed of semantic estimation and detail prediction, as shown below:

[0247]

[0248] in, represents the base frame loss function, represents the detail prediction loss function, represents the semantic estimation loss function, λ d and λ s is a hyperparameter.

[0249] That is to say, in the process of training the semantic feature extraction network and the first semantic segmentation network, the above-mentioned reference frame loss function is actually used to train the semantic feature extraction network and the first semantic segmentation network together.

[0250] The above introduces the training process of the semantic feature extraction network and the first semantic segmentation network for processing the reference frame. The following will introduce the training process of the denoising network and the second semantic segmentation network for processing the non-reference frame.

[0251] For the tth image in the image sequence, i.e. the reference frame Its semantic features and rough masks are recorded as and (h and w represent the height and width of the feature map, both of which are 1 / 16 of the width and height of the original image, and C is the number of channels, that is, the number of features.) For the t+kth image in the image sequence, i.e., the non-reference frame I t+k , its semantic features and rough mask are recorded as F t+k and According to the analysis, and The similarity is high and the error statistics show Gaussian distribution characteristics. Fig.13 As shown, Fig.13 A schematic diagram of a process for processing non-reference frames provided in an embodiment of the present application. The problem to be solved in this solution is how to design a function or filter f so that the result after filtering is As close to the real features as possible

[0252]

[0253] Here, the softmax function φ(·) is used to transform the feature matrix into a probabilistic random field, as shown below.

[0254]

[0255] It should be noted that, because the neural network training process is the process of fitting continuous functions in the real domain, the advantages of converting the feature matrix into a probabilistic random field are that the information of the original matrix can be retained to the greatest extent from a probability perspective, and that discontinuous points can be eliminated through the conversion, that is, elements in the current matrix that may have an absolute value that is too large or too small, to avoid affecting the neural network parameter fitting.

[0256] The problem to be solved in this solution is transformed into a supervised conditional probability distribution generation problem. This filter f(·) can be generated using a neural network f θ (·) to approach.

[0257]

[0258] For example, see Fig.13 , Fig.13 This is a schematic diagram of the structure of the noise reduction network provided in the embodiment of the present application. Fig.13As shown, the denoising network includes a Softmax function, a convolution module 1, a convolution module 2, a 1*1 convolution layer (Conv), a BN function and a ReLU function. Among them, the convolution module 1 and the convolution module 2 can adopt the same structure. Fig.13 In the figure, convolution module 1 includes a 1*1 convolution layer, a BN function, a ReLU function, a 5*5 grouped convolution with an expansion rate of 2, a SE module, a ReLU function, a 1*1 convolution layer, and a linear layer. Using convolution module 1 and convolution module 2 can expand the receptive field to the entire feature map at a scale of 1 / 16.

[0259] In addition, after the semantic features of the non-reference frame are predicted by the denoising network, the denoising loss function can be calculated based on the semantic features predicted by the denoising network and the semantic features of the non-reference frame extracted by the semantic feature extraction network. Specifically, the KL divergence can be used to measure the error between the predicted distribution and the true distribution, as shown in the following formula:

[0260]

[0261] in, represents the denoising loss function; φ(F t+k ) represents the semantic features output by the semantic feature extraction network; Represents the semantic features predicted by the denoising network; KL() represents the KL divergence, which is a metric used to measure the similarity between two probability distributions.

[0262] Furthermore, the semantic features output by the denoising network are input into the second semantic segmentation network for detail prediction, thereby obtaining the mask of the non-reference frame. Similarly, referring to the above-mentioned detail prediction loss function, the detail prediction loss function corresponding to the non-reference frame can also be constructed based on the mask output by the second semantic segmentation network and the true mask of the non-reference frame. Then, based on the above-mentioned denoising loss function and the detail prediction loss function corresponding to the non-reference frame, the final non-reference frame loss function is obtained, as shown in the following formula:

[0263]

[0264] in, represents the non-reference frame loss function, represents the detail prediction loss function corresponding to the non-reference frame, represents the denoising loss function, λ d and λ s is a hyperparameter.

[0265] That is to say, in the process of training the denoising network and the second semantic segmentation network, the denoising network and the second semantic segmentation network are actually trained together using the above-mentioned non-reference frame loss function.

[0266] For example, see Fig.14 , Fig.14 A comparative schematic diagram of processing a reference frame and a non-reference frame provided in an embodiment of the present application. Fig.14 As shown in the figure, the process of processing the reference frame includes two steps: semantic estimation and detail prediction, where semantic estimation takes 4ms and detail prediction takes 2.7ms; the process of processing the non-reference frame also includes two steps: semantic estimation and detail prediction, and the semantic estimation process of processing the non-reference frame reuses the semantic features of the reference frame, so the semantic estimation of the non-reference frame takes 1ms and the detail prediction takes 2.7ms. Obviously, in the process of predicting the semantic features of the non-reference frame, since the semantic features of the reference frame are reused, there is no need to perform a complete semantic feature extraction process, so a lot of calculation and calculation time can be saved.

[0267] The above embodiments introduce the method provided by the embodiments of the present application, and the device for executing the above method will be introduced below.

[0268] See also Fig.15 , Fig.15 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.15 As shown, the image processing device includes: an acquisition module 1501, which is used to acquire a first image and a second image, wherein the second image and the first image are located in the same image sequence, and the second image is located after the first image; a processing module 1502, which is used to process the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; the processing module 1502 is also used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; the processing module 1502 is also used to input the second image into the semantic feature extraction network to obtain target semantic features; the processing module 1502 is also used to update the denoising network based on a loss function value to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0269] In a possible implementation manner, the similarity between the first image and the second image is not less than a first threshold.

[0270] In a possible implementation manner, in the image sequence, the number of images spaced between the first image and the second image is less than a second threshold.

[0271] In one possible implementation, the processing module 1502 is specifically used to: perform a first number of feature extraction processing on the first image through a semantic feature extraction network; perform a second number of feature extraction processing on the second image to obtain the original features of the second image, and the second number is smaller than the first number; process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0272] In a possible implementation, the processing module 1502 is specifically used to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0273] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0274] In one possible implementation, the processing module 1502 is also used to: process the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; process the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; wherein the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

[0275] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0276] See also Fig.16 , Fig.16 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig.16 As shown, the model training device includes: an acquisition module 1601, which is used to acquire the semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; a processing module 1602, which is used to process the semantic features of the first image and the second image through a denoising network, and predict the semantic features of the second image; the processing module 1602 is also used to input the second image into the semantic feature extraction network to obtain the target semantic features; the processing module 1602 is also used to update the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0277] In a possible implementation manner, the similarity between the first image and the second image is not less than a first threshold.

[0278] In a possible implementation manner, in the image sequence, the number of images spaced between the first image and the second image is less than a second threshold.

[0279] In one possible implementation, the processing module 1602 is also used to: perform semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determine the difference value between the image segmentation result of the second image and the true segmentation result of the second image, and the true segmentation result of the second image is pre-annotated; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0280] In a possible implementation, the processing module 1602 is further configured to: update the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0281] In one possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image through a semantic feature extraction network; the processing module 1602 is specifically used to: perform a second-order feature extraction process on the second image to obtain the original features of the second image, the second order is smaller than the first order; process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0282] In a possible implementation, the processing module 1602 is specifically used to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0283] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0284] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0285] Next, an execution device provided by an embodiment of the present application is introduced. Fig.17 , Fig.17This is a schematic diagram of the structure of an execution device provided in an embodiment of the present application. The execution device 1700 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703 and a memory 1704 (wherein the number of processors 1703 in the execution device 1700 can be one or more, Fig.17 In the example of FIG. 1701 , a processor 1703 is used, where the processor 1703 may include an application processor 17031 and a communication processor 17032. In some embodiments of the present application, the receiver 1701, the transmitter 1702, the processor 1703 and the memory 1704 may be connected via a bus or other means.

[0286] The memory 1704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1703. A portion of the memory 1704 may also include a non-volatile random access memory (NVRAM). The memory 1704 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0287] The processor 1703 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0288] The method disclosed in the above embodiment of the present application can be applied to the processor 1703, or implemented by the processor 1703. The processor 1703 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1703. The above processor 1703 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1703 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1704, and the processor 1703 reads the information in the memory 1704 and completes the steps of the above method in combination with its hardware.

[0289] The receiver 1701 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1702 can be used to output digital or character information through the first interface; the transmitter 1702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1702 can also include a display device such as a display screen.

[0290] The execution device provided in the embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the method for determining the model structure described in the above embodiment, or so that the chip in the training device executes the method for determining the model structure described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0291] For details, please refer to Fig.18 , Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1800, NPU 1800 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0292] In some implementations, the operation circuit 1803 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1803 is a two-dimensional systolic array. The operation circuit 1803 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1803 is a general-purpose matrix processor.

[0293] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1802 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1801 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1808.

[0294] The unified memory 1806 is used to store input data and output data. The weight data is directly transferred to the weight memory 1802 through the direct memory access controller (DMAC) 1805. The input data is also transferred to the unified memory 1806 through the DMAC.

[0295] BIU stands for Bus Interface Unit 1810 , which is used for the interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1809 .

[0296] The bus interface unit 1810 (BIU) is used for the instruction fetch memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0297] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1806 or to transfer weight data to the weight memory 1802 or to transfer input data to the input memory 1801.

[0298] The vector calculation unit 1807 includes multiple operation processing units, and further processes the output of the operation circuit 1803 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0299] In some implementations, the vector calculation unit 1807 can store the processed output vector to the unified memory 1806. For example, the vector calculation unit 1807 can apply a linear function; or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1807 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1803, for example, for use in a subsequent layer in a neural network.

[0300] An instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;

[0301] Unified memory 1806, input memory 1801, weight memory 1802 and instruction fetch memory 1809 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0302] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

[0303] See also Fig.19 , Fig.19 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. The present application also provides a computer-readable storage medium. In some embodiments, the above Figure 6 The disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0304] Fig.19 Schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0305] In one embodiment, the computer readable storage medium 1900 is provided using a signal bearing medium 1901. The signal bearing medium 1901 may include one or more program instructions 1902, which when executed by one or more processors may provide the above-mentioned Figure 3 The functions or parts of the functions described. In addition, Fig.19 Program instructions 1902 in also describe example instructions.

[0306] In some examples, signal bearing medium 1901 may include computer readable medium 1903 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, memory, ROM or RAM, and the like.

[0307] In some embodiments, the signal bearing medium 1901 may include a computer recordable medium 1904, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 1901 may include a communication medium 1905, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1901 may be communicated by a wireless form of the communication medium 1905 (e.g., a wireless communication medium complying with the IEEE 802 standard or other transmission protocol).

[0308] The one or more program instructions 1902 may be, for example, computer executable instructions or logic implementation instructions. In some examples, the computing device of the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1902 communicated to the computing device via one or more of the computer readable medium 1903, the computer recordable medium 1904, and / or the communication medium 1905.

[0309] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0310] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0311] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0312] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0313] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0314] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0315] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0316] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is after the first image; Processing the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict an image segmentation result of the first image; The semantic features of the first image and the second image are processed by a denoising network to predict the semantic features of the second image, and the semantic features of the second image are used to obtain an image segmentation result of the second image.

2. The method according to claim 1, characterized in that The similarity between the first image and the second image is not less than a first threshold.

3. The method according to claim 1 or 2, characterized in that: In the image sequence, the number of images between the first image and the second image is less than a second threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The processing of the first image by using a semantic feature extraction network includes: Performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The step of processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image includes: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

5. The method according to claim 4, characterized in that The processing of the semantic features of the first image and the original features of the second image by the denoising network to predict the semantic features of the second image includes: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

6. The method according to any one of claims 1 to 5, characterized in that: The denoising network includes a convolutional neural network or an attention network.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Processing the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; Processing the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; The first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

8. The method according to any one of claims 1 to 7, characterized in that: The image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

9. A model training method, characterized in that: include: Acquire semantic features of a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict an image segmentation result of the first image, and the semantic feature extraction network is a trained network; Processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; Inputting the second image into the semantic feature extraction network to obtain target semantic features; Based on the loss function value, the denoising network is updated to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

10. The method according to claim 9, characterized in that The similarity between the first image and the second image is not less than a first threshold.

11. The method according to claim 9 or 10, characterized in that: In the image sequence, the number of images between the first image and the second image is less than a second threshold.

12. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; Determine a difference value between an image segmentation result of the second image and a true segmentation result of the second image, where the true segmentation result of the second image is pre-marked; The loss function value is obtained based on the difference value and the distance between the semantic feature of the second image and the target semantic feature.

13. The method according to any one of claims 9 to 12, characterized in that: The semantic feature of the first image is obtained by performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The step of processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image includes: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

14. The method according to claim 13, characterized in that The processing of the semantic features of the first image and the original features of the second image by the denoising network to predict the semantic features of the second image includes: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

15. An image processing device, characterized in that: include: an acquisition module, used for acquiring a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is after the first image; A processing module, configured to process the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict an image segmentation result of the first image; The processing module is further used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image, and the semantic features of the second image are used to obtain an image segmentation result of the second image.

16. The device according to claim 15, characterized in that The similarity between the first image and the second image is not less than a first threshold.

17. The device according to claim 15 or 16, characterized in that In the image sequence, the number of images between the first image and the second image is less than a second threshold.

18. The device according to any one of claims 15 to 17, characterized in that: The processing module is specifically used for: Performing a first number of feature extraction processes on the first image by the semantic feature extraction network; Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

19. The device according to claim 18, characterized in that The processing module is specifically used for: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

20. The device according to any one of claims 15 to 19, characterized in that: The denoising network includes a convolutional neural network or an attention network.

21. The device according to any one of claims 15 to 20, characterized in that: The processing module is further used for: Processing the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; Processing the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; The first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

22. A model training device, characterized in that: include: an acquisition module, used to acquire semantic features of a first image and a second image, wherein the second image and the first image are located in the same image sequence and the second image is located after the first image, wherein the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, wherein the semantic features of the first image are used to predict an image segmentation result of the first image, and wherein the semantic feature extraction network is a trained network; The processing module is used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; The processing module is further used to input the second image into the semantic feature extraction network to obtain target semantic features; The processing module is also used to update the denoising network based on a loss function value to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

23. The device according to claim 22, characterized in that The processing module is further used for: Performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determining a difference value between an image segmentation result of the second image and a true segmentation result of the second image; The loss function value is obtained based on the difference value and the distance between the semantic feature of the second image and the target semantic feature.

24. The device according to any one of claims 22-23, characterized in that The semantic feature of the first image is obtained by performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The processing module is further used for: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

25. An image processing device, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 8.

26. A model training device, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 9 to 14.

27. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 14.

28. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 14.

Images