Image processing method, model training method and related apparatus

Abstract

An image processing method, applied to the technical field of artificial intelligence. The method comprises: dividing images in a continuously acquired image sequence into a reference frame and a non-reference frame; with regard to the non-reference frame in the image sequence, multiplexing semantic features of the reference frame before the non-reference frame to predict semantic features of the non-reference frame, instead of extracting the semantic features of the non-reference frame again; and, on the basis of the semantic features of the non-reference frame, predicting an image segmentation result of the non-reference frame. The present solution effectively utilizes the similarity between the reference frame and the non-reference frame, and, by means of multiplexing the semantic features of the reference frame, reduces a semantic feature extraction process for the non-reference frame, thus effectively reducing the calculation amount of the image segmentation process for the non-reference frame, and extending the application scenarios of the image processing method.

Description

Image processing method, model training method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 13, 2023, with application number 202310701005.6 and invention name “An image processing method, model training method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, a model training method, and related devices. Background Art

[0003] Image segmentation is a key branch of computer vision, widely used across industries. Image segmentation involves dividing an image into a number of desired blocks based on its intrinsic characteristics (such as texture, brightness, or topology). Each block has similar characteristics, while different blocks have distinct characteristics. With the increasing popularity of deep learning technology, the methods used to implement image segmentation are gradually shifting from traditional approaches to deep learning-based methods.

[0004] Currently, deep learning often uses a U-net architecture for image segmentation. This structure essentially divides the network into two parts: an expansion path, where features are scaled down, and a contraction path, where features are scaled down. In the expansion path, the input image undergoes layer-by-layer convolution and downsampling from left to right to obtain global semantic information about each object. In the contraction path, upsampling is performed to restore spatial details, and the desired image segmentation result is ultimately achieved by splicing the previous semantic information.

[0005] However, current deep learning-based image processing methods are computationally intensive. In scenarios where real-time segmentation of continuously acquired images is required, such as video calls or video conferencing, devices with limited computing resources often struggle to perform segmentation on each captured image, limiting their applicability.

[0006] Summary of the Invention

[0007] This application provides an image processing method that can effectively reduce the computational complexity of the image segmentation process and expand the application scenarios of the image processing method.

[0008] In a first aspect, the present application provides an image processing method for segmenting images in an image sequence. In this method, a first image and a second image in the image sequence are first acquired. The first image may be a reference frame in the image sequence, and the second image may be a non-reference frame in the image sequence. The second image is located after the first image in the image sequence, and the second image and the first image meet a preset similarity requirement. The image sequence includes multiple continuously acquired images and can essentially be understood as a video.

[0009] Then, the first image is processed by a semantic feature extraction network to obtain semantic features of the first image. The semantic features of the first image are essentially image features stored in a three-dimensional matrix form and can be used to predict the image segmentation result of the first image.

[0010] Secondly, the semantic features of the first image and the second image are processed through a denoising network to predict the semantic features of the second image. These semantic features are used to obtain the image segmentation result of the second image. Since the second image, which is a non-reference frame, has a high similarity with the first image, which is a reference frame, the denoising network predicts the semantic features of the non-reference frame based on the semantic features of the reference frame and the image content of the non-reference frame itself by reusing the semantic features of the reference frame, thus avoiding the need to re-extract the semantic features of the non-reference frame.

[0011] Finally, semantic segmentation processing can be performed on the semantic features of the second image, that is, segmentation of each part in the image can be achieved based on the semantic information of each part contained in the semantic features of the second image to obtain an image segmentation result of the second image.

[0012] In this solution, when the first and second images in an image sequence are acquired, for the second image that follows the first image in the image sequence, the semantic features of the first image are reused to predict the semantic features of the second image, without re-extracting the semantic features of the second image. The image segmentation result of the second image is then obtained based on the semantic feature prediction of the second image. Because this solution effectively utilizes the similarity between images in the image sequence, by reusing the semantic features of the previous image to reduce the semantic feature extraction process for the subsequent image, it can effectively reduce the computational complexity of the image segmentation process for some images, expanding the application scenarios of the image processing method.

[0013] In one possible implementation, the similarity between the first image and the second image is no less than a first threshold. In other words, the similarity between the images can be used to determine whether each image in the image sequence is a reference frame or a non-reference frame, thereby ensuring that the similarity between the reference frames and the non-reference frames is no less than the first threshold. This ensures that the semantic features of the first image are accurately predicted when the semantic features of the second image are reused, while ensuring a high degree of similarity between the first and second images.

[0014] In general, by determining the reference frame and non-reference frame based on the similarity between images, it is possible to ensure that the non-reference frame has a high similarity with the adjacent reference frames, thereby ensuring the accuracy of the semantic features of the non-reference frame predicted based on the semantic features of the reference frame.

[0015] In one possible implementation, the number of images between a first image and a second image in an image sequence is less than a second threshold. In other words, a fixed number of images may be used as an interval, with the first image of each interval serving as a reference frame. This means that the number of non-reference frames between two adjacent reference frames is fixed. Thus, the number of images between the first image serving as the reference frame and the second image serving as the non-reference frame is less than a fixed threshold (i.e., the second threshold).

[0016] In this solution, by limiting the number of images between the first image and the second image in the image sequence, the continuity of the first image and the second image can be guaranteed, thereby ensuring that the first image and the second image have a high degree of similarity, and ensuring the accuracy of reusing the semantic features of the first image to predict the semantic features of the second image.

[0017] In a possible implementation, processing the first image through the semantic feature extraction network may specifically include performing a first number of feature extraction processes on the first image through the semantic feature extraction network.

[0018] Processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image can specifically include: performing a second number of feature extraction processes on the second image to obtain the original features of the second image. The second number of times is smaller than the first number, for example, the first number is 5 and the second number is 4. For example, a feature extraction process is first performed on the second image through a convolutional network to obtain a feature map that is 1 / 2 the size of the second image, and then the feature map is downsampled using a bilinear interpolation method to obtain original features of the same size as the semantic features of the first image. Then, the semantic features of the first image and the original features of the second image are processed through a denoising network to predict the semantic features of the second image.

[0019] In this solution, by reducing the number of feature extraction processes performed on the second image and then using the denoising network to fuse the features of the second image and the semantic features of the first image, the computational complexity of extracting the semantic features of the second image can be effectively reduced, thereby improving image processing efficiency.

[0020] In one possible implementation, the semantic features of the first image and the original features of the second image are processed through a denoising network to predict the semantic features of the second image, including: splicing the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and inputting the spliced ​​features into the denoising network to predict the semantic features of the second image.

[0021] In one possible implementation, after obtaining the semantic features of the first image, semantic segmentation processing can be performed on the semantic features of the first image to obtain an image segmentation result for the first image. The image segmentation result for the first image is predicted by a first semantic segmentation network, and the image segmentation result for the second image is predicted by a second semantic segmentation network. That is, for a reference frame, the first semantic segmentation network is used to process the semantic features of the reference frame to obtain an image segmentation result for the reference frame; for a non-reference frame, the second semantic segmentation network is used to process the semantic features of the non-reference frame to obtain an image segmentation result for the non-reference frame.

[0022] In this scheme, by adopting different semantic segmentation networks to process the semantic features of the reference frame and the semantic features of the non-reference frame respectively, the deviation of the predicted semantic features of the non-reference frame can be well compensated, and the image segmentation effect can be finally achieved based on the predicted semantic features.

[0023] In one possible implementation, the first and second semantic segmentation networks have the same network structure, but have different weight parameters. That is, during training, the training processes for reference frames and non-reference frames are separate and independent. By independently training the first semantic segmentation network for processing reference frames and the second semantic segmentation network for processing non-reference frames, it is possible to effectively ensure that the first and second semantic segmentation networks have different weight parameters.

[0024] In this solution, by adopting the same network structure as the first semantic segmentation network and the second semantic segmentation network, and training the first semantic segmentation network and the second semantic segmentation network separately, the first semantic segmentation network and the second semantic segmentation network have different weight parameters. While reusing the network structure to reduce the network design process, the accuracy of the two semantic segmentation networks in performing image segmentation tasks can be guaranteed.

[0025] In one possible implementation, the denoising network may include a convolutional neural network or an attention network.

[0026] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0027] The second aspect of the present application provides a model training method, including: obtaining semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; inputting the second image into the semantic feature extraction network to obtain target semantic features; updating the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0028] In this scheme, a semantic feature extraction network for processing reference frames is pre-trained. For the denoising network used to predict the semantic features of non-reference frames, the semantic features extracted by the semantic feature extraction network and the semantic features predicted by the denoising network are used to construct a loss function, thereby realizing the training of the denoising network. Ultimately, a denoising network can be obtained that reduces the computational amount of semantic feature extraction for non-reference frames by reusing the semantic features of the reference frames.

[0029] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold.

[0030] In a possible implementation, in the image sequence, the number of images spaced between the first image and the second image is smaller than a second threshold.

[0031] In one possible implementation, the method further includes: performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determining a difference value between the image segmentation result of the second image and a true segmentation result of the second image, where the true segmentation result of the second image is pre-annotated, for example, manually pre-annotated; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0032] In this scheme, the loss function for training the denoising network is constructed from two aspects. On the one hand, it is based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network. On the other hand, it is based on the image segmentation results and the real segmentation results finally output by the semantic segmentation network. This enables the denoising network to take into account the final image segmentation results during the training process, ensuring that accurate image segmentation results can be obtained after the model training.

[0033] In a possible implementation, the method further includes: updating the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0034] In one possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image using a semantic feature extraction network. The semantic features of the first image and the second image are processed using a denoising network to predict the semantic features of the second image, including: performing a second-order feature extraction process on the second image to obtain original features of the second image, where the second order is smaller than the first order; and processing the semantic features of the first image and the original features of the second image using the denoising network to predict the semantic features of the second image.

[0035] In one possible implementation, the semantic features of the first image and the original features of the second image are processed through a denoising network to predict the semantic features of the second image, including: splicing the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and inputting the spliced ​​features into the denoising network to predict the semantic features of the second image.

[0036] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0037] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0038] In a third aspect, the present application provides an image processing device, comprising: an acquisition module, configured to acquire a first image and a second image, wherein the second image and the first image are located in the same image sequence, and the second image is located after the first image; a processing module, configured to process the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict image segmentation results of the first image, and the semantic feature extraction network is a trained network; the processing module, further configured to process the semantic features of the first image and the second image through a denoising network to predict semantic features of the second image; the processing module, further configured to input the second image into the semantic feature extraction network to obtain target semantic features; and the processing module, further configured to update the denoising network based on a loss function value to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0039] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold.

[0040] In a possible implementation, in the image sequence, the number of images spaced between the first image and the second image is smaller than a second threshold.

[0041] In one possible implementation, the processing module is specifically used to: perform a first-order feature extraction process on the first image through a semantic feature extraction network; perform a second-order feature extraction process on the second image to obtain original features of the second image, where the second order is smaller than the first order; and process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0042] In a possible implementation, the processing module is specifically configured to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0043] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0044] In one possible implementation, the processing module is further used to: process the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; process the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; wherein the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

[0045] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0046] The fourth aspect of the present application provides a model training device, including: an acquisition module, used to acquire the semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; a processing module, used to process the semantic features of the first image and the second image through a denoising network, and predict the semantic features of the second image; the processing module is also used to input the second image into the semantic feature extraction network to obtain the target semantic features; the processing module is also used to update the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0047] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold.

[0048] In a possible implementation, in the image sequence, the number of images spaced between the first image and the second image is smaller than a second threshold.

[0049] In one possible implementation, the processing module is further used to: perform semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determine a difference value between the image segmentation result of the second image and a true segmentation result of the second image, where the true segmentation result of the second image is pre-labeled; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0050] In a possible implementation, the processing module is further configured to update the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0051] In one possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image through a semantic feature extraction network; the processing module is specifically used to: perform a second-order feature extraction process on the second image to obtain the original features of the second image, where the second order is smaller than the first order; and process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0052] In a possible implementation, the processing module is specifically configured to: splice the semantic features of the first image and the original features of the second image to obtain spliced ​​features; and input the spliced ​​features into a denoising network to predict the semantic features of the second image.

[0053] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0054] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0055] In a fifth aspect, the present application provides an image processing device, comprising: a memory and a processor; the memory stores code, the processor is configured to execute the code, and when the code is executed, the image processing device executes a method as in any one of the implementation methods in the first aspect.

[0056] In a sixth aspect, the present application provides a model training device, which includes: a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes a method as in any one of the implementation methods in the second aspect.

[0057] In a seventh aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes a method as implemented in any one of the first and second aspects.

[0058] In an eighth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method as implemented in any one of the first or second aspects.

[0059] In a ninth aspect, the present application provides a chip comprising one or more processors, wherein some or all of the processors are configured to read and execute a computer program stored in a memory to perform the method in any one of the implementations of the first or second aspects.

[0060] Optionally, the chip includes a memory, and the memory is connected to the processor via a circuit or wire. Optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing results through the communication interface. The communication interface can be an input / output interface. The method provided in this application can be implemented by a single chip or by multiple chips working together.

[0061] Among them, the technical effects brought about by any design method in the second to ninth aspects can refer to the technical effects brought about by different implementation methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1 is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0063] FIG2 is a schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0064] FIG3 is a schematic diagram of an embodiment of the present application providing a method for implementing image segmentation using a U-shaped network structure;

[0065] FIG4 is a schematic diagram of a system architecture 400 provided in an embodiment of the present application;

[0066] FIG5 is a schematic structural diagram of an electronic device 101 provided in an embodiment of the present application;

[0067] FIG6 is a flow chart of an image processing method provided in an embodiment of the present application;

[0068] FIG7 is a schematic diagram of a process for performing image segmentation on a real-time acquired image according to an embodiment of the present application;

[0069] FIG8 is a schematic diagram of processing reference frames and non-reference frames through different networks according to an embodiment of the present application;

[0070] FIG9 is a flow chart of a model training method provided in an embodiment of the present application;

[0071] FIG10 is a schematic diagram of a system architecture for executing an image processing method provided in an embodiment of the present application;

[0072] FIG11 is a schematic diagram of performing portrait segmentation processing on a reference frame and a non-reference frame according to an embodiment of the present application;

[0073] FIG12 is a schematic diagram of performing portrait segmentation processing on a reference frame according to an embodiment of the present application;

[0074] FIG13 is a schematic diagram of a process for processing non-reference frames according to an embodiment of the present application;

[0075] FIG14 is a comparative diagram of processing a reference frame and a non-reference frame according to an embodiment of the present application;

[0076] FIG15 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0077] FIG16 is a schematic structural diagram of a model training device provided in an embodiment of the present application;

[0078] FIG17 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

[0079] FIG18 is a schematic structural diagram of a chip provided in an embodiment of the present application;

[0080] FIG19 is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of this application, rather than all embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0082] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchangeable where appropriate so that the embodiments can be implemented in a sequence other than that illustrated or described in this application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. In actual application, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Moreover, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application.

[0083] To facilitate understanding, some technical terms involved in the embodiments of this application are first introduced below.

[0084] (1) Neural Network

[0085] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0086] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0087] (2) Deep Neural Networks

[0088] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neural unit in the i-th layer must be connected to any neural unit in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neural unit in the second layer to the 2nd neural unit in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficients from the kth neural unit in the L-1th layer to the jth neural unit in the Lth layer are defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a predictive model with more parameters is more complex and has greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix for all layers of a trained deep neural network (a weight matrix formed by the vectors W of many layers).

[0089] (3) Convolutional Neural Network (CNN)

[0090] A convolutional neural network is a deep neural network with a convolutional architecture. It includes a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter. A convolutional layer refers to the layer of neural units in a convolutional neural network that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). Within a convolutional layer of a convolutional neural network, a neural unit may only be connected to some of the neural units in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neural units. Neural units in the same feature plane share weights, which are referred to as the convolution kernel. Shared weights can be understood as the way image information is extracted, regardless of position. The underlying principle is that the statistical information of one part of the image is the same as that of another part. This means that image information learned in one part can also be applied to another part. Therefore, the same learning can be used to obtain image information at all positions in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0091] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0092] Specifically, as shown in FIG1 , the convolutional neural network 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0093] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in this application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0094] Convolutional layer / pooling layer 120. Convolutional layer: As shown in Figure 2, convolutional layer / pooling layer 120 may include layers 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of a convolutional layer can serve as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0095] Taking convolution layer 121 as an example, convolution layer 121 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used, and multiple weight matrices of the same dimension are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions also have the same dimensions. The multiple feature maps with the same dimensions extracted are then merged to form the output of the convolution operation.

[0096] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0097] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0098] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, that is, as shown in the example 120 in Figure 1, each layer 121-126 can be a convolution layer followed by a pooling layer, or multiple convolution layers can be followed by one or more pooling layers.

[0099] Neural network layer 130: After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. Because as before, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other relevant information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (131, 132 to 13n as shown in Figure 1) and an output layer 140. The parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, text classification, speech classification, etc.

[0100] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140. The output layer 140 has a loss function value similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (the propagation from 110 to 140 in Figure 2 is forward propagation) is completed, the back propagation (the propagation from 140 to 110 in Figure 2 is back propagation) will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0101] It should be noted that the convolutional neural network 100 shown in Figure 1 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network prediction models. For example, as shown in Figure 2, multiple convolutional layers / pooling layers are run in parallel, and the extracted features are input into the full neural network layer 130 for processing.

[0102] Generally, convolution processing can be represented by Conv.

[0103] (4) Residual Network (ResNet)

[0104] Residual networks are an improved form of convolutional neural networks that are easy to optimize and can improve accuracy by increasing their depth. The residual blocks within residual networks use skip connections, which alleviates the vanishing gradient problem associated with increasing the depth of deep neural networks.

[0105] (5) Encoder

[0106] In the embodiments of the present application, the encoder is essentially a neural network model that can convert data such as text or images into vectors in a coding space (i.e., convert text or images into text features or image features). In the encoder-decoder structure, the encoder actually performs feature extraction on the input data to obtain the features of the input data.

[0107] (6) Decoder

[0108] In the embodiments of this application, the decoder is essentially a neural network model that converts the features output by the encoder into outputs. For example, in an image segmentation task, the encoder performs feature extraction on the input image to obtain image features; the decoder processes the image features to obtain the image segmentation result.

[0109] (7) Attention Network

[0110] Attention networks are network models that utilize the attention mechanism to accelerate model training. Currently, typical attention networks include Transformer networks. Models using the attention mechanism assign different weights to each part of the input sequence, thereby extracting more important features from the input sequence and ultimately achieving more accurate output.

[0111] In deep learning, the attention mechanism can be implemented using a weight vector that describes importance: when predicting or inferring an element, the weight vector determines the correlation between that element and other elements. For example, for a pixel in an image or a word in a sentence, the attention vector can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors can be used as an approximation of the target value.

[0112] The attention mechanism in deep learning simulates the attention mechanism of the human brain. For example, when a person looks at a painting, although their eyes can see the entire painting, when they look closely, their eyes actually focus on only a small part of the pattern. At this time, the human brain focuses primarily on this small part. In other words, when a person observes an image carefully, the human brain's attention is not evenly distributed across the entire image; instead, it is weighted differently. This is the core idea of ​​the attention mechanism.

[0113] Simply put, the human visual processing system tends to selectively focus on certain parts of an image and ignore other irrelevant information, which helps the human brain perceive. Similarly, in some problems involving language, speech, or vision, certain parts of the input may be more relevant than others in deep learning. Therefore, through the attention mechanism in the attention model, the attention model can perform different processing on different parts of the input data, so that the attention model dynamically focuses only on data relevant to the task.

[0114] (8) Loss function

[0115] During neural network training, because we want the output of the neural network to be as close as possible to the desired predicted value, we compare the current network's prediction with the desired target value and then update the weight vectors of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters of each layer in the neural network.) For example, if the network's prediction is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (i.e., the loss function value) indicates a greater difference, so neural network training becomes a process of minimizing this loss function value.

[0116] (9) Backpropagation algorithm

[0117] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial prediction model during training, reducing the error loss of the prediction model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial prediction model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal prediction model parameters, such as the weight matrix.

[0118] (10) Gradient descent

[0119] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate minimum-deviation prediction models. To find a local minimum of a function using gradient descent, an iterative search must be performed toward a point on the function at a specified step distance in the opposite direction of the gradient (or approximate gradient) corresponding to the current point. Gradient descent is one of the most commonly used methods for solving unconstrained optimization problems, such as predictive model parameters in machine learning algorithms.

[0120] Specifically, when solving for the minimum value of the loss function, we can use the gradient descent method to iterate step by step to obtain the minimized loss function value and prediction model parameter value. Conversely, if we need to solve for the maximum value of the loss function, we need to use the gradient ascent method to iterate.

[0121] (11) Reference frame

[0122] Typically, frames are used to describe video information, with reference frames being the most representative. Reference frames are frames where objects in the scene exhibit unusual motion. The rest of the frames between these unusual frames maintain a normal scene. A scene is a collection of related shots.

[0123] Simply put, when observing the objective world, people are often most interested in unusual events and gain a wealth of information from unusual changes. For an object, an unusual state refers to a significant change in its motion state, such as a shift from stillness to motion, or from motion to stillness, a change in direction of motion, or a significant change in speed. Similarly, when watching a video, people focus on the changes in the scene, which reflect significant changes in the motion state of the objects within it. Scene changes also include scene switching, which can be thought of as an object in the original scene suddenly moving to infinity, while in the new scene, the object moves from infinity, resulting in a drastic change in its motion state.

[0124] (12) Normalized exponential function (Softmax function)

[0125] The Softmax function is a generalization of the logistic function. It can compress a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that each element ranges between (0, 1) and the sum of all elements is 1.

[0126] (13) Batch Normalization (BN)

[0127] Normalization is a method of simplifying calculations by transforming a dimensionless expression into a scalar. Simply put, normalization is a dimensionless processing technique that converts the absolute values ​​of a physical system into relative values.

[0128] (14) Linear rectification function (ReLU)

[0129] The linear rectification function, also known as the rectified linear unit, is an activation function commonly used in artificial neural networks. It usually refers to nonlinear functions represented by ramp functions and their variants.

[0130] (15) Squeeze-and-Excitation (SE) module

[0131] Generally, the SE module consists of two parts: the Squeeze part and the Excitation part. For the Squeeze part, the original feature map has dimensions H*W*C. The Squeeze part compresses H*W*C to 1*1*C, which is equivalent to compressing H and W into one dimension. This allows this one-dimensional parameter to retain the global perspective of H*W and thus perceive a wider area.

[0132] For the activation part, after obtaining the features of the size of 1*1*C output by the compression part, a fully connected layer is added to predict the importance of each channel. After obtaining the importance of different channels, it is applied (stimulated) to the corresponding channels of the previous original feature map, and then subsequent operations are performed.

[0133] In general, the SE module essentially calculates the similarity in the channel dimension and enhances the information interaction between multiple channels.

[0134] In deep learning, a U-net structure is usually used to achieve image segmentation. Please refer to Figure 3, which is a schematic diagram of an embodiment of the present application that uses a U-net structure to achieve image segmentation. As shown in Figure 3, the U-net structure actually divides the network into two parts: an expansion path with features from large to small and a contraction path with features from small to large. In the expansion path, the input image is convolved and downsampled layer by layer from left to right to obtain the global semantic information of each object; in the contraction path, the spatial details of the image are restored by performing upsampling and the desired image segmentation result is finally obtained by splicing the previous semantic information.

[0135] Because the downsampled image features are concatenated each time the upsampling process predicts the image segmentation result, the U-shaped network architecture is computationally intensive. Consequently, in scenarios where real-time segmentation of continuously acquired images is required, such as video calls or video conferencing, devices with limited computing resources often struggle to continuously segment each acquired image, limiting the application of this image processing method.

[0136] In view of this, an embodiment of the present application provides an image processing method that divides images in a continuously acquired image sequence into reference frames and non-reference frames. For non-reference frames in the image sequence, the semantic features of the reference frame located before the non-reference frame are reused to predict the semantic features of the non-reference frame, without re-extracting the semantic features of the non-reference frame. The image segmentation result of the non-reference frame is then predicted based on the semantic features of the non-reference frame. Because this solution effectively utilizes the similarity between the reference frame and the non-reference frame, by reusing the semantic features of the reference frame to reduce the semantic feature extraction process for the non-reference frame, it can effectively reduce the computational complexity of the image segmentation process for the non-reference frame, thereby expanding the application scenarios of the image processing method.

[0137] For ease of understanding, the following will introduce the application scenarios and system architecture of the image processing method provided in the embodiments of the present application.

[0138] In one possible implementation, the image processing method provided by the embodiment of the present application can be applied in video call, video conference or live video scenarios. In video call, video conference or live video scenarios, users usually collect their own videos through terminal devices such as smart phones, tablet computers, laptops or personal computers, and send them to other users. Moreover, in these scenarios, users may need the terminal device to distinguish their portrait from the background in order to perform specific processing on the portrait or background, such as beautifying the portrait or replacing the background. In this way, after the terminal device collects the video, it can perform image segmentation on the image in the video based on the image processing method provided by the embodiment of the present application to determine the portrait and background in the image, and then perform specific processing on the portrait or background.

[0139] In another possible implementation, the image processing method provided by the embodiments of the present application can be applied in an autonomous driving scenario. In an autonomous driving scenario, an autonomous vehicle collects images of the road in real time while driving and identifies objects in the collected images to determine the driving strategy of the autonomous vehicle. In this case, the autonomous vehicle can use the image processing method provided by the embodiments of the present application to perform image segmentation on the images collected in real time, thereby obtaining image segmentation results corresponding to each image, so that the autonomous vehicle can determine the next driving strategy based on the image segmentation results.

[0140] In another possible implementation, the image processing method provided by the embodiment of the present application can be applied in the field of security or traffic safety. In the field of security, security cameras often collect images in a specific area in real time, and identify objects or pedestrians in the collected images to ensure whether the specific area is safe. Similarly, in the field of traffic safety, traffic cameras often collect images on traffic roads in real time, and identify vehicles or pedestrians in the collected images to determine whether the vehicles or pedestrians comply with traffic rules. Based on this, both security cameras and traffic cameras can use the image processing method provided by the embodiment of the present application to perform image segmentation on the images collected in real time, thereby obtaining image segmentation results corresponding to each image, so as to determine whether the security area is safe or whether vehicles or pedestrians comply with traffic rules.

[0141] In another possible implementation, the image processing method provided by the embodiment of the present application can also be applied in the field of smart home. In the field of smart home, some smart home appliances (such as smart TVs, sweeping robots, cameras or smart speakers) often collect images in the user's home in real time, and need to perform image segmentation on the collected images to continue to perform the next task based on the image segmentation results. For example, the sweeping robot can determine the cleanable area and the area to avoid obstacles based on the image segmentation results; the smart TV can determine to start or shut down based on the image segmentation results; the smart speaker can adjust the volume in all directions based on the image segmentation results, etc. Therefore, smart home appliances in the field of smart home can also use the image processing method provided by the embodiment of the present application to implement image segmentation on the collected images, so as to perform the next task based on the obtained image segmentation results.

[0142] In general, the image processing method provided in the embodiments of the present application can be applied to scenarios where image segmentation needs to be performed on continuously acquired images, and is not limited to the scenarios exemplified above.

[0143] Please refer to Figure 4, which is a schematic diagram of a system architecture 400 provided in an embodiment of the present application. As shown in Figure 4, in this system architecture 400, execution device 410 can be implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc.; execution device 410 can be deployed at a single physical site or distributed across multiple physical sites. Execution device 410 can use data in data storage system 420, or call program code in data storage system 420, to implement the classification model training method provided in an embodiment of the present application, thereby obtaining a model.

[0144] Users can operate their respective user devices (e.g., local device 401 and local device 402) to interact with execution device 410. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0145] Each user's local device can interact with the execution device 410 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0146] In one implementation, the execution device 410 is used to implement the training method of the classification model provided in the embodiment of the present application, and send the obtained model to the local device 401 and the local device 402 through the communication network, so that the local device 401 and the local device 402 can deploy and run the model, thereby implementing the image processing method provided in the embodiment of the present application.

[0147] In another implementation, one or more aspects of the execution device 410 can be implemented by each local device. For example, the local device 401 can provide local data or feedback calculation results to the execution device 410, or execute the image processing method and model training method provided in the embodiments of the present application.

[0148] It should be noted that all functions of the execution device 410 can also be implemented by the local device. For example, the local device 401 implements the functions of the execution device 410 and provides services to its own user, or provides services to the user of the local device 402.

[0149] In general, the image processing method and model training method provided in the embodiments of the present application can be applied to electronic devices, such as the above-mentioned execution device 410, local device 401 or local device 402. For example, the electronic device can be, for example, a server, a wireless electronic device in industrial control, a smart phone (mobile phone), a personal computer (personal computer, PC), a laptop computer, a tablet computer and other devices. For example, in order to take the application of the image processing method to terminal devices such as laptop computers or smart phones and the application of the model training method to the server as an example, the method provided in the embodiments of the present application is introduced in detail.

[0150] Please refer to Figure 5, which is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of the present application. As shown in Figure 5, electronic device 101 includes a processor 103, which is coupled to a system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A display adapter (video adapter) 107 can drive a display 109, which is coupled to system bus 105. System bus 105 is coupled to an input / output (I / O) bus via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with various I / O devices, such as an input device 117 (e.g., a touch screen), an external memory 121 (e.g., a hard disk, floppy disk, optical disk, or USB flash drive), a multimedia interface, etc., a transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and dynamic digital video images), and an external USB port 125. Optionally, the interface connected to the I / O interface 115 may be a USB interface.

[0151] The processor 103 may be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an ASIC.

[0152] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network card. Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Alternatively, network 127 can be a wireless network, such as a WiFi network or a cellular network.

[0153] The hard drive interface 131 is coupled to the system bus 105. The hard drive interface is connected to the hard drive 133. The internal memory 135 is coupled to the system bus 105. The data running in the internal memory 135 may include the operating system (OS) 137 of the electronic device 101, the application 143, and the scheduler.

[0154] The operating system consists of a shell 139 and a kernel 141. Shell 139 is an interface between the user and the operating system's kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: it waits for user input, interprets user input to the operating system, and processes various operating system output.

[0155] The kernel 141 consists of the parts of the operating system that manage memory, files, peripherals, and system resources. The kernel 141 directly interacts with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time slice management, interrupts, memory management, and I / O management.

[0156] Please refer to Figure 6, which is a flowchart of an image processing method provided by an embodiment of the present application. As shown in Figure 6, the image processing method includes the following steps 601-604.

[0157] Step 601: Acquire a first image and a second image, where the second image and the first image are in the same image sequence and the second image is located after the first image.

[0158] Specifically, the image processing method provided in this embodiment can be applied to perform image segmentation on images in a video. A video is essentially an image sequence, which is composed of a plurality of continuously captured images. Moreover, for each image in the image sequence, the image can be divided into a reference frame or a non-reference frame. Among them, a reference frame may refer to an image in which the motion state of a person or object in the image sequence changes significantly. A non-reference frame refers to an image other than a reference frame in the image sequence. Generally speaking, in an image sequence, one or more non-reference frames are usually included between two adjacent reference frames, so the difference between two adjacent reference frames is relatively large, while the difference between a non-reference frame and an adjacent reference frame is relatively small.

[0159] In this embodiment, the first image is a reference frame, and the second image is a non-reference frame. Furthermore, the reference frame and the non-reference frame meet a preset similarity requirement, that is, the first image and the second image meet the preset similarity requirement. The following describes how to determine the reference frame and the non-reference frame to ensure that the reference frame and the non-reference frame meet the preset similarity requirement.

[0160] In one possible implementation, whether each image in an image sequence is a reference frame can be determined by the similarity between images. Specifically, for an image sequence, the first image in the image sequence can be regarded as the reference frame, and for any image after the first image in the image sequence, the similarity between the image and the most recent reference frame before the image can be determined. If the similarity between the image and the most recent reference frame is greater than or equal to a first threshold, the image can be determined as a non-reference frame; if the similarity between the image and the most recent reference frame is less than the first threshold, the image can be determined as a new reference frame. The similarity between the two images can be measured by cosine similarity or structural similarity index measure (SSIM), which is not specifically limited in this embodiment. The first threshold can be, for example, a value such as 90, 92, or 95%, which can be adjusted according to the actual scenario, and is not specifically limited in this embodiment.

[0161] That is, for the first image and the second image described above, since the first image is the reference frame and the second image is a non-reference frame located after the first image, the similarity between the first image and the second image is no less than the first threshold. However, the similarity between the first image and the previous reference frame of the first image in the image sequence (i.e., the reference frame adjacent to the first image) is less than the first threshold.

[0162] In another possible implementation, a fixed number of images may be used as an interval, with the first image of each interval serving as the reference frame. This means that the number of non-reference frames between two adjacent reference frames is fixed. For example, assuming five consecutive images are used as an interval, with the first image of each interval serving as the reference frame, four non-reference frames are spaced between two adjacent reference frames. For example, assuming an image sequence includes 20 images, with the 1st to 5th images serving as the first interval, the 6th to 10th images serving as the second interval, the 11th to 15th images serving as the third interval, and the 16th to 20th images serving as the fourth interval; the 1st, 6th, 11th, and 16th images serving as reference frames, and the 2nd to 5th, 7th to 10th, 12th to 15th, and 17th to 20th images serving as non-reference frames.

[0163] That is, in an image sequence, the number of images between the first and second images is less than a second threshold. Specifically, a fixed number of images may be used as an interval, with the first image of each interval serving as the reference frame. This means that the number of non-reference frames between two adjacent reference frames is fixed. Thus, the number of images between the first image serving as the reference frame and the second image serving as the non-reference frame is less than a fixed threshold (i.e., the second threshold).

[0164] Generally speaking, in this embodiment, the reference frame is a representative image in the image sequence, including rich information, and the difference between two adjacent reference frames is large, while the difference between a non-reference frame and an adjacent reference frame is small.

[0165] Step 602: Process the first image through a semantic feature extraction network to obtain semantic features of the first image. The semantic features of the first image are used to predict an image segmentation result of the first image.

[0166] In this embodiment, the first image, serving as the reference frame, can be input into a semantic feature extraction network to extract image features and obtain semantic features of the first image. The semantic features of the first image are essentially image features stored in a three-dimensional matrix format and can be used to predict image segmentation results for the first image.

[0167] Specifically, the semantic feature extraction network can be a neural network for performing semantic estimation, capable of extracting image features based on the semantic information of each object in the image, so that the extracted semantic features can roughly indicate the position information of each object in the image, thereby facilitating subsequent image segmentation. The semantic feature extraction network can be, for example, a convolutional neural network or a Transformer network. This embodiment does not specifically limit the structure of the semantic feature extraction network.

[0168] Step 603: Process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image.

[0169] Since the difference between the non-reference frame and the adjacent reference frame is small, this embodiment can reuse the semantic features of the reference frame and predict the semantic features of the non-reference frame based on the semantic features of the reference frame and the image content of the non-reference frame itself, avoiding the need to re-extract the semantic features of the non-reference frame and reducing the large amount of computation required to extract the semantic features.

[0170] Specifically, the denoising network is a neural network, such as a convolutional neural network, whose input may include the semantic features of the reference frame and the non-reference frame itself, and can eliminate the difference between the semantic features of the reference frame and the semantic features of the non-reference frame based on the non-reference frame itself, and then predict the semantic features of the non-reference frame. Therefore, in this embodiment, by using the denoising network to process the semantic features of the first image and inputting the second image into the denoising network, the semantic features of the second image (i.e., the semantic features of the second image) can be predicted by the denoising network. The semantic features of the second image are essentially also a kind of image features stored in the form of a three-dimensional matrix, which can be used to predict the image segmentation result of the second image.

[0171] Optionally, processing the first image through the semantic feature extraction network may specifically include: performing a first number of feature extraction processes on the first image through the semantic feature extraction network.

[0172] In addition, the semantic features of the first image and the second image are processed by a denoising network to predict the semantic features of the second image. Specifically, this may include: performing a second number of feature extraction processes on the second image to obtain the original features of the second image. The second number is smaller than the first number, for example, the first number is 5 and the second number is 4. For example, a feature extraction process is first performed on the second image through a convolutional network to obtain a feature map with a size of 1 / 2 of the second image, and then the feature map is downsampled using a bilinear interpolation method to obtain original features with the same size as the semantic features of the first image. Then, the semantic features of the first image and the original features of the second image are spliced ​​to obtain spliced ​​features; and then the spliced ​​features are input into the denoising network to predict the semantic features of the second image.

[0173] In this way, by reducing the number of feature extraction processes performed on the second image and then using the denoising network to fuse the features of the second image and the semantic features of the first image, the amount of computation required to extract the semantic features of the second image can be effectively reduced, thereby improving image processing efficiency.

[0174] Step 604 : Perform semantic segmentation processing on the semantic features of the second image to obtain an image segmentation result of the second image.

[0175] After obtaining the semantic features of the second image corresponding to the second image, semantic segmentation processing can be performed on the semantic features of the second image, that is, the segmentation of each part in the image is realized based on the semantic information of each part contained in the semantic features of the second image, thereby obtaining the image segmentation result of the second image.

[0176] In this approach, images in a continuously acquired image sequence are divided into reference frames and non-reference frames. For non-reference frames in the image sequence, the semantic features of the reference frame preceding the non-reference frame are reused to predict the semantic features of the non-reference frame, without re-extracting the semantic features of the non-reference frame. The image segmentation results for the non-reference frame are then predicted based on the semantic features of the non-reference frame. Because this approach effectively leverages the similarity between reference and non-reference frames and reduces the semantic feature extraction process for non-reference frames by reusing the semantic features of the reference frame, the computational complexity of the image segmentation process for non-reference frames can be effectively reduced, expanding the application scenarios of the image processing method.

[0177] Optionally, steps 601-604 above describe the process of performing image segmentation on the second image, which is a non-reference frame. In practical applications, image segmentation is often required for each image in the entire image sequence. Therefore, before performing image segmentation on the second image, image segmentation is actually required on the first image.

[0178] Exemplarily, before the above-mentioned step 603, after obtaining the semantic features of the first image, semantic segmentation processing can be performed on the semantic features of the first image to obtain an image segmentation result of the first image. That is, in the process of performing image segmentation on each image in the image sequence, semantic features of the first image that is at the front of the image sequence and serves as a reference frame can be first extracted and image segmentation can be achieved based on the extracted semantic features of the first image. Then, by reusing the semantic features of the first image corresponding to the first image, the semantic features of the second image that is at the back of the image sequence and serves as a non-reference frame are predicted, thereby achieving image segmentation on the second image. That is, the semantic features of the front reference frame in the image sequence are used to predict the semantic features of the back non-reference frame, thereby simplifying the image segmentation process of the non-reference frame.

[0179] The image segmentation result of the first image can be predicted by a first semantic segmentation network, and the image segmentation result of the second image can be predicted by a second semantic segmentation network. That is, for the reference frame, the first semantic segmentation network is used to process the semantic features of the reference frame to obtain the image segmentation result of the reference frame; for the non-reference frame, the second semantic segmentation network is used to process the semantic features of the non-reference frame to obtain the image segmentation result of the non-reference frame.

[0180] Since the first image is a reference frame, the semantic features of the first image corresponding to the first image are extracted through a semantic feature extraction network, while the semantic features of the second image corresponding to the second image as a non-reference frame are predicted based on the semantic features of the first image. Therefore, the semantic features corresponding to the two images are obtained in different ways, and the semantic features of the second image predicted for the second image may deviate from the actual semantic features of the second image. Therefore, in this embodiment, different semantic segmentation networks can be used to process the semantic features of the reference frame and the semantic features of the non-reference frame respectively, so as to effectively compensate for the deviation of the semantic features of the predicted non-reference frame and ensure that the image segmentation effect is ultimately achieved based on the predicted semantic features.

[0181] Optionally, the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters. For example, the first semantic segmentation network and the second semantic segmentation network are both neural network structures. Since the weight parameters of the first semantic segmentation network and the second semantic segmentation network are determined during the training process, the first semantic segmentation network and the second semantic segmentation network having different weight parameters actually represent that the first semantic segmentation network and the second semantic segmentation network are trained separately during the training process. That is, during the training process, the training processes for the reference frames and the non-reference frames are separate and independent. By independently training the first semantic segmentation network for processing the reference frames and the second semantic segmentation network for processing the non-reference frames, the performance of the first semantic segmentation network and the second semantic segmentation network can be effectively guaranteed.

[0182] In this solution, by adopting the same network structure as the first semantic segmentation network and the second semantic segmentation network, and training the first semantic segmentation network and the second semantic segmentation network separately, the first semantic segmentation network and the second semantic segmentation network have different weight parameters. While reusing the network structure to reduce the network design process, the accuracy of the two semantic segmentation networks in performing image segmentation tasks can be guaranteed.

[0183] In summary, in this embodiment, reference frames and non-reference frames in an image sequence can be processed separately using two different branches. Furthermore, during the processing of non-reference frames, semantic features of the reference frames are reused, thereby reducing the computational effort required to process the non-reference frames.

[0184] For example, please refer to Figures 7 and 8. Figure 7 is a schematic diagram of a process for performing image segmentation on a real-time acquired image, provided in an embodiment of the present application; Figure 8 is a schematic diagram of processing a reference frame and a non-reference frame via different networks, provided in an embodiment of the present application. As shown in Figure 7, the process of performing image segmentation on a real-time acquired image includes the following steps 701-706.

[0185] Step 701: Acquire images collected in real time.

[0186] Since the image segmentation process is performed on images in a video, and the video is composed of multiple images continuously acquired in real time, during the execution of the method of this embodiment, the real-time acquired images can be continuously acquired and the image segmentation process can be performed on the acquired images.

[0187] Step 702: Determine whether the acquired image is a reference frame.

[0188] For an image acquired in real time, it may be determined whether the acquired image is a reference frame to determine a subsequent image segmentation processing method for the image.

[0189] In one possible implementation, whether an image is a reference frame can be determined based on the similarity between the image and the previous reference frame. Specifically, the closest reference frame to the image is first obtained, and then the similarity between the image and the reference frame is calculated. If the similarity between the image and the most recent reference frame is greater than or equal to a first threshold, the image can be determined as a non-reference frame. If the similarity between the image and the most recent reference frame is less than the first threshold, the image can be determined as a new reference frame.

[0190] In another possible implementation, whether an image is a reference frame can also be determined based on the number of images between the image and the previous reference frame. Specifically, the reference frame closest to the image is first obtained. If the number of images between the image and the reference frame is less than a specific threshold (e.g., 4), the image is determined to be a non-reference frame. If the number of images between the image and the reference frame is not less than the specific threshold, the image is determined to be a reference frame.

[0191] Step 703: If the acquired image is a reference frame, semantic feature extraction is performed on the image through a semantic feature extraction network to obtain semantic features of the image.

[0192] As shown in FIG8 , for the reference frame, in this embodiment, the image is input into a semantic feature extraction network, and the semantic feature extraction network is used to extract the semantic features of the image.

[0193] Step 704: Perform semantic segmentation processing on the semantic features of the image through the first semantic segmentation network to obtain an image segmentation result.

[0194] For the reference frame, after the semantic features of the reference frame are extracted by the semantic feature extraction network, the semantic features of the reference frame are further subjected to semantic segmentation processing by the first semantic segmentation network to obtain an image segmentation result of the reference frame.

[0195] Step 705: If the acquired image is a non-reference frame, the semantic features of the image are predicted using a denoising network based on the semantic features of the reference frame closest to the image in the image sequence and the image.

[0196] If the acquired image is a non-reference frame, the semantic features of the image are no longer extracted through the semantic feature extraction network. Instead, the denoising network is used to predict the semantic features of the image. The input of the denoising network includes the non-reference frame (i.e., the image itself) and the semantic features of the previous reference frame closest to the image (i.e., the features extracted by the semantic feature extraction network). The output of the denoising network is the predicted semantic features of the non-reference frame.

[0197] Step 706: Perform semantic segmentation processing on the semantic features of the image through the second semantic segmentation network to obtain an image segmentation result.

[0198] For an image that is a non-reference frame, after the semantic features of the non-reference frame are predicted by the denoising network, semantic segmentation processing can be performed on the semantic features of the non-reference frame by the second semantic segmentation network to obtain an image segmentation result of the non-reference frame.

[0199] Optionally, in this embodiment, the image segmentation results of the first image and the image segmentation results of the second image are both portrait segmentation results, and the image segmentation results of the first image and the second image are both used to perform background replacement for the portrait. In other words, the image processing method provided in this embodiment can be applied in scenarios such as video calls, video conferences, or live video broadcasts to segment the portrait and background of each image in a video, and then implement background replacement based on the segmented portrait and background.

[0200] In this scheme, since the content of different images in the video usually does not change much in scenarios such as video calls, video conferences or live video broadcasts, the main change is the posture of the characters, that is, the similarity between the reference frame and the non-reference frame is large. Therefore, an image processing method based on feature reuse is used to realize portrait segmentation in the video, which can effectively reduce the amount of calculation while ensuring the effect of portrait segmentation.

[0201] The above introduces the image processing method provided by the embodiment of the present application. The following introduces the model training method provided by the embodiment of the present application to facilitate understanding of the training method of the network mentioned in the above image processing method.

[0202] It is understandable that since the above-mentioned image processing method uses different networks to process the reference frame and the non-reference frame, during the model training process, different networks for processing the reference frame and the non-reference frame can also be trained independently to ensure that the trained network is suitable for executing the image processing method.

[0203] Please refer to Figure 9, which is a flow chart of a model training method provided in an embodiment of the present application. As shown in Figure 9, the model training method includes the following steps 901-905.

[0204] Step 901: Acquire a first image and a second image, where the second image and the first image are in the same image sequence and the second image is located after the first image.

[0205] In this embodiment, the first image and the second image are training data in a training set for training a denoising network. Furthermore, the first image and the second image are also images in an image sequence, so the first image and the second image have a high degree of similarity.

[0206] Step 902: Process the first image through a semantic feature extraction network to obtain semantic features of the first image. The semantic features of the first image are used to predict an image segmentation result of the first image. The semantic feature extraction network is a trained network.

[0207] In this embodiment, the network for processing reference frames and the network for processing non-reference frames are trained independently, and the network for processing reference frames is trained first, followed by the network for processing non-reference frames. Therefore, when the denoising network for processing non-reference frames is trained, the semantic feature extraction network for processing reference frames is already trained.

[0208] In addition, in some embodiments, when the first image has been processed by the semantic feature extraction network, the semantic features of the first image and the second image can be directly obtained without performing semantic feature extraction processing on the first image.

[0209] Step 903: Process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image.

[0210] After extracting semantic features of the first image using the trained semantic feature extraction network, the semantic features of the first image and the second image can be further processed using the denoising network to be trained to obtain semantic features of the second image output by the denoising network. The semantic features of the second image are the semantic features of the second image predicted by the denoising network.

[0211] Step 904: Input the second image into a semantic feature extraction network to obtain target semantic features.

[0212] Since this embodiment uses a denoising network to predict the semantic features of the second image, in order to guide the denoising network to accurately predict semantic features, this embodiment inputs the second image into a semantic feature extraction network to obtain target semantic features. The target semantic features are the semantic features obtained through the complete semantic feature extraction process, i.e., the actual semantic features of the second image.

[0213] In addition, the structures of the denoising network and the semantic feature extraction network in this embodiment are similar to those of the denoising network and the semantic feature extraction network mentioned in the above-mentioned image processing method. Please refer to the contents introduced in the above-mentioned image processing method for details, and no further details will be given here.

[0214] Step 905: Update the denoising network based on the loss function value to obtain an updated denoising network, where the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

[0215] After obtaining the semantic features of the second image predicted by the denoising network and the target semantic features extracted by the semantic feature extraction network, a loss function can be constructed based on the semantic features of the second image and the target semantic features, and then a loss function value can be calculated and the denoising network can be updated based on the loss function value. The loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features, and can represent the difference between the semantic features of the second image and the target semantic features.

[0216] During the denoising network training process, the training goal is to minimize the loss function value so that the semantic features predicted by the denoising network are as close as possible to the semantic features extracted by the semantic feature network. During the denoising network training process based on the loss function value, the backpropagation algorithm and gradient descent method can be used to gradually update the weight parameters in the denoising network, thereby achieving denoising network training.

[0217] It can be understood that the above steps 901-905 introduce a round of iterative update process in the process of training the denoising network. In actual application, the denoising network can be iteratively updated for multiple rounds, and the termination condition of the iterative update can be that the iterative update discussion of the denoising network reaches a preset number of rounds or the loss function value is not greater than a preset threshold.

[0218] In this scheme, a semantic feature extraction network for processing reference frames is pre-trained. For the denoising network used to predict the semantic features of non-reference frames, the semantic features extracted by the semantic feature extraction network and the semantic features predicted by the denoising network are used to construct a loss function, thereby realizing the training of the denoising network. Ultimately, a denoising network can be obtained that reduces the computational amount of semantic feature extraction for non-reference frames by reusing the semantic features of the reference frames.

[0219] Optionally, since the denoising network described above is only used to predict semantic features of non-reference frames, the predicted semantic features are ultimately used to predict image segmentation results. Therefore, the training goal is not only to make the semantic features predicted by the denoising network as close as possible to the semantic features extracted by the semantic feature extraction network, but also to make the final image segmentation results as close as possible to the actual segmentation results. Based on this, the loss function can be constructed from two aspects, and thus the update of the denoising network can be constrained from two aspects.

[0220] Specifically, after obtaining the semantic features of the second image predicted by the denoising network, a semantic segmentation process is performed on the semantic features of the second image using a semantic segmentation network to obtain an image segmentation result for the second image. A difference value between the image segmentation result of the second image and a true segmentation result of the second image is then determined. The true segmentation result of the second image may be a segmentation result obtained by manual labeling in advance and represents the segmentation result of the second image that is expected to be output by the network.

[0221] During the training of the denoising network, the loss function used to train the denoising network is derived based on the difference between the image segmentation result of the second image and the true segmentation result of the second image, as well as the distance between the semantic features of the second image and the target semantic features. In other words, the loss function consists of two parts: one part is derived based on the image segmentation result output by the semantic segmentation network and the true segmentation result, and the other part is derived based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network.

[0222] In this scheme, the loss function for training the denoising network is constructed from two aspects. On the one hand, it is based on the semantic features predicted by the denoising network and the semantic features actually extracted by the semantic feature extraction network. On the other hand, it is based on the image segmentation results and the real segmentation results finally output by the semantic segmentation network. This enables the denoising network to take into account the final image segmentation results during the training process, ensuring that accurate image segmentation results can be obtained after the model training.

[0223] Optionally, after obtaining the loss function value based on the above method, the semantic segmentation network used to predict the image segmentation result can be updated based on the loss function value to obtain an updated semantic segmentation network.

[0224] That is to say, the denoising network and semantic segmentation network used to process non-reference frames are trained together based on the same loss function, thereby ensuring that the denoising network and semantic segmentation network finally trained can work together to output accurate image segmentation results.

[0225] The above introduces the process of training the denoising network and semantic segmentation network for processing non-reference frames. Before training the denoising network and semantic segmentation network, the semantic feature network and semantic segmentation network for processing reference frames are also trained. The semantic feature network and semantic segmentation network for processing reference frames can be trained using the traditional training method (that is, constructing the loss function based on the image segmentation result and the real segmentation result output by the network). It will not be repeated here.

[0226] For ease of understanding, the image processing method and model training method provided in this embodiment will be described in detail below with reference to specific examples.

[0227] Taking the application of the image processing method in scenarios such as screen recording, video calls, video conferencing or live video broadcasting as an example, the terminal device used by the user (such as a smart phone, laptop or tablet) continuously acquires an image sequence and uses the image processing method provided in this embodiment to perform portrait segmentation on the images in the image sequence, thereby obtaining a portrait segmentation result, so as to further beautify the portrait or replace the background based on the portrait segmentation result.

[0228] For example, taking the video conferencing software on a laptop as an example, please refer to Figure 10, which is a schematic diagram of the system architecture for executing an image processing method provided by an embodiment of the present application. As shown in Figure 10, the conferencing software on the laptop obtains the original image through the image acquisition module. When the original image is in YUV image format, the software converts the original image into RGB image format and performs a certain degree of scaling to meet the algorithm input requirements. In the RGB image format, each pixel of the image is composed of Red, Green, and Blue, that is, the three primary colors of red, green, and blue, and the entire image is a matrix of length*width*3.

[0229] The conferencing software then feeds the processed RGB image into the portrait segmentation module, which uses image processing methods to generate a mask. The mask is a two-dimensional matrix where a 1 indicates that the corresponding pixel belongs to the portrait, while a 0 indicates that the corresponding pixel belongs to the background. After obtaining the mask, the portrait segmentation module passes it to the conferencing software for processing, such as portrait extraction and background replacement. Finally, the conferencing software uploads the processed image to the user interface for display.

[0230] When the portrait segmentation module processes RGB images, it first distinguishes whether the RGB image is a reference frame or a non-reference frame. For reference frames, the portrait segmentation module uses a semantic feature extraction network to perform complete semantic feature extraction, and further performs portrait detail prediction based on the first semantic segmentation network, ultimately obtaining the portrait segmentation result (i.e., the mask described above). For non-reference frames, the portrait segmentation module reuses the semantic features of the reference frame, predicts the semantic features of the non-reference frame through the denoising network, and further performs portrait detail prediction based on the second semantic segmentation network, ultimately obtaining the portrait segmentation result.

[0231] For example, see Figure 11, which is a schematic diagram of performing portrait segmentation processing on reference frames and non-reference frames, provided in an embodiment of the present application. As shown in Figure 11, for an image sequence requiring portrait segmentation processing, the image sequence includes a plurality of continuously acquired images, such as I0, I1, I2, I3, I4, ..., Ik. Among them, I0, I3, and Ik are reference frames, and I1, I2, and I4 are non-reference frames.

[0232] For the reference frame, the portrait segmentation result is mainly obtained through two steps: semantic estimation and detail prediction. Semantic estimation refers to the use of a semantic feature extraction network to extract the semantic features of the reference frame, and detail prediction refers to the use of the first semantic segmentation network to process the semantic features of the reference frame and predict the portrait segmentation result.

[0233] For non-reference frames, after downsampling, the semantic features of the reference frame preceding the non-reference frame and the non-reference frame itself are used to predict the semantic features of the non-reference frame (i.e., noise reduction as shown in Figure 11). After obtaining the semantic features of the non-reference frame, the second semantic segmentation network is used to perform portrait detail prediction, ultimately obtaining the portrait segmentation result for the non-reference frame.

[0234] As shown in FIG11 , for non-reference frames, after processing using the method provided in this embodiment, accurate portrait segmentation results can be obtained without performing a complete semantic feature extraction process, thus saving a large amount of computational effort.

[0235] During model training, the network for processing reference frames and the network for processing non-reference frames are trained separately. The network for processing reference frames includes a semantic feature extraction network that performs semantic estimation and a first semantic segmentation network that performs detail prediction. The network for processing non-reference frames includes a denoising network that performs denoising and a second semantic segmentation network that performs detail prediction.

[0236] The semantic feature extraction network used to process the reference frame is primarily used to extract semantic features from the reference frame to obtain rough outline information of the portrait. The semantic feature extraction network can adopt an encoder-decoder structure. Specifically, the semantic feature extraction network can include a lightweight residual network (MobileNetV2), a feature pyramid module (Pyramid Pooling Module, PPM), and two MobileV2 separable convolutions connected in sequence.

[0237] MobileNetV2 can be thought of as a feature extractor. Its input is an H*W*3 RGB image matrix, and its output is an h*w*320 feature matrix. Here, H and W are the height and width of the original image, while h and w are the height and width of the feature map. As shown in Figure 12, in the feature map output by MobileNetV2, h = H / 32 and w = W / 32.

[0238] PPM is similar to a plug-in in the network, which can improve network performance. PPM is usually placed after the feature extractor MobilenetV2. The input is a feature matrix of h*w*320, and the output feature map dimension is also h*w*320.

[0239] In the semantic feature extraction network, the input image is processed by MobileV2 to obtain 320 feature maps with a size of 1 / 32 of the original image. These feature maps are then processed by the PPM structure to obtain fused features. After the fused features are upsampled, they are subjected to two separate convolutions of MobileV2 to output 128 semantic features with a size of 1 / 16 of the original image. Among them, the semantic features are processed in two branches: one branch outputs the detailed prediction to the next stage; the other branch is processed with the Sigmoid activation function in the training stage to obtain a rough mask s p , making this stage more focused on estimating the rough mask s p , the loss function is designed as follows:

[0240] in, represents the semantic estimation loss function, G() represents Gaussian blur, α g is the target mask (i.e. the real portrait segmentation result), ‖·‖2 represents the mean square error. p is a rough mask with fuzzy edges, while the target mask α g The edge of the target mask α is clearer, so g Apply Gaussian blur to compare the rough mask s p and target mask α g .

[0241] The first semantic segmentation network, which receives the output of the semantic feature extraction network as input, further predicts details based on the semantic features obtained through semantic estimation, thereby restoring relatively fine portrait edges and details. Compared to the semantic feature extraction network, the first semantic segmentation network has fewer convolutional modules and a maximum number of channels of 128 (the maximum number of channels for the semantic feature extraction network is 320).

[0242] Specifically, the first semantic segmentation network repeats the "2x upsampling and separable convolution processing" 4 times on the semantic features output by the semantic feature extraction network to obtain the final decoding features, and then passes through the Sigmoid output layer to obtain the final mask. Among them, the intermediate features extracted from the semantic estimation are spliced ​​in the 1 / 2 size feature map in a form similar to U-NET jump connection. Separable convolution processing refers to dividing the original ordinary convolution into two steps: 1. Channel-by-channel convolution: perform k*k convolution on each channel of the original feature map (c1 in total) to obtain h*w*c1; 2. Point-by-point convolution: perform 1x1 convolution on the result of the channel-by-channel convolution, change the number of channels to c2, and obtain h*w*c2.

[0243] For example, using α d To represent the final output of the detail prediction part, the loss function of this part is as follows:

[0244] in, represents the detail prediction loss function, The first term is the absolute value loss, α d is the mask output by the first semantic segmentation network, α g is the target mask (i.e. the real portrait segmentation result), ∈ is a constant, The second term is the edge loss, Represents the Sobel edge detection operator to improve the image edge effect.

[0245] Therefore, combining the semantic estimation loss function and detail prediction loss function introduced above, it can be obtained that the reference frame loss function for training the semantic feature extraction network and the first semantic segmentation network for processing the reference frame can be composed of two parts: semantic estimation and detail prediction, as shown below:

[0246] in, represents the base frame loss function, represents the detail prediction loss function, represents the semantic estimation loss function, λ d and λ s is a hyperparameter.

[0247] That is to say, in the process of training the semantic feature extraction network and the first semantic segmentation network, the above-mentioned reference frame loss function is actually used to train the semantic feature extraction network and the first semantic segmentation network together.

[0248] The above introduces the training process of the semantic feature extraction network and the first semantic segmentation network for processing the reference frame. The following will then introduce the training process of the denoising network and the second semantic segmentation network for processing the non-reference frame.

[0249] For the tth image in the image sequence, that is, the reference frame Its semantic features and rough masks are respectively denoted as and (h and w represent the height and width of the feature map, both of which are 1 / 16 of the original image width and height, and C is the number of channels, that is, the number of features.) For the t+kth image in the image sequence, i.e., the non-reference frame I t+k , its semantic features and rough mask are recorded as F t+k and According to the analysis, and The similarity is high and the error statistics show Gaussian distribution characteristics. Therefore, as shown in Figure 13, Figure 13 is a flow chart of a process for processing non-reference frames provided by an embodiment of the present application. The problem to be solved in this solution is how to design a function or filter f so that the filtered result As close to the real features as possible

[0250] Here, the softmax function φ(·) is used to convert the feature matrix into a probabilistic random field, as shown below.

[0251] It should be noted that because the neural network training process is the process of fitting continuous functions in the real domain, the benefits of converting the feature matrix into a probabilistic random field are: first, it preserves the original matrix information to the greatest extent possible from a probabilistic perspective; second, it eliminates discontinuities, i.e., elements in the matrix with excessively large or small absolute values, thus preventing them from affecting the neural network parameter fitting.

[0252] The problem to be solved by this solution is transformed into a supervised conditional probability distribution generation problem. This filter f(·) can be used with the neural network f θ (·) to approach.

[0253] For example, please refer to Figure 13, which is a schematic diagram of the structure of the noise reduction network provided in an embodiment of the present application. As shown in Figure 13, the noise reduction network includes a Softmax function, a convolution module 1, a convolution module 2, a 1*1 convolution layer (Conv), a BN function, and a ReLU function. Among them, the convolution module 1 and the convolution module 2 can adopt the same structure. In Figure 13, the convolution module 1 includes a 1*1 convolution layer, a BN function, a ReLU function, a 5*5 grouped convolution with an expansion rate of 2, a SE module, a ReLU function, a 1*1 convolution layer, and a linear layer (Linear). Using convolution module 1 and convolution module 2, the receptive field can be expanded to the entire feature map at a scale of 1 / 16.

[0254] In addition, after the semantic features of the non-reference frame are predicted by the denoising network, the denoising loss function can be calculated based on the semantic features predicted by the denoising network and the semantic features of the non-reference frame extracted by the semantic feature extraction network. Specifically, the KL divergence can be used to measure the error between the predicted distribution and the true distribution, as shown in the following formula:

[0255] in, represents the noise reduction loss function; φ(F t+k ) represents the semantic features output by the semantic feature extraction network; Represents the semantic features predicted by the denoising network; KL() represents the KL divergence, which is a metric used to measure the similarity between two probability distributions.

[0256] Furthermore, the semantic features output by the denoising network are input into the second semantic segmentation network for detail prediction, thereby obtaining the mask of the non-reference frame. Similarly, referring to the above-mentioned detail prediction loss function, the detail prediction loss function corresponding to the non-reference frame can also be constructed based on the mask output by the second semantic segmentation network and the true mask of the non-reference frame. Then, based on the above-mentioned denoising loss function and the detail prediction loss function corresponding to the non-reference frame, the final non-reference frame loss function is obtained, as shown in the following formula:

[0257] in, represents the non-reference frame loss function, represents the detail prediction loss function corresponding to the non-reference frame, represents the denoising loss function, λ d and λ s is a hyperparameter.

[0258] That is to say, in the process of training the denoising network and the second semantic segmentation network, the denoising network and the second semantic segmentation network are actually trained together using the above-mentioned non-reference frame loss function.

[0259] For example, please refer to Figure 14, which is a comparative schematic diagram of processing a reference frame and a non-reference frame provided in an embodiment of the present application. As shown in Figure 14, the process of processing the reference frame includes two steps: semantic estimation and detail prediction, wherein the semantic estimation takes 4ms and the detail prediction takes 2.7ms; the process of processing the non-reference frame also includes two steps: semantic estimation and detail prediction, and the semantic estimation process of processing the non-reference frame reuses the semantic features of the reference frame, so the semantic estimation of the non-reference frame takes 1ms and the detail prediction takes 2.7ms. Obviously, in the process of predicting the semantic features of the non-reference frame, since the semantic features of the reference frame are reused, there is no need to fully perform a semantic feature extraction process, which can save a lot of calculation amount and calculation time.

[0260] The above embodiments introduce the methods provided by the embodiments of the present application, and the following will introduce the devices for executing the above methods.

[0261] Please refer to Figure 15, which is a structural diagram of an image processing device provided by an embodiment of the present application. As shown in Figure 15, the image processing device includes: an acquisition module 1501, which is used to acquire a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is located after the first image; a processing module 1502, which is used to process the first image through a semantic feature extraction network to obtain the semantic features of the first image, and the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; the processing module 1502 is also used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; the processing module 1502 is also used to input the second image into the semantic feature extraction network to obtain the target semantic features; the processing module 1502 is also used to update the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0262] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold.

[0263] In a possible implementation, in the image sequence, the number of images spaced between the first image and the second image is smaller than a second threshold.

[0264] In one possible implementation, the processing module 1502 is specifically used to: perform a first-order feature extraction process on the first image through a semantic feature extraction network; perform a second-order feature extraction process on the second image to obtain the original features of the second image, where the second order is smaller than the first order; process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0265] In a possible implementation, the processing module 1502 is specifically configured to: concatenate the semantic features of the first image and the original features of the second image to obtain concatenated features; and input the concatenated features into a denoising network to predict the semantic features of the second image.

[0266] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0267] In one possible implementation, the processing module 1502 is further used to: process the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; process the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; wherein the first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

[0268] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0269] Please refer to Figure 16, which is a structural diagram of a model training device provided by an embodiment of the present application. As shown in Figure 16, the model training device includes: an acquisition module 1601, which is used to acquire the semantic features of a first image and a second image, the second image and the first image are in the same image sequence, and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict the image segmentation result of the first image, and the semantic feature extraction network is a trained network; a processing module 1602, which is used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; the processing module 1602 is also used to input the second image into the semantic feature extraction network to obtain the target semantic features; the processing module 1602 is also used to update the denoising network based on the loss function value to obtain an updated denoising network, and the loss function value is obtained based on the distance between the semantic features of the second image and the target semantic features.

[0270] In a possible implementation, the similarity between the first image and the second image is not less than a first threshold.

[0271] In a possible implementation, in the image sequence, the number of images spaced between the first image and the second image is smaller than a second threshold.

[0272] In one possible implementation, the processing module 1602 is further used to: perform semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determine a difference value between the image segmentation result of the second image and a true segmentation result of the second image, where the true segmentation result of the second image is pre-labeled; wherein the loss function value is obtained based on the difference value and the distance between the semantic features of the second image and the target semantic features.

[0273] In a possible implementation, the processing module 1602 is further configured to update the semantic segmentation network based on the loss function value to obtain an updated semantic segmentation network.

[0274] In one possible implementation, the semantic features of the first image are obtained by performing a first-order feature extraction process on the first image through a semantic feature extraction network; the processing module 1602 is specifically used to: perform a second-order feature extraction process on the second image to obtain the original features of the second image, where the second order is smaller than the first order; and process the semantic features of the first image and the original features of the second image through a denoising network to predict the semantic features of the second image.

[0275] In a possible implementation, the processing module 1602 is specifically configured to: concatenate the semantic features of the first image and the original features of the second image to obtain concatenated features; and input the concatenated features into a denoising network to predict the semantic features of the second image.

[0276] In one possible implementation, the denoising network includes a convolutional neural network or an attention network.

[0277] In a possible implementation, the image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

[0278] Next, an execution device provided in an embodiment of the present application is introduced. Please refer to Figure 17. Figure 17 is a structural diagram of an execution device provided in an embodiment of the present application. The execution device 1700 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, a server, etc., which is not limited here. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703 and a memory 1704 (wherein the number of processors 1703 in the execution device 1700 can be one or more, and Figure 17 takes one processor as an example), wherein the processor 1703 may include an application processor 17031 and a communication processor 17032. In some embodiments of the present application, the receiver 1701, the transmitter 1702, the processor 1703 and the memory 1704 may be connected via a bus or other means.

[0279] Memory 1704 may include read-only memory and random access memory, and provides instructions and data to processor 1703. A portion of memory 1704 may also include non-volatile random access memory (NVRAM). Memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0280] Processor 1703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0281] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1703. Processor 1703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1703. The above processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1703 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1704, and processor 1703 reads the information in memory 1704 and, in conjunction with its hardware, completes the steps of the above method.

[0282] Receiver 1701 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1702 can be used to output digital or character information through the first interface. Transmitter 1702 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1702 can also include a display device such as a display screen.

[0283] The execution device provided in the embodiment of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the method for determining the model structure described in the above embodiment, or so that the chip in the training device executes the method for determining the model structure described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0284] Specifically, see Figure 18 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1800. NPU 1800 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1803, which is controlled by controller 1804 to extract matrix data from memory and perform multiplication operations.

[0285] In some implementations, the arithmetic circuit 1803 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1803 is a two-dimensional systolic array. The arithmetic circuit 1803 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1803 is a general-purpose matrix processor.

[0286] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1802 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1801 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1808.

[0287] Unified memory 1806 is used to store input and output data. Weight data is directly transferred to weight memory 1802 through the Direct Memory Access Controller (DMAC) 1805. Input data is also transferred to unified memory 1806 through the DMAC.

[0288] BIU stands for Bus Interface Unit 1810 , which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1809 .

[0289] The bus interface unit 1810 (BIU) is used for the instruction fetch memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0290] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1806 or to move weight data to the weight memory 1802 or to move input data to the input memory 1801.

[0291] The vector calculation unit 1807 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1803, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0292] In some implementations, the vector calculation unit 1807 can store the processed output vector in the unified memory 1806. For example, the vector calculation unit 1807 can apply a linear function or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1807 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1803, for example, for use in subsequent layers in a neural network.

[0293] An instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;

[0294] Unified memory 1806, input memory 1801, weight memory 1802, and instruction fetch memory 1809 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0295] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0296] Please refer to Figure 19, which is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. The present application also provides a computer-readable storage medium. In some embodiments, the method disclosed in Figure 6 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or products.

[0297] 19 schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0298] In one embodiment, computer-readable storage medium 1900 is provided using signal-bearing medium 1901. Signal-bearing medium 1901 may include one or more program instructions 1902 that, when executed by one or more processors, may provide the functionality or portions of the functionality described above with respect to FIG. 3. Furthermore, program instructions 1902 in FIG. 19 also depict example instructions.

[0299] In some examples, signal bearing medium 1901 may include computer readable medium 1903 such as, but not limited to, a hard drive, compact disk (CD), digital video disk (DVD), digital tape, memory, ROM or RAM, and the like.

[0300] In some embodiments, the signal-bearing medium 1901 may include a computer-recordable medium 1904, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, or the like. In some embodiments, the signal-bearing medium 1901 may include a communication medium 1905, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, or the like). Thus, for example, the signal-bearing medium 1901 may be communicated via a wireless form of the communication medium 1905 (e.g., a wireless communication medium conforming to the IEEE 802 standard or other transmission protocol).

[0301] The one or more program instructions 1902 may be, for example, computer-executable instructions or logic-implemented instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1902 communicated to the computing device via one or more of computer-readable media 1903, computer-recordable media 1904, and / or communication media 1905.

[0302] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0303] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0304] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0305] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0306] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0307] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0308] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0309] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is after the first image; Processing the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict an image segmentation result of the first image; The semantic features of the first image and the second image are processed by a denoising network to predict the semantic features of the second image, and the semantic features of the second image are used to obtain an image segmentation result of the second image.

2. The method according to claim 1, characterized in that The similarity between the first image and the second image is not less than a first threshold.

3. The method according to claim 1 or 2, characterized in that: In the image sequence, the number of images between the first image and the second image is less than a second threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The processing of the first image by using a semantic feature extraction network includes: Performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The step of processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image includes: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

5. The method according to claim 4, characterized in that The processing of the semantic features of the first image and the original features of the second image by the denoising network to predict the semantic features of the second image includes: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

6. The method according to any one of claims 1 to 5, characterized in that: The denoising network includes a convolutional neural network or an attention network.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Processing the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; Processing the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; The first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

8. The method according to any one of claims 1 to 7, characterized in that: The image segmentation result of the first image and the image segmentation result of the second image are both portrait segmentation results, and the image segmentation result of the first image and the image segmentation result of the second image are both used to perform background replacement of the portrait.

9. A model training method, characterized in that: include: Acquire semantic features of a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is located after the first image, the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, the semantic features of the first image are used to predict an image segmentation result of the first image, and the semantic feature extraction network is a trained network; Processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; Inputting the second image into the semantic feature extraction network to obtain target semantic features; Based on the loss function value, the denoising network is updated to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

10. The method according to claim 9, characterized in that The similarity between the first image and the second image is not less than a first threshold.

11. The method according to claim 9 or 10, characterized in that: In the image sequence, the number of images between the first image and the second image is less than a second threshold.

12. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; Determine a difference value between an image segmentation result of the second image and a true segmentation result of the second image, where the true segmentation result of the second image is pre-marked; The loss function value is obtained based on the difference value and the distance between the semantic feature of the second image and the target semantic feature.

13. The method according to any one of claims 9 to 12, characterized in that: The semantic feature of the first image is obtained by performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The step of processing the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image includes: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

14. The method according to claim 13, characterized in that The processing of the semantic features of the first image and the original features of the second image by the denoising network to predict the semantic features of the second image includes: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

15. An image processing device, characterized in that: include: an acquisition module, used for acquiring a first image and a second image, wherein the second image and the first image are in the same image sequence and the second image is after the first image; A processing module, configured to process the first image through a semantic feature extraction network to obtain semantic features of the first image, wherein the semantic features of the first image are used to predict an image segmentation result of the first image; The processing module is further used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image, and the semantic features of the second image are used to obtain an image segmentation result of the second image.

16. The device according to claim 15, characterized in that The similarity between the first image and the second image is not less than a first threshold.

17. The device according to claim 15 or 16, characterized in that In the image sequence, the number of images between the first image and the second image is less than a second threshold.

18. The device according to any one of claims 15 to 17, characterized in that: The processing module is specifically used for: Performing a first number of feature extraction processes on the first image by the semantic feature extraction network; Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

19. The device according to claim 18, characterized in that The processing module is specifically used for: splicing the semantic features of the first image and the original features of the second image to obtain a spliced ​​feature; The splicing features are input into the denoising network to predict the semantic features of the second image.

20. The device according to any one of claims 15 to 19, characterized in that: The denoising network includes a convolutional neural network or an attention network.

21. The device according to any one of claims 15 to 20, characterized in that: The processing module is further used for: Processing the semantic features of the first image through a first semantic segmentation network to obtain an image segmentation result of the first image; Processing the semantic features of the second image through a second semantic segmentation network to obtain an image segmentation result of the second image; The first semantic segmentation network and the second semantic segmentation network have the same network structure, and the first semantic segmentation network and the second semantic segmentation network have different weight parameters.

22. A model training device, characterized in that: include: an acquisition module, used to acquire semantic features of a first image and a second image, wherein the second image and the first image are located in the same image sequence and the second image is located after the first image, wherein the semantic features of the first image are obtained by processing the first image through a semantic feature extraction network, wherein the semantic features of the first image are used to predict an image segmentation result of the first image, and wherein the semantic feature extraction network is a trained network; The processing module is used to process the semantic features of the first image and the second image through a denoising network to predict the semantic features of the second image; The processing module is further used to input the second image into the semantic feature extraction network to obtain target semantic features; The processing module is also used to update the denoising network based on a loss function value to obtain an updated denoising network, wherein the loss function value is obtained based on the distance between the semantic feature of the second image and the target semantic feature.

23. The device according to claim 22, characterized in that The processing module is further used for: Performing semantic segmentation processing on the semantic features of the second image through a semantic segmentation network to obtain an image segmentation result of the second image; determining a difference value between an image segmentation result of the second image and a true segmentation result of the second image; The loss function value is obtained based on the difference value and the distance between the semantic feature of the second image and the target semantic feature.

24. The device according to any one of claims 22-23, characterized in that The semantic feature of the first image is obtained by performing a first number of feature extraction processes on the first image by the semantic feature extraction network; The processing module is further used for: Performing feature extraction processing of a second number of times on the second image to obtain original features of the second image, the second number of times being less than the first number of times; The semantic features of the first image and the original features of the second image are processed by the denoising network to predict the semantic features of the second image.

25. An image processing device, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 8.

26. A model training device, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 9 to 14.

27. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 14.

28. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 14.

Images