An image segmentation model, method, device and terminal equipment

Abstract

The application is suitable for the technical field of image processing, and provides an image segmentation model, method, device and terminal equipment, the model comprising: an encoder for extracting semantic features of a current frame; a decoder for performing image segmentation according to the semantic features of the current frame and historical features stored in the decoder; at least one decoder network layer of the decoder comprises an inter-frame feature transmission module; the inter-frame feature transmission module is used for obtaining fused features according to the semantic features of the current frame and the historical features, and storing the fused features as new historical features in the inter-frame feature transmission module. Since the above scheme fully utilizes the continuity between adjacent frame images in video data, image segmentation processing is performed through the semantic features of the current frame and the historical features. Since there is continuity between the semantic features of the current frame and the historical features, the occurrence of flickering or jitter in the image segmentation result can be well inhibited.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and in particular relates to an image segmentation model, method, apparatus and terminal device. Background Technology

[0002] Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and extracting targets of interest. It is a key step from image processing to image analysis. Existing image segmentation methods can be mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories. From a mathematical perspective, image segmentation is the process of dividing a digital image into non-overlapping regions. The image segmentation process is also a labeling process, that is, assigning the same number to pixels belonging to the same region.

[0003] However, when segmenting video data, the results obtained by traditional image segmentation methods often exhibit flickering or jitter, which is a technical problem that urgently needs to be solved.

[0004] Application content

[0005] In view of this, embodiments of this application provide an image segmentation model, method, apparatus, and terminal device to solve the technical problem that image segmentation results obtained by traditional image segmentation methods often exhibit flickering or jitter.

[0006] A first aspect of this application provides an image segmentation model, comprising: an encoder for acquiring the current frame in video data and extracting semantic features of the current frame; a decoder for performing image segmentation based on the semantic features of the current frame and historical features stored in the decoder; the decoder includes multiple decoder network layers, at least one decoder network layer including an inter-frame feature transfer module; and the inter-frame feature transfer module for obtaining fused features based on the semantic features of the current frame and historical features, and storing the fused features as new historical features in the inter-frame feature transfer module.

[0007] Furthermore, the inter-frame feature transfer module is used to divide the input features into two parts to obtain sub-features x and y; to concatenate the historical features and sub-features x to obtain fused features, and to store the fused features as new historical features in the inter-frame feature transfer module; to concatenate the fused features and sub-features y to obtain inter-frame features, and to pass the inter-frame features to the next decoder network layer or to perform image segmentation based on the inter-frame features.

[0008] A second aspect of this application provides an image segmentation method, comprising: acquiring the current frame in video data and extracting semantic features of the current frame; preprocessing and segmenting the semantic features to obtain sub-features x and y; fusing historical features, sub-features x and y to obtain inter-frame features, and storing the fused features of historical features and sub-features x as new historical features; and performing image segmentation on the current frame based on the inter-frame features to obtain an image segmentation result.

[0009] Furthermore, the historical features, sub-features x and y are fused to obtain inter-frame features, and the fused features of the historical features and sub-features x are stored as new historical features. This includes: fusing the historical features and sub-features x to obtain fused features and storing the fused features as new historical features; and fusing the fused features and sub-features y to obtain inter-frame features.

[0010] A third aspect of this application provides an image segmentation apparatus, comprising: an extraction unit for acquiring the current frame in video data and extracting semantic features of the current frame; a segmentation unit for preprocessing and segmenting the semantic features to obtain sub-features x and y; a fusion unit for fusing historical features, sub-features x and y to obtain inter-frame features, and storing the fused features of historical features and sub-features x as new historical features; and a segmentation unit for segmenting the current frame according to the inter-frame features to obtain an image segmentation result.

[0011] A fourth aspect of this application provides a training method for an image segmentation model of the first aspect, comprising: acquiring a plurality of data sets with labeled information as a training sample set; wherein the data sets include historical frames and current frames; inputting historical frames into the image segmentation model to be trained to obtain historical features and storing them in the image segmentation model to be trained; inputting the current frame into the image segmentation model to be trained storing historical features to obtain segmentation results of the data sets; calculating loss function values ​​based on the segmentation results of the data sets and the labeled information of the data sets; and iteratively optimizing the image segmentation model to be trained based on the loss function values ​​to obtain a target image segmentation model.

[0012] The fifth aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image segmentation method described in the second aspect or the training method described in the fourth aspect.

[0013] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image segmentation method described in the second aspect or the training method described in the fourth aspect.

[0014] The beneficial effects of this application's embodiments compared to existing technologies are as follows: The model adopted in this application includes: an encoder for acquiring the current frame in video data and extracting the semantic features of the current frame; a decoder for performing image segmentation based on the semantic features of the current frame and historical features stored in the decoder; the decoder includes multiple decoder network layers, at least one of which includes an inter-frame feature transfer module; the inter-frame feature transfer module is used to obtain fused features based on the semantic features of the current frame and historical features, and stores the fused features as new historical features in the inter-frame feature transfer module. Because the above scheme fully utilizes the continuity between adjacent frames in video data, it performs image segmentation processing through the semantic features of the current frame and historical features. Since there is continuity between the semantic features of the current frame and historical features, it can effectively suppress flickering or jitter in the image segmentation results. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of the network architecture of the target image segmentation model provided in this application is shown;

[0017] Figure 2 A schematic diagram of the network layer mapping relationship provided in this application is shown;

[0018] Figure 3 A schematic diagram illustrating the processing flow of multiple decoder network layers provided in this application is shown.

[0019] Figure 4 A schematic flowchart of an image segmentation method provided by the present invention is shown;

[0020] Figure 5 A schematic flowchart of step 403 in an image segmentation method provided in this application is shown;

[0021] Figure 6 A schematic diagram illustrating the specific processing flow of the decoder network layer provided in this application is shown;

[0022] Figure 7 A schematic flowchart of another image segmentation method provided in this application is shown;

[0023] Figure 8 A schematic diagram of an image segmentation apparatus provided in this application is shown;

[0024] Figure 9 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] The image segmentation method provided in this application can be applied to different types of image recognition models. In order to better explain the technical solution of this application, this application takes a semantic recognition model as an example for explanation and illustration.

[0027] Semantic segmentation refers to using a semantic segmentation model to perform semantic recognition on images or video frames, and then predicting the category of each object in the image based on the recognition results. Videos are composed of still images, called frames or video frames. When using a semantic segmentation model to perform semantic segmentation on images or video frames, the model can perform semantic recognition on each pixel in the image or video frame, predict the category of each pixel based on the semantic recognition results, and generate a semantically segmented image. The semantically segmented image is used to classify each pixel in the video frame, realizing semantic annotation of the video frame. In other words, a semantically segmented image includes one or more target regions segmented by semantic recognition. The same target region corresponds to pixels of the same predicted category, and is generally labeled with the same identifier (e.g., color). Different target regions correspond to pixels of different predicted categories, and are generally labeled with different identifiers (e.g., color).

[0028] In this application, the semantic segmentation model is described as an encoder-decoder structure, which can be understood as a type of convolutional neural network model. The semantic segmentation model can include an input layer, multiple network layers, and an output layer. Each network layer in the semantic segmentation model is used to extract features from the input image or feature map and output a feature map.

[0029] Please see Figure 1 , Figure 1A schematic diagram of the network architecture of the target image segmentation model provided in this application is shown. In this embodiment, the target image segmentation model is a semantic segmentation model, including an encoder and a decoder. The current frame image is input into the encoder of the target image segmentation model, and undergoes convolution processing, downsampling processing, batch normalization processing, and activation function processing. It then undergoes convolution processing, batch normalization processing, and activation function processing through multiple basic blocks to obtain semantic features. These semantic features are then input into the decoder for image segmentation processing.

[0030] In this embodiment, the target image segmentation model performs image segmentation on video data, which refers to a continuous sequence of image frames, essentially composed of a set of consecutive images. As for the images themselves, apart from their order of appearance, they contain no structural information. However, the dynamic changes between adjacent frames are continuous, meaning there is a certain correlation between the current frame and the previous frame. Therefore, this application utilizes this natural law, performing image segmentation processing based on the historical features of the previous frame and the semantic features of the current frame.

[0031] In this embodiment, the encoder includes a first encoder network layer and multiple second encoder network layers. The first encoder network layer includes a first convolution module, a first downsampling module, a first batch normalization module, and a first activation function module. The second encoder network layers include multiple basic blocks; each basic block includes a single second downsampling module and multiple convolutional blocks, each convolutional block including a second convolution module, a second batch normalization module, and a second activation function module.

[0032] The first and second convolution modules are used to obtain feature data based on convolution calculations. Convolution is a mathematical operation that generates a third function from two functions f and g. Essentially, it is a special integral transformation that represents the integral of the product of the overlapping function values ​​of functions f and g after flipping and translation with respect to the overlap length.

[0033] The first and second downsampling modules are used to reduce the number of sampling points in the matrix, thereby reducing the amount of data in the image.

[0034] The first and second batch normalization modules are used to perform batch normalization on the feature data. Batch normalization (BatchNorm) aims to standardize the outputs of intermediate layers in a neural network, making them more stable. Normalization of neural network data typically results in a sample dataset with a mean of 0 and a variance of 1. This is because a relatively fixed distribution of input data is beneficial for algorithm stability and convergence. However, for deep neural networks, since parameters are constantly updated, even after standardization, the input received by later layers still varies drastically, often leading to numerical instability and difficulty in model convergence. Batch normalization makes the outputs of intermediate layers more stable and has three advantages: it allows for faster learning (allowing the use of a larger learning rate); it reduces the model's sensitivity to initial values; and it suppresses overfitting to some extent. The main idea of ​​batch normalization is to normalize the values ​​of neurons during training, ensuring that the data distribution satisfies a mean of 0 and a variance of 1.

[0035] The first and second activation function modules are used to transform the hidden variables using element-wise non-linear functions before using them as input to the next fully connected layer. Activation functions introduce non-linear operations into the neural network, allowing neurons to be applied to non-linear functions. Because convolution operates linearly on the input image, but the information in the input image is not always linearly separable, non-linear operations using activation functions can better map features, remove redundancy from the data, and enhance the expressive power of the convolutional neural network. Commonly used activation functions in convolutional neural networks include sigmoid, tanh, ReLU, and Leaky ReLU.

[0036] In this embodiment, the decoder includes multiple decoder network layers, each comprising a third convolutional module, an upsampling module, a third batch normalization module, and a third activation function module. One or more decoder network layers may also include an inter-frame feature transfer module; that is, each decoder network may or may not include an inter-frame feature transfer module as needed. The inter-frame feature transfer module is used to fuse feature data. The upsampling module is used for interpolation processing, which can be simply understood as enlarging the image and increasing the number of sampling points in the matrix.

[0037] The inter-frame feature transfer module is used to: divide the input features into two parts to obtain sub-features x and y; concatenate the historical features and sub-features x to obtain fused features, and store the fused features as new historical features in the inter-frame feature transfer module; concatenate the fused features and sub-features y to obtain inter-frame features, and pass the inter-frame features to the next decoder network layer or perform image segmentation based on the inter-frame features.

[0038] When the semantic features output by the encoder are input into the decoder, multiple decoder network layers in the decoder perform convolution, batch normalization, activation function processing, and upsampling on the semantic features step by step. If the decoder network layer includes an inter-frame feature transfer module, this module fuses the feature data output from the previous network layer with the historical features corresponding to the current network layer. After step-by-step processing by multiple decoder network layers, the last decoder network layer outputs the image segmentation result.

[0039] Please see Figure 2 , Figure 2 A schematic diagram of the network layer mapping relationship provided in this application is shown. In this embodiment, the preset mapping relationship between multiple encoder network layers and multiple decoder network layers of the target image segmentation model is a head-to-tail correspondence. Encoder network layer A1 corresponds to encoder network layer B5, encoder network layer A2 corresponds to encoder network layer B4, encoder network layer A3 corresponds to encoder network layer B3, encoder network layer A4 corresponds to encoder network layer B2, and encoder network layer A5 corresponds to encoder network layer B1. It is worth noting that... Figure 2 Just as an example, for Figure 2 There are no restrictions on the network layer structure, number of network layers, input data, or output data. In practical applications, the number of network layers can be more or less.

[0040] Please see Figure 3 , Figure 3 This diagram illustrates the processing flow of multiple decoder network layers provided in this application. Network layer B1 is the decoder network layer for the current node, and network layers B2 through B... nThis is the decoder network layer for subsequent nodes. In network layer B1, the semantic features are segmented into a first semantic feature x1 and a first semantic feature y1. The first historical sub-feature data h1 corresponding to network layer B1 is obtained. The first semantic feature x1 and the first historical sub-feature data h1 are fused to obtain the first fused feature data hx1. The first fused feature data hx1 and the first semantic feature y1 are concatenated along the channel dimension to obtain the feature data processed by network layer B1, and this feature data is sent to the next stage of the decoder. In network layer B2, the feature data processed by network layer B1 is segmented into a second semantic feature x2 and a second semantic feature y2. The second historical sub-feature data h2 corresponding to network layer B2 is obtained. The second semantic feature x2 and the second historical sub-feature data h2 are fused to obtain the second fused feature data hx2. The second fused feature data hx2 and the second semantic feature y2 are concatenated along the channel dimension to obtain the feature data processed by network layer B2, and this feature data is sent to the next stage of the decoder. In network layer B3, the feature data processed by network layer B2 is segmented into a third semantic feature x3 and a third semantic feature y3. Network layer B3 to network layer B n The processing logic follows the same pattern. It should be noted that the first semantic feature x1 and the first historical sub-feature data h1 are fused to obtain the first fused feature data hx1. The first fused feature data hx1 will be concatenated with the first semantic feature y1 in the channel dimension for output. The first fused feature data hx1 will also be used as the first historical sub-feature data h1 in the next frame.

[0041] The image segmentation model used in this embodiment includes an encoder and a decoder. The decoder includes multiple decoder network layers, at least one of which includes an inter-frame feature transfer module. The encoder is used to acquire the current frame in the video data and extract the semantic features of the current frame. The decoder is used to perform image segmentation based on the semantic features of the current frame and the historical features stored in the inter-frame feature transfer module. The inter-frame feature transfer module is used to obtain fused features based on the semantic features of the current frame and the historical features, and stores the fused features as new historical features in the inter-frame feature transfer module. Because the above scheme fully utilizes the continuity between adjacent frames in the video data, it performs image segmentation processing through the semantic features of the current frame and the historical features. Due to the continuity between the semantic features of the current frame and the historical features, it can effectively suppress flickering or jitter in the image segmentation results.

[0042] Please see Figure 4 , Figure 4 A schematic flowchart of an image segmentation method provided by the present invention is shown. The image segmentation method may include the following steps:

[0043] Step 401: Obtain the current frame from the video data and extract the semantic features of the current frame.

[0044] Video data, as we understand it, refers to a continuous sequence of frame images, essentially composed of a series of consecutive images. However, the images themselves, aside from their order of appearance, lack any structural information. The dynamic changes between adjacent frames are continuous, meaning there is a certain correlation between the current frame and the previous frame. Therefore, this application utilizes this natural law, performing image segmentation processing through the historical features of the previous frame and the semantic features of the current frame. Here, the current frame refers to the image currently undergoing segmentation, and the previous frame is the frame that precedes the current frame ("previous" refers to the temporal sequence).

[0045] In this embodiment, the target image segmentation model is an encoder and decoder structure. The encoder is used to extract semantic features and input the semantic features into the decoder. The decoder extracts target feature data based on the semantic features, and the target feature data is used to perform image segmentation processing on the current frame.

[0046] Step 402: Preprocess and segment the semantic features to obtain sub-features x and y.

[0047] In order to preserve the feature information of semantic features, this application divides semantic features into sub-features x and sub-features y, and performs different fusion processing on sub-features x and sub-features y respectively, so as to take advantage of the correlation between the current frame and the previous frame and improve the image segmentation effect.

[0048] Step 403: The historical features, sub-feature x, and sub-feature y are fused to obtain inter-frame features, and the fused features of the historical features and sub-feature x are stored as new historical features.

[0049] Since the frames in the video data are input into the target image segmentation model one by one according to their temporal relationship during image segmentation, the historical features of the previous frame output by the encoder are already obtained before step 403. Therefore, when executing step 403, the pre-stored historical features can be directly retrieved from the memory.

[0050] The number of historical features from the previous frame can be one or more. The number of historical features is determined by the number of encoder network layers; that is, after the previous frame is input into the encoder, one or more encoder network layers output historical features. The encoder network layers acquire their corresponding historical features.

[0051] Specifically, step 403 includes steps 4031 to 4032. Please refer to [link / reference]. Figure 5 , Figure 5 A schematic flowchart of step 403 in an image segmentation method provided in this application is shown.

[0052] Step 4031: The historical features and sub-features x are fused to obtain fused features, and the fused features are stored as new historical features.

[0053] Among them, the fused features are used as historical features of the next frame for feature fusion processing, and the processing process is similar to steps 4031 to 4032.

[0054] Step 4032: The fused features and sub-features y are fused to obtain inter-frame features.

[0055] This application divides semantic features into two parts (sub-feature x and sub-feature y), and then merges them in stages and levels, which can reduce the amount of computation while ensuring accuracy.

[0056] Step 404: Perform image segmentation on the current frame based on inter-frame features to obtain the image segmentation result.

[0057] For a better explanation of the specific processing flow of the decoder network layer, please refer to [link / reference]. Figure 6 , Figure 6 A schematic diagram illustrating the specific processing flow of the decoder network layer provided in this application is shown. Figure 6 As shown, the first decoder network layer performs convolution (Conv), batch normalization (BatchNormalization), and ReLU activation on feature data 1 to obtain feature data 2. The first decoder network layer then splits feature data 2 using a split function to obtain feature data 3 and feature data 4. The first decoder network layer then merges feature data 3 with historical feature data h using a concat function to obtain feature data 5. The first decoder network layer performs convolution (Conv) and TURN activation on feature data 5 to obtain feature data 6. The first decoder network layer then merges feature data 6 with feature data 4 using a concat function to obtain feature data 7. The first decoder network layer inputs feature data 7 into the second decoder network layer, and the second decoder network layer repeats the above process, and so on, until the last decoder network layer.

[0058] This embodiment obtains the current frame from video data and extracts its semantic features. The semantic features are preprocessed and segmented to obtain sub-features x and y. Historical features, sub-features x and y are fused to obtain inter-frame features, and the fused features of historical features and sub-features x are stored as new historical features. Image segmentation is then performed on the current frame based on the inter-frame features to obtain the image segmentation result. Because this scheme fully utilizes the continuity between adjacent frames in the video data, it performs image segmentation using the semantic features of the current frame and historical features. Since there is continuity between the semantic features of the current frame and historical features, it can effectively suppress flickering or jitter in the image segmentation result.

[0059] Optionally, this application also provides a method for training an image segmentation model. See [link to relevant documentation]. Figure 7 , Figure 7 A schematic flowchart of another image segmentation method provided in this application is shown.

[0060] Step 701: Obtain several data sets with labeled information as a training sample set; wherein, the data sets include historical frames and the current frame.

[0061] The goal of training an image segmentation model is to make its output as close as possible to the desired predicted value. This is achieved by comparing the current model's predicted value with the target value, and then updating the weight vector of each layer based on the difference (usually an initialization process occurs before the first update, pre-configuring parameters for each layer). (For example, if the predicted value is too high, the weight vector is adjusted to predict a lower value; this adjustment continues until the model can predict the desired target value.) Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the image segmentation model becomes a process of minimizing this loss. Generally, an image segmentation model can be considered trained successfully when it meets preset constraints. These constraints could be reaching a preset number of iterations or achieving preset performance metrics after parameter adjustments.

[0062] The training sample set includes multiple video datasets, each with adjacent image frames. The annotation information represents the correct image segmentation results.

[0063] Optionally, this application can also train the model using independent, discontinuous single-frame images. Specifically, it generates the previous frame data, i.e., simulated historical frames, by performing data augmentation such as translation, scaling, rotation, or Gaussian blur on the single-frame images. Multiple single-frame images and their corresponding historical frames are then used as the training sample set.

[0064] Step 702: Input historical frames into the image segmentation model to be trained, obtain historical features, and store them in the image segmentation model to be trained.

[0065] Step 703: Input the current frame into the image segmentation model to be trained, which stores historical features, to obtain the segmentation result of the data set.

[0066] The current frame is processed by the image segmentation model to be trained to obtain semantic features. These semantic features are then preprocessed and segmented to obtain sub-features x and y. Historical features and sub-features x are fused to obtain fused features, which are stored as new historical features. The fused features and sub-features y are then fused to obtain inter-frame features. Based on these inter-frame features, the current frame is segmented to obtain the segmentation result.

[0067] Step 704: Calculate the loss function value based on the data group segmentation results and the data group annotation information.

[0068] Step 705: Iteratively optimize the image segmentation model to be trained based on the loss function value to obtain the target image segmentation model.

[0069] The network parameters of the image segmentation model to be trained are adjusted according to the multiple loss function values ​​corresponding to the sample data in the data set to obtain the target image segmentation model.

[0070] In this embodiment, by pre-training the image segmentation model to be trained, a high-precision target image segmentation model is obtained, which can effectively suppress flickering or jitter in the image segmentation results.

[0071] like Figure 8 This application provides an image segmentation apparatus 8, please refer to... Figure 8 , Figure 8 A schematic diagram of an image segmentation apparatus provided in this application is shown, such as... Figure 8 An image segmentation device shown includes:

[0072] The first acquisition unit 81 is used as an extraction unit to acquire the current frame in the video data and extract the semantic features of the current frame.

[0073] Segmentation unit 82 is used for preprocessing and segmenting semantic features to obtain sub-features x and y;

[0074] The fusion unit 83 is used to fuse historical features, sub-feature x and sub-feature y to obtain inter-frame features, and store the fused features of historical features and sub-feature x as new historical features.

[0075] The segmentation unit 84 is used to segment the current frame based on inter-frame features to obtain the image segmentation result.

[0076] This application provides an image segmentation apparatus that acquires the current frame from video data and extracts its semantic features. The semantic features are preprocessed and segmented to obtain sub-features x and y. Historical features, sub-features x, and sub-features y are fused to obtain inter-frame features, and the fused features of historical features and sub-features x are stored as new historical features. The current frame is then segmented based on these inter-frame features to obtain the image segmentation result. This approach fully utilizes the continuity between adjacent frames in the video data, performing image segmentation through the semantic features of the current frame and historical features. Because of the continuity between the semantic features of the current frame and historical features, flickering or jitter in the image segmentation result can be effectively suppressed.

[0077] Figure 9 This is a schematic diagram of a terminal device provided in one embodiment of this application. For example... Figure 9 As shown, a terminal device 9 in this embodiment includes a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90, such as an image segmentation program. When the processor 90 executes the computer program 92, it implements the steps described in the various embodiments of the image segmentation method above, for example... Figure 4 Steps 401 to 404 shown or Figure 7 Steps 701 to 705 are shown. Alternatively, when processor 90 executes computer program 92, it implements the functions of each unit in the above-described device embodiments, for example... Figure 8 The functions of units 81 to 84 are shown.

[0078] For example, computer program 92 can be divided into one or more units, one or more of which are stored in memory 91 and executed by processor 90 to complete this application. The one or more units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 92 in a terminal device 9. For example, the specific functions of each unit in computer program 92 can be divided as follows:

[0079] The extraction unit is used to obtain the current frame from the video data and extract the semantic features of the current frame;

[0080] The segmentation unit is used for preprocessing and segmenting semantic features to obtain sub-features x and y;

[0081] The fusion unit is used to fuse historical features, sub-feature x, and sub-feature y to obtain inter-frame features, and to store the fused features of historical features and sub-feature x as new historical features.

[0082] The segmentation unit is used to segment the current frame based on inter-frame features to obtain the image segmentation result.

[0083] The terminal device includes, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of a terminal device 9 and does not constitute a limitation on a terminal device 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, a terminal device may also include input / output devices, network access devices, buses, etc.

[0084] The processor 90 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0085] The memory 91 can be an internal storage unit of the terminal device 9, such as a hard disk or memory of the terminal device 9. The memory 91 can also be an external storage device of the terminal device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 9. Furthermore, the memory 91 can include both internal and external storage units of the terminal device 9. The memory 91 is used to store the computer program and other programs and data required by the roaming control device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0086] In a specific embodiment, the terminal device 9 stores a target image segmentation model, which includes an encoder and a decoder. The decoder includes multiple decoder network layers, and at least one decoder network layer includes an inter-frame feature transfer module. The encoder is used to acquire the current frame in the video data and extract the semantic features of the current frame. The decoder is used to perform image segmentation based on the semantic features of the current frame and the historical features stored in the inter-frame feature transfer module. The inter-frame feature transfer module is used to obtain fused features based on the semantic features of the current frame and the historical features, and store the fused features as new historical features in the inter-frame feature transfer module.

[0087] The encoder includes five encoder network layers. The first layer includes a first convolution module, a first downsampling module, a first batch normalization module, and a first activation function module. Each of the other layers includes two basic blocks. Each basic block has one second downsampling module and two convolution blocks. The convolution blocks include a second convolution module, a second batch normalization module, and a second activation function module.

[0088] The decoder consists of five decoder network layers, each including a third convolutional module, an upsampling module, a third batch normalization module, and a third activation function module. One or more decoder network layers also include an inter-frame feature transfer module for fusing feature data. A skip connection (where x is an integer from 1 to 5) is established between the encoder's x-th layer and the decoder's (6-x)-th layer to transfer detailed information.

[0089] The inter-frame feature transfer module is set in some or all decoding network layers. This module performs the following steps: First, it divides the features obtained from the previous layer into two equal parts along the channel dimension, obtaining sub-features x and y. Then, it concatenates the historical feature h with the sub-feature x along the channel dimension using the concat function, followed by convolution and the tanh activation function to obtain feature hx. Feature hx replaces the historical feature h and is stored as the new historical feature. The convolution integrates information from both the historical feature h and the sub-feature x, while the tanh activation function limits the range of each element in feature hx, preventing outliers. Finally, the concatenated feature hx and sub-feature y along the channel dimension are sent to the next stage of the decoder.

[0090] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0091] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0093] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0094] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0096] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0100] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0101] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0102] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0103] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0104] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0105] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image segmentation model, characterized in that, include: The encoder, comprising multiple encoder network layers, is used to acquire the current frame in video data and extract the semantic features of the current frame; A decoder is used to perform image segmentation based on the semantic features of the current frame and historical features stored in the decoder; The decoder includes multiple decoder network layers, and at least one of the decoder network layers includes an inter-frame feature transfer module. The inter-frame feature transfer module is used to obtain fused features based on the semantic features of the current frame and the historical features, and store the fused features as new historical features in the inter-frame feature transfer module; The inter-frame feature transfer module is used for: The input features are divided into two parts to obtain sub-features x and y; The historical features and the sub-feature x are concatenated to obtain the fused features, and the fused features are stored as new historical features in the inter-frame feature transfer module; By concatenating the fused feature and the sub-feature y, inter-frame features are obtained, and the inter-frame features are passed to the next decoder network layer or used for image segmentation based on the inter-frame features. The pre-defined mapping relationship between multiple encoder network layers and multiple decoder network layers is a first-to-last correspondence.

2. The image segmentation model as described in claim 1, characterized in that, The encoder includes a first encoder network layer and a second encoder network layer; The first encoder network layer includes a first convolution module, a first downsampling module, a first batch normalization module, and a first activation function module; The second encoder network layer includes at least one basic block; wherein the basic block includes a second downsampling module and a convolutional block, and the convolutional block includes a second convolutional module, a second batch normalization module and a second activation function module.

3. The image segmentation model as described in claim 1, characterized in that, The decoder includes multiple decoder network layers, and each decoder network layer includes a third convolution module, an upsampling module, a third batch normalization module, and a third activation function module; One or more decoder network layers also include an inter-frame feature transfer module.

4. An image segmentation method, characterized in that, The image segmentation model described in any one of claims 1-3 includes: Obtain the current frame from the video data and extract the semantic features of the current frame; The semantic features are preprocessed and segmented to obtain sub-features x and y; The historical features, the sub-feature x, and the sub-feature y are fused to obtain inter-frame features, and the fused features of the historical features and the sub-feature x are stored as new historical features. The current frame is segmented based on the inter-frame features to obtain the image segmentation result.

5. The image segmentation method according to claim 4, characterized in that, The process of fusing historical features, sub-feature x, and sub-feature y to obtain inter-frame features, and storing the fused features of historical features and sub-feature x as new historical features, includes: The historical features and the sub-feature x are fused to obtain a fused feature, and the fused feature is stored as a new historical feature. The fused feature and the sub-feature y are fused together to obtain the inter-frame feature.

6. An image segmentation apparatus, characterized in that, The image segmentation model described in any one of claims 1-3 includes: An extraction unit is used to acquire the current frame in video data and extract the semantic features of the current frame; The segmentation unit is used to preprocess and segment the semantic features to obtain sub-features x and y; The fusion unit is used to fuse historical features, sub-feature x, and sub-feature y to obtain inter-frame features, and to store the fused features of historical features and sub-feature x as new historical features. The segmentation unit is used to segment the current frame based on the inter-frame features to obtain the image segmentation result.

7. A training method for the image segmentation model as described in claim 1, characterized in that, include: Acquire several data sets with labeled information as a training sample set; wherein, the data sets include historical frames and the current frame; The historical frames are input into the image segmentation model to be trained to obtain historical features and store them in the image segmentation model to be trained; The current frame is input into the image segmentation model to be trained, which stores historical features, to obtain the segmentation result of the data set; Calculate the loss function value based on the segmentation result of the data group and the annotation information of the data group; The image segmentation model to be trained is iteratively optimized based on the loss function value to obtain the target image segmentation model.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image segmentation method as described in any one of claims 4 to 5 or the training method for the image segmentation model as described in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method as described in any one of claims 4 to 5 or the training method of the image segmentation model as described in claim 7.

Images