Video-based image segmentation method, segmentation model training method, and device

Abstract

Embodiments of the present application provide a video-based image segmentation method, a segmentation model training method and equipment, relating to the image technology field and the video technology field, wherein the method comprises: acquiring a first image and a second image, inputting the first image into a first model of a segmentation model to obtain image features of the first image; storing the image features into a storage pool of the segmentation model to obtain first features; the first features comprise image features of multiple frames of images, and each frame of image in the multiple frames of images is labeled with a target object; inputting the second image into a second model of the segmentation model to obtain second features of the second image; the first model and the second model are two models under a twin network; and based on the segmentation model, the first features and the second features are encoded and decoded to obtain a segmented image of the second image. The repeated extraction of features is avoided, the features can be reused, the calculation amount of image segmentation is reduced, and the efficiency and speed of image segmentation are improved.

Description

Technical Field

[0001] This application relates to the fields of image technology and video technology, and in particular to a video-based image segmentation method, a segmentation model training method, and a device. Background Technology

[0002] With the development of image and video technologies, images in videos can be segmented to identify target objects, such as moving objects.

[0003] In existing technologies, when performing image segmentation on video images, a matching method can be used to calculate the similarity of target objects between consecutive image frames from the pixel level, thereby determining the target object in the next image frame and thus achieving image segmentation of the next image frame.

[0004] In the process of implementing this application, the inventors discovered that the prior art has at least the following problems: In the above matching method, the calculation is based only on the similarity of the target objects in the previous and next image frames. It is necessary to maintain feature encoders for the previous and next image frames respectively, which makes the features unusable during the segmentation process and increases the amount of computation, resulting in the inability to segment the image quickly. Summary of the Invention

[0005] This application provides a video-based image segmentation method, a segmentation model training method, and a device to solve the problem of high computational cost and inability to quickly segment images during image segmentation.

[0006] In a first aspect, embodiments of this application provide a video-based image segmentation method, the method comprising:

[0007] Acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is preceding the second image;

[0008] The first image is input into the first model of the segmentation model to obtain the image features of the first image, and the image features are stored in the storage pool of the segmentation model to obtain the first feature; wherein, the first feature includes the image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with the target object;

[0009] The second image is input into the second model of the segmentation model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network;

[0010] The first feature and the second feature are encoded and decoded based on the segmentation model to obtain a segmented image of the second image, wherein the target object is marked on the segmented image of the second image.

[0011] One embodiment of the above invention has the following advantages or beneficial effects: Image features from each frame of multiple images are incorporated into the first feature. Based on the weight-sharing characteristic of Siamese networks, as the segmentation process continues, the current frame image is transformed into the subsequent first image, i.e., the current frame becomes a past frame. Thus, the features of the current frame image can be used as features of the subsequent first image, avoiding redundant feature extraction. Then, the first and second features are encoded and decoded based on the segmentation model to obtain the segmented image of the second image. Because redundant feature extraction is avoided in the above process, features can be reused, reducing the computational load of image segmentation and improving the efficiency and speed of image segmentation.

[0012] Alternatively, another embodiment of the above invention has the following advantages or beneficial effects: To reduce computational cost, features of all images preceding the second image are not stored in the memory pool; instead, the 0th frame of the video, the first image (i.e., the (t-1)th image), and N frames between the 0th frame and the first image are selected. Furthermore, a dynamic sampling method is used, where the 0th frame can be replaced by a new 0th frame over time, and then the first image and the N frames are also updated and replaced. This reduces computational cost and allows full utilization of the features of the target object in past frames, reusing the temporal information constituted by past frames.

[0013] Alternatively, another embodiment of the above invention has the following advantages or beneficial effects: The first feature is input into the feature encoder in the segmentation model, and the encoded feature is output. The second feature is input into the feature decoder of the segmentation model, and the decoded feature is obtained. Then, in subsequent steps, the encoded and decoded features are combined to perform image segmentation on the second image. In the above process, during the segmentation of the second image, the feature encoder and the feature decoder are distinguished; the first feature, composed of multiple past frames, is used as input to the feature encoder, thereby enhancing the representation and feature robustness of the target object in the past frames; the second feature of the second image to be segmented is used as input to the feature decoder, thereby enhancing the representation and feature robustness of the target object in the current frame image.

[0014] Alternatively, another embodiment of the above invention has the following advantages or beneficial effects: In the process of image segmentation of the second image by combining encoded features and decoded features, the features of past frame images are processed based on the first cross-attention model. In addition to processing the features of past frame images, the first cross-attention model also introduces the features of the current frame image; the features of the current frame image are processed based on the second cross-attention model. In addition to processing the features of the current frame image, the second cross-attention model also introduces the features of past frame images; thereby, the representation of the target object output by the first cross-attention model to the encoded features is enhanced, and the representation of the target object output by the second cross-attention model to the decoded features is enhanced, thereby making the representation of the target object more identifiable and improving the accuracy and robustness of the transmission of the target object information in the past frame image compared to the previous frame.

[0015] Secondly, embodiments of this application provide a method for training a segmentation model for image segmentation, the method comprising:

[0016] Repeat the following steps until the preset conditions are met:

[0017] Acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image that appears before the second image in time; the target object is marked on the second image;

[0018] The first image is input into the first model of the initial model to obtain the image features of the first image, and the image features are stored in the storage pool of the initial model to obtain the first feature; wherein, the first feature includes the image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with the target object;

[0019] The second image is input into the second model of the initial model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network;

[0020] Based on the initial model, the first feature and the second feature are encoded and decoded to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object;

[0021] Based on the segmented image of the second image and the target object labeled in the second image, the parameters of the initial model are adjusted;

[0022] The segmentation model obtained when the preset conditions are met is used to segment the image to obtain the target object.

[0023] One embodiment of the above invention has the following advantages or beneficial effects: Based on the weight-sharing characteristic of Siamese networks, as the segmentation process continues, the current frame image is transformed into the subsequent first image, that is, the current frame is transformed into a past frame; thus, the features of the current frame image can be used as the features of the subsequent first image, avoiding repeated feature extraction. Then, based on the initial model, the first and second features are encoded and decoded to obtain the segmented image of the second image. Because repeated feature extraction is avoided in the above process, features can be reused, reducing the computational load of image segmentation and improving the training efficiency and speed of the model.

[0024] Thirdly, embodiments of this application provide a video-based image segmentation apparatus, the apparatus comprising:

[0025] The first acquisition unit is used to acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is preceding the second image;

[0026] The first processing unit is used to input the first image into the first model of the segmentation model to obtain the image features of the first image.

[0027] A storage unit is used to store the image features into the storage pool of the segmentation model to obtain a first feature; wherein, the first feature includes image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with a target object;

[0028] The second processing unit is used to input the second image into the second model of the segmentation model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network;

[0029] The third processing unit is used to encode and decode the first feature and the second feature based on the segmentation model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object.

[0030] Fourthly, embodiments of this application provide a training apparatus for a segmentation model used for image segmentation, the apparatus comprising:

[0031] An execution unit is used to repeat the following steps until a preset condition is met:

[0032] The first acquisition unit is used to acquire a first image and a second image, wherein the second image is the current frame image, and the first image is an image that occurs before the second image in time; the target object is marked on the second image;

[0033] The first processing unit is used to input the first image into the first model of the initial model to obtain the image features of the first image;

[0034] A storage unit is used to store the image features into the storage pool of the initial model to obtain a first feature; wherein, the first feature includes image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with a target object;

[0035] The second processing unit is used to input the second image into the second model of the initial model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network;

[0036] The third processing unit is used to perform encoding and decoding processing on the first feature and the second feature based on the initial model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object;

[0037] An adjustment unit is used to adjust the parameters of the initial model based on the segmented image of the second image and the target object labeled in the second image;

[0038] The segmentation model obtained when the preset conditions are met is used to segment the image to obtain the target object.

[0039] Fifthly, embodiments of this application provide an electronic device, the electronic device comprising: a memory and a processor;

[0040] Memory; memory for storing executable instructions of the processor;

[0041] The processor is configured to execute the method described in the first aspect or the second aspect.

[0042] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first or second aspect.

[0043] In a seventh aspect, embodiments of this application provide a computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in the first aspect or the second aspect.

[0044] The video-based image segmentation method, segmentation model training method, and device provided in this application acquire a first image and a second image, where the second image is the current frame image and the first image is an image that occurs before the second image in time. The first image is input into a first model of the segmentation model to obtain image features of the first image, which are then stored in the segmentation model's storage pool to obtain a first feature. The first feature includes image features from multiple frames, all of which are images that occur before the second image in time, and each frame is labeled with a target object. The second image is input into a second model of the segmentation model to obtain a second feature of the second image. The first and second models are two models under a Siamese network. Thus, the first feature incorporates image features from each frame of the multiple frames. Based on the weight-sharing characteristic of the Siamese network, as the segmentation process continues, the current frame image is transformed into a subsequent first image, i.e., the current frame is transformed into a past frame. Therefore, the current frame image features can be used as features of subsequent first images, avoiding redundant feature extraction. Then, the first and second features are encoded and decoded based on the segmentation model to obtain a segmented image of the second image. Because the above process avoids the repeated extraction of features, the features can be reused, reducing the computational load of image segmentation and improving the efficiency and speed of image segmentation. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 A flowchart illustrating a video-based image segmentation method provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram of a segmentation model for a video-based image segmentation method provided in an embodiment of this application;

[0048] Figure 3 A flowchart illustrating another video-based image segmentation method provided in this application embodiment;

[0049] Figure 4 A schematic diagram of a segmentation model for another video-based image segmentation method provided in an embodiment of this application;

[0050] Figure 5 A schematic diagram of the feature interaction module of another video-based image segmentation method provided in an embodiment of this application;

[0051] Figure 6 A flowchart of the third step of step 307 in another video-based image segmentation method provided in an embodiment of this application;

[0052] Figure 7 A schematic diagram of a segmentation decoder for another video-based image segmentation method provided in an embodiment of this application;

[0053] Figure 8 A flowchart illustrating a training method for a segmentation model for image segmentation provided in an embodiment of this application;

[0054] Figure 9 A flowchart illustrating another method for training an initial model for image segmentation provided in this application embodiment;

[0055] Figure 10 A schematic diagram of a video-based image segmentation device provided in an embodiment of this application;

[0056] Figure 11 A schematic diagram of another video-based image segmentation apparatus provided in this application embodiment;

[0057] Figure 12 A schematic diagram of the structure of a training device for a segmentation model for image segmentation provided in an embodiment of this application;

[0058] Figure 13 A schematic diagram of the structure of a training device for another segmentation model for image segmentation provided in an embodiment of this application;

[0059] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 15 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0061] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] With the development of image and video technologies, it is possible to segment images in videos to identify target objects, such as moving objects. Video Object Segmentation (VOS) separates the foreground and background of each image in the video to determine the pixel-level segmentation of the target object.

[0064] In one example, a semi-supervised approach can be used to segment images in a video. A segmentation model can be pre-trained; target objects are labeled in the first frame of the video, and then a pixel-level label is assigned to each target object in the first frame. Based on the semi-supervised algorithm and segmentation model, the pixel-level positions of target objects in each subsequent frame of the video are parsed, thereby achieving the segmentation of the target objects in the video.

[0065] In one example, a semi-supervised approach can employ a matching-based method for image segmentation. This method calculates the similarity of target objects between consecutive image frames at the pixel level, thereby determining the target object in the subsequent image frame and achieving image segmentation of that frame. Examples of matching-based methods include the FEELVOS method and the STM method.

[0066] However, the matching method mentioned above only calculates the similarity of the target objects in the preceding and following image frames. This requires maintaining feature encoders for the preceding and following image frames separately, which results in the inability to reuse features during the segmentation process and increases the computational load, making it impossible to segment the image quickly.

[0067] In one example, image segmentation can be performed using a feature propagation-based method. This method uses features from preceding frames in the video as temporal information, incorporating this information into the segmentation of the current frame to generate robust spatiotemporal features of the target object. Examples of feature propagation-based methods include the MaskTrack method, which segments the current frame based on the previous frame. Another example is the RGMP method, which segments the current frame based on the first frame and the previous frame. A third example is the SAT method, which segments the current frame based on multiple preceding frames.

[0068] In one example, a tracking-based approach can be used for image segmentation, which involves dividing the segmentation process into target object tracking and target object segmentation. The tracking process determines the target object's location within the image; then, the detected location is used to segment the target object. Examples of tracking-based methods include the SiamMask method and the TAN method.

[0069] However, the above methods still have the following problems. Maintaining separate feature encoders for each previous and subsequent image frame results in features not being reusable during segmentation and increases computational load, hindering rapid image segmentation. In video segmentation, the appearance and scale of the target object in the image sequence dynamically change over time, and situations such as occlusion, disappearance, and the presence of similar target objects can occur, interfering with the segmentation process and leading to segmentation errors and decreased accuracy. When segmenting the current frame based on previous frames, a fixed step size is used, which can lead to insufficient utilization of target object information if the number of frames is small, resulting in decreased segmentation accuracy.

[0070] The video-based image segmentation method, segmentation model training method, and device provided in this application are intended to solve the above-mentioned technical problems in the prior art.

[0071] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0072] Figure 1 This is a flowchart illustrating a video-based image segmentation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0073] 101. Obtain the first image and the second image, wherein the second image is the current frame image and the first image is the image whose time is before the second image.

[0074] For example, when segmenting an image, the current frame image to be processed is obtained from the monitoring device or local storage and used as the second image. As the video image segmentation process progresses, the target object has been segmented in each of the multiple frames, meaning that the target object has been labeled in each of the multiple frames. A frame image preceding the current frame image can be obtained and used as the first image; the target object has already been labeled in the first image. The target object in the first frame of the multiple frames can be labeled by the user.

[0075] For example, the first image is the (t-1)th frame image, and the second image is the tth frame image. Here, t is a positive integer greater than or equal to 1.

[0076] 102. Input the first image into the first model of the segmentation model to obtain the image features of the first image, and store the image features in the storage pool of the segmentation model to obtain the first feature; wherein, the first feature includes the image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with the target object.

[0077] For example, a segmentation model is pre-trained, which includes a Siamese network, comprising a first model and a second model; the first and second models in the Siamese network can share weights. For instance, the first model of the Siamese network is a Residual Network (ResNet) model, and the second model is also a Residual Network model.

[0078] Figure 2 A schematic diagram of a segmentation model for a video-based image segmentation method provided in this application embodiment is shown below. Figure 2 As shown, the first image is input into the first model of the segmentation model, and then feature extraction is performed on the first image based on the first model to obtain the image features of the first image. The image features can be hierarchical features of the first image. A memory bank is deployed in the segmentation model; the image features of the first image are stored in the memory bank of the segmentation model, and then the image features of the first image are fused into the image features of each image preceding the first image. Since the target object has been labeled in each frame of the multi-frame image as the video image segmentation proceeds, the multi-frame image includes the first image and multiple images preceding the second image; the image features of each frame of the multi-frame image are placed in the memory bank; then, the first feature is extracted from the memory bank. It can be seen that the first feature includes the image features of the multi-frame image, and the images in the multi-frame image are all images that are in time preceding the second image, and each frame of the multi-frame image is labeled with the target object.

[0079] For example, the second image is the t-th frame. Here, t is a positive integer greater than or equal to 1. The multiple frames include the 0th frame of the video, the first image (i.e., the (t-1)th image), and N frames between the 0th frame and the first image. Here, N is an integer greater than or equal to 0. The 0th frame can be replaced by a new 0th frame as time progresses.

[0080] 103. Input the second image into the second model of the segmentation model to obtain the second feature of the second image; wherein the first model and the second model are two models under the Siamese network.

[0081] For example, such as Figure 2 As shown, the second image is input into the second model of the segmentation model, and then the second image is extracted based on the second model to obtain the second feature of the second image. The second feature can be the hierarchical feature of the second image.

[0082] In the above process, the image features of each frame in multiple frames are added to the first feature. Based on the weight sharing characteristics of Siamese networks, as the segmentation process continues, the current frame image is transformed into the subsequent first image, that is, the current frame is transformed into a past frame. Thus, the features of the current frame image can be used as the features of the subsequent first image. This avoids the repeated extraction of features and improves the efficiency and speed of image segmentation.

[0083] 104. Based on the segmentation model, the first feature and the second feature are encoded and decoded to obtain the segmented image of the second image, wherein the target object is marked in the segmented image of the second image.

[0084] For example, after obtaining the first and second features, the first and second features are encoded and decoded based on a segmentation model (including encoding and decoding), thereby segmenting the second image to obtain a segmented image of the second image. The target object is then labeled on the second image, and the segmented image of the second image is also labeled with the target object. The second image obtained in this process can be used as the first image in subsequent segmentation processes.

[0085] For example, a segmented image can be a binary image. Figure 2 As shown, each target object in the second image can be segmented individually.

[0086] In this embodiment, a first image and a second image are acquired. The second image is the current frame image, and the first image is an image that occurs before the second image in time. The first image is input into the first model of the segmentation model to obtain image features of the first image, and these image features are stored in the storage pool of the segmentation model to obtain the first feature. The first feature includes image features from multiple frames, all of which are images that occur before the second image in time, and each frame in the multiple frames is labeled with the target object. The second image is input into the second model of the segmentation model to obtain the second feature of the second image. The first model and the second model are two models under a Siamese network. Thus, the image features of each frame in the multiple frames are added to the first feature. Based on the weight sharing characteristic of the Siamese network, as the segmentation process continues, the current frame image is transformed into the subsequent first image, that is, the current frame is transformed into a past frame. Therefore, the image features of the current frame image can be used as the features of the subsequent first image, avoiding the repeated extraction of features. Then, the first feature and the second feature are encoded and decoded based on the segmentation model to obtain the segmented image of the second image. Because the above process avoids the repeated extraction of features, the features can be reused, reducing the computational load of image segmentation and improving the efficiency and speed of image segmentation.

[0087] Figure 3 A flowchart illustrating another video-based image segmentation method provided in this application embodiment is shown below. Figure 3 As shown, the method includes:

[0088] 301. Obtain the first image and the second image, wherein the second image is the current frame image and the first image is the image whose time is before the second image.

[0089] For example, this step can be referred to step 101, and will not be repeated here.

[0090] 302. Input the first image into the first model of the segmentation model and output the hierarchical features of the first image; determine the C4 feature in the hierarchical features of the first image as the image feature of the first image. Perform transformation mapping processing on the image features along the channel dimension to obtain the processed image features; wherein, the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0091] For example, a segmentation model is pre-trained, which includes a Siamese network, comprising a first model and a second model; the first and second models in the Siamese network can share weights. For instance, the first model of the Siamese network is a residual network model, and the second model is a residual network model; or, the first model of the Siamese network is a convolutional neural network (CNN) model, and the second model is a convolutional neural network model.

[0092] Figure 4 A schematic diagram of a segmentation model for another video-based image segmentation method provided in this application embodiment is shown below. Figure 4 As shown, the first image is input into the first model, and then features are extracted from the first image based on the first model to obtain the hierarchical features of the first image. The hierarchical features (C1, C2, C3, C4, C5) of the first image include features C1, C2, C3, C4, and C5. Feature C4 in the hierarchical features of the first image is used as the image feature of the first image. In this embodiment, feature C4 in the hierarchical features of the first image is used as the image feature of the first image; since feature C4 has a lower resolution and smaller scale, it is beneficial to reduce the subsequent computational complexity.

[0093] For example, if the size of the first image is 256*256, then after the first model extracts features from the first image, the size of the C1 feature is 128*128, the size of the C2 feature is 64*64, the size of the C3 feature is 32*32, the size of the C4 feature is 16*16, and the size of the C5 feature is 8*8. Taking the C4 feature from the hierarchical features of the first image as the image feature of the first image, it can be seen that the size of the C4 feature is 16 times smaller than the size of the first image.

[0094] Then, a transformation mapping process is performed on the image features of the first image along the channel dimension, thereby reducing the number of channels of the image features of the first image to obtain the processed image features. In one example, based on the convolutional neural network model in the segmentation model, a transformation mapping process is performed on the C4 features of the first image along the channel dimension to reduce the number of channels of the C4 features of the first image. It can be seen that the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0095] 303. Store the processed image features in the storage pool of the segmentation model to obtain the first feature; wherein, the first feature includes the image features of multiple frames, all of which are images that are located before the second image in time, and each of the multiple frames is labeled with the target object.

[0096] For example, such as Figure 4As shown, the image features of the first image are stored in the segmentation model's memory bank. Since the target object has been labeled in each frame of the multi-frame image as the video's image segmentation progresses, and the multi-frame image includes the first image and several images preceding the second image, the image features of each frame are placed in the memory bank. Then, the image features of each image in the multi-frame image are taken as the first feature. It can be seen that the first feature includes the image features of the multi-frame image, where all images are images preceding the second image, and each frame is labeled with the target object.

[0097] In this embodiment, the second image is the t-th frame image. Here, t is a positive integer greater than or equal to 1. The multiple frames include the 0th frame image of the video, the first image (i.e., the (t-1)th image), and N frames located between the 0th frame image and the first image. Here, N is an integer greater than or equal to 0. Then, the C4 features of each frame in the N+2 frames are taken as the first feature. The 0th frame image can be replaced by a new 0th frame image as time continues.

[0098] In this embodiment, to reduce computational cost, features from all images preceding the second image are not stored in the memory pool. Instead, the 0th frame, the first image (i.e., the (t-1)th image), and the N frames between the 0th and first frames are selected. Dynamic sampling is employed, allowing the 0th frame to be replaced by a new 0th frame over time, followed by updates to the first and N frames. This reduces computation and allows for full utilization of the target object features from past frames, reusing the temporal information generated by those past frames.

[0099] This leads to the first feature M∈R ZHW×C It can be seen that the first feature M is a feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image that constitutes the first feature.

[0100] 304. Input the second image into the second model of the segmentation model and output the hierarchical features of the second image; perform transformation mapping on the C4 features in the hierarchical features of the second image in the channel dimension to obtain the second feature; wherein, the number of channels of the second feature is less than the number of channels of the C4 features in the hierarchical features of the second image.

[0101] Among them, the first model and the second model are two models under the twin network.

[0102] For example, such as Figure 4As shown, the second image is input into the second model, and then features are extracted from the second image based on the second model to obtain the hierarchical features of the second image. The hierarchical features (C1, C2, C3, C4, C5) of the second image include features C1, C2, C3, C4, and C5. In this embodiment, feature C4 in the hierarchical features of the second image is used as the second feature of the second image; since feature C4 has a lower resolution and smaller scale, it helps to reduce the computational complexity in the subsequent stages.

[0103] Then, in the channel dimension of the image, a transformation mapping process is performed on the C4 feature in the hierarchical features of the second image, thereby reducing the number of channels of the C4 feature in the hierarchical features of the second image, resulting in the second feature. In one example, based on the convolutional neural network model in the segmentation model, a transformation mapping process is performed on the C4 feature in the hierarchical features of the second image in the channel dimension to reduce the number of channels of the C4 feature in the hierarchical features of the second image. It can be seen that the number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

[0104] This leads to the second feature Q∈R HW×C It can be seen that the second feature Q is a feature map, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0105] In this embodiment, since each frame of the multi-frame images in the storage pool has been segmented and the target object has been labeled in each frame, it can be seen that the segmented images of each frame of the multi-frame images in the storage pool have been obtained. For example, the segmented images are binary images.

[0106] During each segmentation process, each frame of the multi-frame image has been labeled. This labeling process involves extracting features from the first image preceding the current frame to obtain the hierarchical features of the segmented image. These hierarchical features (C1, C2, C3, C4, C5) include features C1, C2, C3, C4, and C5. In this embodiment, feature C4 is used as the label feature of the segmented image. Then, based on the convolutional neural network model in the segmentation model, a transformation mapping is performed on the label features of the segmented image along the channel dimension to reduce the number of channels. As a result, the number of channels in the processed label features is less than the number of channels in the unprocessed label features. This yields the processed label features of the segmented image.

[0107] Alternatively, during each segmentation process, each frame of the multi-frame image has already been labeled. In this case, the "labeling" process is as follows: during each segmentation process, such as... Figure 4 As shown, the segmented images of each frame of a multi-frame image are input into a lightweight label encoder. Based on the lightweight label encoder in the segmentation model (such as an AlexNet encoder or a shallow ResNet encoder), feature extraction is performed on the segmented image of the first image preceding the current frame to obtain the label features of the segmented image. Then, based on the convolutional neural network model in the segmentation model, a transformation mapping process is performed on the label features of the segmented image along the channel dimension of the image, reducing the number of channels in the label features of the segmented image. It can be seen that the number of channels in the processed label features is less than the number of channels in the unprocessed label features. Thus, the processed label features of the segmented image are obtained. It is important to note that each segmentation is performed sequentially on the current first image, resulting in the third feature accumulating the features of each segmented image from multiple frames. Furthermore, in this embodiment, each target object in the first image can be processed, thereby ensuring that each target object in the second image can be segmented.

[0108] Then, the processed label features of the segmented images of each frame in the multi-frame images in the storage pool are used to form the third feature M. E ∈R ZHW×C That is, the third feature M E The data includes the processed label features of segmented images from multiple frames in the storage pool.

[0109] It can be known that the third feature M E Let H be a feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image that constitutes the first feature.

[0110] It can be seen that the first feature, the second feature, and the third feature are the same in height; and the first feature, the second feature, and the third feature are the same in width; and the first feature, the second feature, and the third feature are the same in number of channels.

[0111] In this embodiment, steps 301-307 are performed once to segment a second image. As the image sequence in the video is segmented, the second feature and the third feature are continuously accumulated. However, in this embodiment, the second feature only requires features from N+2 frames, and the third feature only requires features from N+2 frames. For example, the 0th frame of the video, the first image (i.e., the tl-th image), and the N frames between the 0th frame and the first image are taken; wherein the 0th frame can be replaced by a new 0th frame as time continues.

[0112] 305. Input the first feature into the feature encoder in the segmentation model to obtain the encoded feature.

[0113] In one example, step 305 includes the following steps:

[0114] The first step of step 305 involves inputting the first feature into the feature encoder in the segmentation model to perform feature modeling processing, mapping the first feature to multiple dimensions to obtain the enhanced first feature. In one example, the enhanced first feature includes the first feature in the first dimension, the first feature in the second dimension, and the first feature in the third dimension.

[0115] The second step of step 305 involves performing convolution processing on the enhanced first feature to obtain the encoded feature; wherein the number of channels in the encoded feature is less than the number of channels in the enhanced first feature. In one example, the second step of step 305 includes performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature.

[0116] For example, in this embodiment, steps 305 and 306 can be executed simultaneously.

[0117] The segmentation model deploys a feature encoder and a feature decoder. The first feature is input into the feature encoder in the segmentation model, outputting coded features. In step 306, the second feature is input into the feature decoder of the segmentation model to obtain decoded features. Then, in subsequent steps, the coded and decoded features are combined to segment the second image. In the above process, during the segmentation of the second image, the feature encoder and feature decoder are distinguished; the first feature, composed of multiple past frames, is used as input to the feature encoder, thereby enhancing the representation and feature robustness of the target object in past frames; the second feature of the second image to be segmented is used as input to the feature decoder, thereby enhancing the representation and feature robustness of the target object in the current frame image.

[0118] In one example, step 305 can be implemented as follows: The first feature is input into the feature encoder in the segmentation model, and then feature modeling is performed on the first feature based on the feature encoder, thereby enhancing the first feature in time and space. The first feature is mapped to multiple dimensions to obtain the enhanced first feature. The enhanced first feature includes the first feature on the first dimension q, the first feature on the second dimension k, and the first feature on the third dimension v.

[0119] Then, the enhanced first feature is convolved based on the feature encoder to reduce the number of channels in the enhanced first feature, thus obtaining the encoded feature. In this embodiment, steps 305 and 306 are executed simultaneously, and the enhanced second feature has already been obtained in step 306; the enhanced second feature includes the second feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v. In one example, when convolving the enhanced first feature, the first feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v can be convolved to obtain the encoded feature. The second feature belongs to the feature of the current frame image, and the first feature belongs to the feature of the past frame image. The target object has been labeled in the past frame image, so the temporal context information of the target object in the past frame image is passed to the feature of the current frame image, so as to complete the segmentation processing of the specific target object in the current frame image.

[0120] In the above process, steps 305 and 306 separate the feature encoder and the feature decoder. The feature encoder processes the first feature (i.e., processes features from multiple past frames) to construct the spatiotemporal sequence features of the target object, thereby enhancing the feature representation of the target object. The spatiotemporal sequence features of the target object have high discriminative power, which is beneficial for segmenting the target object in the current frame image. The feature decoder, as the carrier of feature processing for the current frame image, combines features from multiple past frames to enhance the features of the target object in the current frame image, thus achieving the segmentation of the target object in the current frame image.

[0121] When implementing step 305, the following methods can be used.

[0122] In one example, the first step of step 305 includes: performing feature modeling processing on the first feature based on the first self-attention (SA) model in the feature encoder to map the first feature onto multiple dimensions to obtain the enhanced first feature.

[0123] The second step of step 305 includes: performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention (CA) model in the feature encoder to obtain the convolution-processed features; and performing residual connection processing and normalization processing on the convolution-processed features to obtain the encoded features.

[0124] For example, in this embodiment, the feature encoder in the segmentation model includes a first self-attention model SA1 and a first cross-attention model CA1. The feature decoder in the segmentation model includes a second self-attention model SA2, a second cross-attention model CA2, and a third cross-attention model CA3.

[0125] like Figure 4 As shown, the first feature is input into the first self-attention model SA1 in the feature encoder. Based on the convolutional layer of the first self-attention model SA1, the first feature is mapped to three dimensions: the first dimension q, the second dimension k, and the third dimension v. That is, feature modeling processing (i.e., feature enhancement of the first feature in time and space) is performed on the first feature, resulting in the first feature of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v. Then, based on the formula of the first self-attention model, the first feature of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v are calculated to obtain the enhanced first feature M. SA ∈R ZHW×C The enhanced first feature M SA A feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image constituting the first feature. The enhanced first feature M... SA It includes the first feature of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v.

[0126] The formula for the first self-attention model is: Where Q1 is the first feature of the first dimension q, K1 is the first feature of the second dimension k, and V1 is the first feature of the third dimension v. T d1 is the transpose of K1. k Let K1 be the number of channels. Based on the formula of this attention model, a similarity matrix is ​​calculated using Q1 and K1. Using each element in the similarity matrix as a weight, V1 is mapped to generate a new output. In the formula of the first self-attention model, Q1, K1, and V1 have the same value.

[0127] Then, Figure 5 A schematic diagram of the feature interaction module of another video-based image segmentation method provided in this application embodiment is shown below. Figure 5 As shown, Figure 4The feature interaction module includes a first cross-attention model and a second cross-attention model. Based on the formula of the first cross-attention model CA1 in the feature encoder, convolution processing is performed on the first feature in the first dimension q, the second feature in the second dimension k, and the second feature in the third dimension v to obtain the convolutionally processed features. The formula of the first cross-attention model CA1 can be the same as the formula of the first self-attention model mentioned above. However, the values ​​of Q1, K1, and V1 in the formula of the first self-attention model can be different. Then, residual connection processing and normalization processing are performed on the convolutionally processed features to obtain the encoded features M. out ∈R HW×C .

[0128] It can be seen that the encoded features include the features of the target objects in past frame images and the features of the current frame image; the temporal context information of the target objects in past frame images is passed to the features of the current frame image so as to complete the segmentation processing of specific target objects in the current frame image.

[0129] 306. Input the second feature into the feature decoder of the segmentation model to obtain the decoded feature.

[0130] In one example, step 306 includes the following steps:

[0131] The first step of step 306 involves inputting the second feature into the feature decoder of the segmentation model to perform feature modeling processing on the second feature, mapping it to multiple dimensions to obtain the enhanced second feature. In one example, the enhanced second feature includes the second feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension.

[0132] The second step of step 306 involves convolutional processing of the enhanced second feature to obtain the decoded feature; wherein the number of channels in the decoded feature is less than the number of channels in the enhanced second feature. In one example, the second step of step 306 includes: acquiring the features of the target object in each image across multiple frames; fusing the features of the target object in each image across multiple frames with the first feature to obtain a fused feature; and then convolving the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein the fused feature includes the features of the target object in each image across multiple frames, as well as the first feature.

[0133] For example, in step 306, the second feature is input into the feature decoder of the segmentation model to obtain the decoded feature. Then, in subsequent steps, the encoded and decoded features are combined to perform image segmentation on the second image. Using the second feature of the second image to be segmented as input to the feature decoder enhances the representation of the target object and the feature robustness of the current frame image.

[0134] In one example, step 306 can be implemented as follows: The second feature is input into the feature decoder of the segmentation model, and then feature modeling is performed on the second feature based on the feature decoder, thereby enhancing the second feature in time and space. The second feature is mapped to multiple dimensions to obtain the enhanced second feature. The enhanced second feature includes the second feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v.

[0135] Then, the enhanced second feature is convolved based on the feature decoder to reduce the number of channels in the enhanced second feature, thus obtaining the decoded feature. In this embodiment, steps 305 and 306 are executed simultaneously, and thus the enhanced first feature has already been obtained in step 305; the enhanced first feature includes the first feature on the first dimension q, the first feature on the second dimension k, and the first feature on the third dimension v.

[0136] In one example, when performing convolution processing on the enhanced second feature, the fusion feature can be obtained first: the feature of the target object in each frame of the multi-frame images in the storage pool. Then, referring to the steps above, the feature of the target object in each frame is the processed label feature of the segmented image of each frame in the multi-frame images in the storage pool. The processed label features of the segmented image of each frame in the multi-frame images in the storage pool constitute the third feature. The third feature M... E The first feature M and the second feature q are fused together to obtain the fused feature. Then, the second feature q, the first feature k, and the fused feature are convolved to obtain the decoded feature. The second feature belongs to the current frame image, and the first feature belongs to the past frame image. The target object has been labeled in the past frame image, so the temporal context information of the target object in the past frame image is passed to the feature of the current frame image. At the same time, the feature of the target object in the past frame image is combined with the feature of the current frame image to complete the segmentation of the specific target object in the current frame image.

[0137] When implementing step 306, the following methods can be used.

[0138] In one example, the first step of step 306 includes: performing feature modeling processing on the second feature based on the second self-attention model in the feature decoder to map the second feature onto multiple dimensions to obtain the enhanced second feature.

[0139] The second step of step 306 includes: performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain the convolution-processed feature; and performing residual connection processing and normalization processing on the convolution-processed feature to obtain the decoded feature.

[0140] For example, the feature decoder in the segmentation model includes a second self-attention model SA2, a second cross-attention model CA2, and a third cross-attention model CA3.

[0141] like Figure 4 As shown, the second feature is input into the second self-attention model SA2 of the feature decoder. Based on the second self-attention model SA2, the second feature is mapped to three dimensions: the first dimension q, the second dimension k, and the third dimension v. That is, feature modeling processing (i.e., feature enhancement of the second feature in space and time) is performed on the second feature to obtain the second feature of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v. Then, based on the formula of the second self-attention model, the second features of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v are calculated to obtain the enhanced second feature Q. SA ∈R HW×C Enhanced second feature Q SA A feature map, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels in the feature map. The enhanced first feature is Q. SA It includes the second feature of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v.

[0142] The formula for the second self-attention model is as follows: Where Q2 is the second feature of the first dimension q, K2 is the second feature of the second dimension k, and V2 is the second feature of the third dimension v. T d2 is the transpose of K2. k Let K2 be the number of channels. Based on the formula of this attention model, a similarity matrix is ​​calculated using Q2 and K2. Using each element in the similarity matrix as a weight, V2 is mapped to generate a new output. In the formula of the second self-attention model, Q2, K2, and V2 have the same value.

[0143] Following the steps above, the fusion features can be obtained.

[0144] Then, as Figure 5 As shown, the second feature on the first dimension q, the first feature on the second dimension k, and the fused feature are convolved based on the formula of the second cross-attention model CA2 in the feature decoder to obtain the convolved feature. The formula of the second cross-attention model CA2 can be the same as the formula of the second self-attention model described above. However, the values ​​of Q2, K2, and V2 in the formula of the second self-attention model can be different. Then, the convolved feature is subjected to residual connection processing and normalization to obtain the decoded feature Q. out ∈R HW×C .

[0145] The residual connection processing described above involves adding features together.

[0146] In the feature interaction module, the first cross-attention model processes features from past frames and also incorporates features from the current frame; the second cross-attention model processes features from the current frame and also incorporates features from past frames. This enhances the encoded feature M output by the first cross-attention model. out The representation of the target object enhances the decoding features Q output by the second cross-attention model. out The representation of the target object is improved, making the representation of the target object more recognizable and improving the accuracy and robustness of the transmission of target object information in previous frames.

[0147] Through steps 305 and 306, the features of the target object in the current frame image are incorporated into the features of past frames using the first cross-attention model CA1 on the feature encoder and the second cross-attention model CA2 on the feature decoder, resulting in encoded features M. out Furthermore, the temporal context information of the target object in past frames is passed to the features of the current frame to obtain the decoded feature Q. out Then, the encoded features M output by the feature encoder are... out The decoded feature Q output by the feature decoder out It serves as the input to the third cross-attention model (CA3) in the feature decoder, thereby enabling the effective transfer of feature information of the target object from past frames to the current frame.

[0148] 307. Based on the segmentation model, the encoded and decoded features are processed to obtain the segmented image of the second image. The target object is labeled in the segmented image of the second image.

[0149] Step 307 includes the following steps:

[0150] The first step of step 307 involves convolving the encoded and decoded features to obtain the features to be processed. In one example, the first step of step 307 includes: convolving the encoded and decoded features based on the third cross-attention model in the feature decoder to obtain the features to be processed.

[0151] The second step of step 307 is to perform matrix transformation on the feature to be processed to obtain the transformed feature to be processed; wherein the transformed feature to be processed is a three-dimensional matrix, and the feature to be processed before transformation is a two-dimensional matrix.

[0152] The third step of step 307 involves the segmentation decoder based on the segmentation model processing the features to be processed to obtain a segmented image of the second image. In one example, the third step of step 307 includes: the segmentation decoder based on the segmentation model performing convolution processing on the features to be processed and the C3 feature in the hierarchical features of the second image to obtain a segmented image of the second image.

[0153] For example, based on the segmentation model, the encoded feature M out and decoding features Q out The image is then processed to obtain a segmented image of the second image.

[0154] In one example, a feature decoder based on a segmentation model encodes features M. out and decoding features Q out Convolution processing is performed to obtain the features to be processed. At this point, the encoded features M can be processed based on the formula of the third cross-attention model in the feature decoder. out Sum and decoding features Q out Perform convolution processing to obtain the feature T to be processed. out ∈R HW×C The formula for the third cross-attention model CA3 can be the same as the formula for the second self-attention model described above. However, the values ​​of Q2, K2, and V2 in the third cross-attention model CA3 can be different. Then, the feature T to be processed... out ∈R HW×C Perform matrix transformation to obtain the transformed feature T`. out ∈R H×W×C This transforms the features to be processed from a two-dimensional matrix into the features to be processed from a three-dimensional matrix.

[0155] Then, the segmentation decoder of the segmentation model is used to process the feature T`. outThe process is performed to output a segmented image of the second image. In one example, since the hierarchical features of the second image were obtained through the previous steps, the C3 feature is extracted from the hierarchical features of the second image. Then, the segmentation decoder of the segmentation model is used to process the feature T' to be processed. out The C3 feature in the hierarchical features of the second image is processed to output the segmented image of the second image.

[0156] In one example, Figure 6 A flowchart of the third step of step 307 in another video-based image segmentation method provided in this application embodiment is shown below. Figure 6 As shown, the third step of step 307 can be implemented in the following way:

[0157] Step 3071: The segmentation decoder based on the segmentation model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature.

[0158] In one example, step 3071 includes: performing upsampling processing on the features to be processed by a segmentation decoder based on a segmentation model to obtain a first upsampled feature; wherein the dimension of the first upsampled feature is greater than the dimension of the features to be processed; fusing the first upsampled feature and the C3 feature in the hierarchical features of the second image to obtain a first intermediate feature, and performing convolution processing on the first intermediate feature to obtain a first output feature.

[0159] Step 3072: The segmentation decoder based on the segmentation model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature.

[0160] In one example, step 3072 includes: the segmentation decoder based on the segmentation model performs upsampling processing on the first output feature to obtain a second upsampled feature; wherein the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain a second intermediate feature, and the second intermediate feature is convolved to obtain a second output feature.

[0161] Step 3073: Perform convolution and segmentation processing on the second output features to obtain the segmented image of the second image.

[0162] In one example, step 3073 includes: performing convolution and segmentation processing on the second output features to obtain an initial segmented image; and performing upsampling processing on the initial segmented image to obtain a segmented image of the second image.

[0163] For example, since the hierarchical features of the second image have been obtained through the preceding steps, the C3 feature is extracted from the hierarchical features of the second image; then, the segmentation decoder of the segmentation model is used to process the feature T' to be processed. out The first output feature is obtained by processing the C3 feature in the hierarchical features of the first and second images. Then, the C2 feature in the hierarchical features of the second image is extracted. The segmentation decoder of the segmentation model is used to convolve the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature. After the previous two processing steps, the feature T` to be processed has been obtained. out By adjusting the size of the second image and then performing convolution and segmentation processing on the second output features, a segmented image of the second image with the same size as the second image can be obtained.

[0164] In one example, Figure 7 A schematic diagram of a segmentation decoder for another video-based image segmentation method provided in this application embodiment is shown below. Figure 7 As shown, the segmentation decoder includes a first refinement module, a second refinement module, a 2-channel convolutional layer, and a softmax layer. First, the features T` to be processed are... out The C3 feature from the hierarchical features of the second image is input into the first thinning module; based on the feature T' to be processed in the first thinning module... out Perform a 2x upsampling process to further refine the feature T' to be processed. out The size is doubled (the feature T to be processed is increased in the H and W dimensions). out The size of the feature is increased by two times to obtain the first upsampled feature; it can be seen that the dimension of the first upsampled feature is larger than that of the feature T to be processed. out The dimension (i.e., the size of the first upsampled feature is larger than the feature T` to be processed) out (size).

[0165] The first upsampled feature and the C3 feature from the hierarchical features of the second image are then added together to complete feature fusion and obtain the first intermediate feature. The first intermediate feature is then subjected to two convolutional processes based on the two convolutional layers of the first thinning module to obtain the first output feature.

[0166] At this point, the size of the first output feature is the feature T` to be processed. out The size is twice that of the previous one; and the first output feature is fused with the feature T` to be processed. out And the C3 feature in the hierarchical features of the second image.

[0167] The first output feature is input into the second refinement module; the first output feature is upsampled by 2 times based on the second refinement module, thereby increasing the size of the first output feature by 2 times (in the two dimensions of H and W by 2 times) to obtain the second upsampled feature; it can be seen that the dimension of the second upsampled feature is larger than the dimension of the first output feature (that is, the size of the second upsampled feature is larger than the size of the first output feature).

[0168] The second upsampled feature and the C2 feature from the hierarchical features of the second image are then added together to complete feature fusion and obtain the second intermediate feature. The second intermediate feature is then convolved twice by the two convolutional layers of the second thinning module to obtain the second output feature.

[0169] The second output feature is input into a 2-channel convolutional layer of the segmentation decoder for convolution processing, resulting in the processed second output feature. This processed second output feature is then input into the softmax layer of the segmentation decoder for segmentation processing, yielding the initial segmented image. At this point, the size of the initial segmented image is 1 / 4 the size of the second image.

[0170] Finally, the initial segmentation image is upsampled by 4 times, thereby increasing the size of the initial segmentation image by 4 times, and generating the final segmentation prediction image, that is, the segmentation image of the second image.

[0171] The second image obtained in this process can be used as the first image in the subsequent segmentation process.

[0172] In this embodiment, based on the above embodiment, to reduce computational cost, features from all images preceding the second image are not stored in the memory pool. Instead, the 0th frame, the first image (i.e., the (t-1)th image), and N frames between the 0th and first frames are selected. Dynamic sampling is employed, allowing the 0th frame to be replaced by a new 0th frame over time, followed by updates to the first and N frames. This reduces computation and fully utilizes the features of the target object from past frames, reusing the temporal information generated by those past frames. The first feature is input to the feature encoder in the segmentation model, outputting encoded features. The second feature is input to the feature decoder in the segmentation model, obtaining decoded features. In subsequent steps, the encoded and decoded features are combined to segment the second image. In the above process, during the segmentation of the second image, a feature encoder and a feature decoder are distinguished. The first feature, composed of multiple past frames, is used as input to the feature encoder, thereby enhancing the representation and robustness of the target object in the past frames. Similarly, the second feature of the second image to be segmented is used as input to the feature decoder, further enhancing the representation and robustness of the target object in the current frame. Furthermore, during image segmentation of the second image by combining encoded and decoded features, the features of past frames are processed based on a first cross-attention model, which incorporates features from the current frame in addition to those of past frames. The features of the current frame are processed based on a second cross-attention model, which incorporates features from past frames in addition to those of the current frame. This enhances the representation of the target object from the encoded features output by the first cross-attention model and the representation of the target object from the decoded features output by the second cross-attention model. This makes the representation of the target object more distinctive and improves the accuracy and robustness of transmitting information about the target object from past frames compared to previous frames.

[0173] Figure 8 A flowchart illustrating a training method for a segmentation model for image segmentation provided in this application embodiment is shown below. Figure 8 As shown, the method includes:

[0174] 401. Repeat the following steps until the preset conditions are met.

[0175] For example, this embodiment requires training an initial model to obtain a segmentation model. Steps 402-406 are repeated until a preset condition is met. The segmentation model obtained when the preset condition is met is used to segment the image to obtain the target object.

[0176] In one example, the preset conditions are any of the following: the number of iterations is greater than a preset threshold, and the target object in the segmented image of each second image is the same as the target object labeled in each second image.

[0177] 402. Obtain the first image and the second image, wherein the second image is the current frame image and the first image is the image whose time is before the second image; the target object is marked on the second image.

[0178] For example, the current frame image to be processed is obtained from the monitoring device or local storage, and this current frame image is used as the second image. The target object has already been segmented in each of the multiple frames; that is, the target object has been labeled in each of the multiple frames, meaning each of the multiple frames has a segmented image. The target object in the first frame of the multiple frames can be labeled by the user. Furthermore, the second image also has a segmented image, and the target object is labeled in the second image.

[0179] It should be noted that in this embodiment, both the first and second images are images to be trained.

[0180] 403. Input the first image into the first model of the initial model to obtain the image features of the first image, and store the image features in the storage pool of the initial model to obtain the first feature; wherein, the first feature includes the image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with the target object.

[0181] For example, the initial model includes a Siamese network, which includes a first model and a second model; the first and second models in the Siamese network can share weights. For instance, the first model of the Siamese network is a residual network model, and the second model is a residual network model.

[0182] The first image is input into the first model of the initial model, and then features are extracted from the first image based on the first model to obtain the image features of the first image. The image features can be hierarchical features of the first image. A memory bank is deployed in the initial model; the image features of the first image are stored in the memory bank of the initial model, and then the image features of the first image are fused into the image features of each image preceding the first image. As the video image segmentation progresses, the target object has been labeled in each of the multiple frames, which includes the first image and multiple images preceding the second image. The image features of each of the multiple frames are placed in the memory bank; then, the first feature is extracted from the memory bank. It can be seen that the first feature includes the image features of the multiple frames, all of which are images that occur before the second image, and each of the multiple frames is labeled with the target object.

[0183] The structure of the initial model can be found in [reference]. Figure 2 The structure shown.

[0184] For example, the second image is the t-th frame. Here, t is a positive integer greater than or equal to 1. The multiple frames include the 0th frame of the video, the first image (i.e., the (t-1)th image), and N frames between the 0th frame and the first image. Here, N is an integer greater than or equal to 0. The 0th frame can be replaced by a new 0th frame as time progresses.

[0185] 404. Input the second image into the second model of the initial model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network.

[0186] For example, such as Figure 2 As shown, the second image is input into the second model of the initial model, and then the second image is extracted based on the second model to obtain the second feature of the second image. The second feature can be the hierarchical feature of the second image.

[0187] In the above process, the image features of each frame in multiple frames are added to the first feature. Based on the weight sharing characteristics of Siamese networks, as the segmentation process continues, the current frame image is transformed into the subsequent first image, that is, the current frame is transformed into a past frame. Thus, the features of the current frame image can be used as the features of the subsequent first image. This avoids the repeated extraction of features and improves the efficiency and speed of training.

[0188] 405. Based on the initial model, the first feature and the second feature are encoded and decoded to obtain the segmented image of the second image, wherein the target object is marked on the segmented image of the second image.

[0189] For example, after obtaining the first and second features, the first and second features are encoded and decoded based on the initial model (including encoding and decoding), and then the second image is segmented to obtain a segmented image of the second image. The target object is then labeled on the second image, and the segmented image of the second image is also labeled with the target object. The second image obtained in this process can be used as the first image in subsequent segmentation processes.

[0190] For example, a segmented image can be a binary image. Figure 2 As shown, each target object in the second image can be segmented individually.

[0191] 406. Based on the segmented image of the second image and the target objects labeled in the second image, adjust the parameters of the initial model.

[0192] In one example, step 406 includes: performing calculations on the segmented image of the second image and the labeled target objects of the second image based on the loss function to obtain the calculation results; and adjusting the parameters of the initial model based on the calculation results.

[0193] For example, since the target object has been labeled in the second image, the segmented image of the second image obtained in step 406 is compared with the target object labeled in the second image, and the parameters of the initial model are adjusted according to the consistency between the two.

[0194] The segmented image of the second image and the labeled target objects of the second image can be calculated based on the loss function; then, based on the calculation, the parameters of the initial model can be adjusted.

[0195] When the number of executions (i.e., the number of iterations) in steps 402-406 exceeds a preset threshold, the execution of the steps in this embodiment is stopped, and a segmentation model can be obtained. Alternatively, when the target object in the segmented image of each second image is consistent with the target object labeled in each second image, the execution of the steps in this embodiment is stopped, and a segmentation model can be obtained.

[0196] In this embodiment, during the training of the initial model, a first image and a second image are acquired. The second image is the current frame image, and the first image is an image that occurs before the second image in time. The first image is input into the first model of the initial model to obtain the image features of the first image, and the image features are stored in the storage pool of the initial model to obtain the first feature. The first feature includes the image features of multiple frames, all of which are images that occur before the second image in time, and each frame in the multiple frames is labeled with the target object. The second image is input into the second model of the initial model to obtain the second feature of the second image. The first model and the second model are two models under a Siamese network. Thus, the image features of each frame in the multiple frames are added to the first feature. Based on the weight sharing characteristic of the Siamese network, as the segmentation process continues, the current frame image is transformed into the subsequent first image, that is, the current frame is transformed into a past frame. Thus, the image features of the current frame image can be used as the features of the subsequent first image, avoiding the repeated extraction of features. Then, the first feature and the second feature are encoded and decoded based on the initial model to obtain the segmented image of the second image. Because the above process avoids the repeated extraction of features, the features can be reused, reducing the computational load of image segmentation and improving the training efficiency and speed of the model.

[0197] Figure 9 A flowchart of another training method for an initial model for image segmentation provided in this application embodiment is shown below. Figure 9 As shown, the method includes:

[0198] 501. Repeat the following steps until the preset conditions are met.

[0199] For example, this embodiment requires training an initial model to obtain a segmentation model. Steps 502-509 are repeated until a preset condition is met. The segmentation model obtained when the preset condition is met is used to segment the image to obtain the target object.

[0200] 502. Obtain the first image and the second image, wherein the second image is the current frame image and the first image is the image whose time is before the second image.

[0201] For example, this step can be referred to step 402, and will not be repeated here.

[0202] 503. Input the first image into the first model of the initial model, and output the hierarchical features of the first image; determine the C4 feature in the hierarchical features of the first image as the image feature of the first image. Perform transformation mapping processing on the image features along the channel dimension to obtain the processed image features; wherein, the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0203] For example, an initial model is pre-trained, which includes a Siamese network, and the Siamese network includes a first model and a second model; the first model and the second model in the Siamese network can share weights. For example, the first model of the Siamese network is a residual network model, and the second model is a residual network model; or, the first model of the Siamese network is a convolutional neural network (CNN) model, and the second model is a convolutional neural network model.

[0204] The structure of the initial model can be found in [reference]. Figure 4 The structure shown is as follows: Figure 4 As shown, the first image is input into the first model, and then features are extracted from the first image based on the first model to obtain the hierarchical features of the first image. The hierarchical features (C1, C2, C3, C4, C5) of the first image include features C1, C2, C3, C4, and C5. Feature C4 in the hierarchical features of the first image is used as the image feature of the first image. In this embodiment, feature C4 in the hierarchical features of the first image is used as the image feature of the first image; since feature C4 has a lower resolution and smaller scale, it is beneficial to reduce the subsequent computational complexity.

[0205] For example, if the size of the first image is 256*256, then after the first model extracts features from the first image, the size of the C1 feature is 128*128, the size of the C2 feature is 64*64, the size of the C3 feature is 32*32, the size of the C4 feature is 16*16, and the size of the C5 feature is 8*8. Taking the C4 feature from the hierarchical features of the first image as the image feature of the first image, it can be seen that the size of the C4 feature is 16 times smaller than the size of the first image.

[0206] Then, a transformation mapping process is performed on the image features of the first image along the channel dimension, thereby reducing the number of channels of the image features of the first image to obtain the processed image features. In one example, based on the convolutional neural network model in the initial model, a transformation mapping process is performed on the C4 features of the first image along the channel dimension to reduce the number of channels of the C4 features of the first image. It can be seen that the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0207] 504. Store the processed image features in the storage pool of the initial model to obtain the first feature; wherein, the first feature includes the image features of multiple frames, all of which are images that are located before the second image in time, and each of the multiple frames is labeled with the target object.

[0208] For example, such as Figure 4 As shown, the image features of the first image are stored in the memory bank of the initial model. Since the target object has been labeled in each frame of the multi-frame image as the video image segmentation progresses, and the multi-frame image includes the first image and several images preceding the second image, the image features of each frame of the multi-frame image are placed in the memory bank. Then, the image features of each image in the multi-frame image are taken as the first feature. It can be seen that the first feature includes the image features of the multi-frame image, where all images are images that occur before the second image, and each frame of the multi-frame image is labeled with the target object.

[0209] In this embodiment, the second image is the t-th frame image. Here, t is a positive integer greater than or equal to 1. The multiple frames include the 0th frame image of the video, the first image (i.e., the (t-1)th image), and N frames located between the 0th frame image and the first image. Here, N is an integer greater than or equal to 0. Then, the C4 features of each frame in the N+2 frames are taken as the first feature. The 0th frame image can be replaced by a new 0th frame image as time continues.

[0210] In this embodiment, to reduce computational cost, features from all images preceding the second image are not stored in the memory pool. Instead, the 0th frame, the first image (i.e., the (t-1)th image), and the N frames between the 0th and first frames are selected. Dynamic sampling is employed, allowing the 0th frame to be replaced by a new 0th frame over time, followed by updates to the first and N frames. This reduces computation and allows for full utilization of the target object features from past frames, reusing the temporal information generated by those past frames.

[0211] This leads to the first feature M∈R ZHW×C It can be seen that the first feature M is a feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image that constitutes the first feature.

[0212] 505. Input the second image into the second model of the initial model and output the hierarchical features of the second image; perform transformation mapping on the C4 features in the hierarchical features of the second image in the channel dimension to obtain the second feature; wherein, the number of channels of the second feature is less than the number of channels of the C4 features in the hierarchical features of the second image.

[0213] Among them, the first model and the second model are two models under the twin network.

[0214] For example, such as Figure 4As shown, the second image is input into the second model, and then features are extracted from the second image based on the second model to obtain the hierarchical features of the second image. The hierarchical features (C1, C2, C3, C4, C5) of the second image include features C1, C2, C3, C4, and C5. In this embodiment, feature C4 in the hierarchical features of the second image is used as the second feature of the second image; since feature C4 has a lower resolution and smaller scale, it helps to reduce the computational complexity in the subsequent stages.

[0215] Then, in the channel dimension of the image, a transformation mapping process is performed on the C4 feature in the hierarchical features of the second image, thereby reducing the number of channels of the C4 feature in the hierarchical features of the second image, resulting in the second feature. In one example, based on the convolutional neural network model in the initial model, a transformation mapping process is performed on the C4 feature in the hierarchical features of the second image in the channel dimension to reduce the number of channels of the C4 feature in the hierarchical features of the second image. It can be seen that the number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

[0216] This leads to the second feature Q∈R HW×C It can be seen that the second feature Q is a feature map, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0217] In this embodiment, since each frame of the multi-frame images in the storage pool has been segmented and the target object has been labeled in each frame, it can be seen that the segmented images of each frame of the multi-frame images in the storage pool have been obtained. For example, the segmented images are binary images.

[0218] During each segmentation process, each frame of the multi-frame image has been labeled. This labeling process involves extracting features from the first image preceding the current frame to obtain the hierarchical features of the segmented image. These hierarchical features (C1, C2, C3, C4, C5) include features C1, C2, C3, C4, and C5. In this embodiment, feature C4 is used as the label feature of the segmented image. Then, based on the convolutional neural network model in the initial model, a transformation mapping is performed on the label features of the segmented image along the channel dimension to reduce the number of channels. It is evident that the number of channels in the processed label features is less than the number of channels in the unprocessed label features. This results in the processed label features of the segmented image.

[0219] Alternatively, during each segmentation process, each frame of the multi-frame image has already been labeled. In this case, the "labeling" process is as follows: during each segmentation process, such as... Figure 4 As shown, the segmented images of each frame of a multi-frame image are input into a lightweight label encoder. Based on the lightweight label encoder in the initial model (such as an AlexNet encoder or a shallow ResNet encoder), features are extracted from the segmented image of the first image preceding the current frame to obtain the label features of the segmented image. Then, based on the convolutional neural network model in the initial model, a transformation mapping process is performed on the label features of the segmented image along the channel dimension of the image to reduce the number of channels of the label features of the segmented image. It can be seen that the number of channels of the label features after processing is less than the number of channels of the label features before processing. Thus, the processed label features of the segmented image are obtained. It should be noted that each segmentation is performed sequentially on the current first image, so that the third feature is accumulated from the features of each segmented image of multiple frames; and, in this embodiment, each target object in the first image can be processed to ensure that each target object in the second image can be segmented.

[0220] Then, the processed label features of the segmented images of each frame in the multi-frame images in the storage pool are used to form the third feature M. E ∈R ZHW×C That is, the third feature M E The data includes the processed label features of segmented images from multiple frames in the storage pool.

[0221] It can be known that the third feature M E Let H be a feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image that constitutes the first feature.

[0222] It can be seen that the first feature, the second feature, and the third feature are the same in height; and the first feature, the second feature, and the third feature are the same in width; and the first feature, the second feature, and the third feature are the same in number of channels.

[0223] In this embodiment, steps 502-509 are performed once to segment a second image. As the image sequence in the video is segmented, the second feature and the third feature are continuously accumulated. However, in this embodiment, the second feature only requires features from N+2 frames, and the third feature only requires features from N+2 frames. For example, the 0th frame of the video, the first image (i.e., the (t-1)th image), and the N frames between the 0th frame and the first image are taken; wherein the 0th frame can be replaced by a new 0th frame as time continues.

[0224] 506. Input the first feature into the feature encoder in the initial model to obtain the encoded feature.

[0225] In one example, step 506 includes the following steps:

[0226] The first step of step 506 involves inputting the first feature into the feature encoder in the initial model to perform feature modeling processing on the first feature, mapping the first feature to multiple dimensions to obtain the enhanced first feature. In one example, the enhanced first feature includes the first feature in the first dimension, the first feature in the second dimension, and the first feature in the third dimension.

[0227] The second step of step 506 involves performing convolution processing on the enhanced first feature to obtain the encoded feature; wherein the number of channels in the encoded feature is less than the number of channels in the enhanced first feature. In one example, the second step of step 506 includes performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature.

[0228] For example, in this embodiment, steps 506 and 507 can be executed simultaneously.

[0229] An encoder and a decoder are deployed in the initial model. The first feature is input into the encoder in the initial model, outputting the encoded feature. In step 507, the second feature is input into the decoder in the initial model to obtain the decoded feature. Then, in subsequent steps, the encoded and decoded features are combined to perform image segmentation on the second image. In the above process, during the segmentation of the second image, the encoder and decoder are distinguished; the first feature, composed of multiple past frames, is used as input to the encoder, thereby enhancing the representation and feature robustness of the target object in past frames; the second feature of the second image to be segmented is used as input to the decoder, thereby enhancing the representation and feature robustness of the target object in the current frame image.

[0230] In one example, step 506 can be implemented as follows: The first feature is input into the feature encoder in the initial model, and then feature modeling is performed on the first feature based on the feature encoder, thereby enhancing the first feature in time and space. The first feature is mapped to multiple dimensions to obtain the enhanced first feature. The enhanced first feature includes the first feature on the first dimension q, the first feature on the second dimension k, and the first feature on the third dimension v.

[0231] Then, the enhanced first feature is convolved based on the feature encoder to reduce the number of channels in the enhanced first feature, thus obtaining the encoded feature. In this embodiment, steps 506 and 507 are executed simultaneously, and the enhanced second feature has already been obtained in step 507; the enhanced second feature includes the second feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v. In one example, when convolving the enhanced first feature, the first feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v can be convolved to obtain the encoded feature. The second feature belongs to the feature of the current frame image, and the first feature belongs to the feature of the past frame image. The target object has been labeled in the past frame image, thereby passing the temporal context information of the target object in the past frame image to the feature of the current frame image, so as to complete the segmentation processing of the specific target object in the current frame image.

[0232] In the above process, steps 506 and 507 separate the feature encoder and the feature decoder. The feature encoder processes the first feature (i.e., processes features from multiple past frames) to construct the spatiotemporal sequence features of the target object, thereby enhancing the feature representation of the target object. The spatiotemporal sequence features of the target object have high discriminative power, which is beneficial for segmenting the target object in the current frame image. The feature decoder, as the carrier of feature processing for the current frame image, combines features from multiple past frames to enhance the features of the target object in the current frame image, thus achieving the segmentation of the target object in the current frame image.

[0233] When implementing step 506, the following methods can be used.

[0234] In one example, the first step of step 506 includes: performing feature modeling processing on the first feature based on the first self-attention (SA) model in the feature encoder to map the first feature onto multiple dimensions to obtain the enhanced first feature.

[0235] The second step of step 506 includes: performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention (CA) model in the feature encoder to obtain the convolution-processed features; and performing residual connection processing and normalization processing on the convolution-processed features to obtain the encoded features.

[0236] For example, in this embodiment, the feature encoder in the initial model includes a first self-attention model SA1 and a first cross-attention model CA1. The feature decoder in the initial model includes a second self-attention model SA2, a second cross-attention model CA2, and a third cross-attention model CA3.

[0237] like Figure 4 As shown, the first feature is input into the first self-attention model SA1 in the feature encoder. Based on the convolutional layers of the first self-attention model SA1, the first feature is mapped to three dimensions: the first dimension q, the second dimension k, and the third dimension v. That is, feature modeling processing (i.e., feature enhancement of the first feature in time and space) is performed on the first feature, resulting in the first feature of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v. Then, based on the formula of the first self-attention model, the first features of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v are calculated to obtain the enhanced first feature M. SA ∈R ZHW×C The enhanced first feature M SA A feature map, where H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map, and Z is the number of frames of the image constituting the first feature. The enhanced first feature M... SA It includes the first feature of the first dimension q, the first feature of the second dimension k, and the first feature of the third dimension v.

[0238] The formula for the first self-attention model is: Where Q1 is the first feature of the first dimension q, K1 is the first feature of the second dimension k, and V1 is the first feature of the third dimension v. T d1 is the transpose of K1. k Let K1 be the number of channels. Based on the formula of this attention model, a similarity matrix is ​​calculated using Ql and K1. Using each element in the similarity matrix as a weight, V1 is mapped to generate a new output. In the formula of the first self-attention model, Ql, K1, and Vl have the same value.

[0239] Then, the feature interaction module of the initial model can be found in [reference needed]. Figure 5 As shown, Figure 5 As shown, Figure 4The feature interaction module includes a first cross-attention model and a second cross-attention model. Based on the formula of the first cross-attention model CA1 in the feature encoder, convolution processing is performed on the first feature in the first dimension q, the second feature in the second dimension k, and the second feature in the third dimension v to obtain the convolutional features. The formula of the first cross-attention model CA1 can be the same as the formula of the first self-attention model mentioned above. However, the values ​​of Q1, K1, and V1 in the formula of the first self-attention model can be different. Then, residual connection processing and normalization processing are performed on the convolutional features obtained at this time to obtain the encoded features M. out ∈R HW×C .

[0240] It can be seen that the encoded features include the features of the target objects in past frame images and the features of the current frame image; the temporal context information of the target objects in past frame images is passed to the features of the current frame image so as to complete the segmentation processing of specific target objects in the current frame image.

[0241] 507. Input the second feature into the feature decoder of the initial model to obtain the decoded feature.

[0242] In one example, step 507 includes the following steps:

[0243] The first step of step 507 involves inputting the second feature into the feature decoder of the initial model to perform feature modeling processing on the second feature, mapping it to multiple dimensions to obtain the enhanced second feature. In one example, the enhanced second feature includes the second feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension.

[0244] The second step of step 507 involves convolutional processing of the enhanced second feature to obtain the decoded feature; wherein the number of channels in the decoded feature is less than the number of channels in the enhanced second feature. In one example, the second step of step 507 includes: acquiring the features of the target object in each image across multiple frames; fusing the features of the target object in each image across multiple frames with the first feature to obtain a fused feature; and then convolving the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein the fused feature includes the features of the target object in each image across multiple frames, as well as the first feature.

[0245] For example, in step 507, the second feature is input into the feature decoder of the initial model to obtain the decoded feature. Then, in subsequent steps, the encoded and decoded features are combined to perform image segmentation on the second image. Using the second feature of the second image to be segmented as input to the feature decoder enhances the representation of the target object and the feature robustness of the current frame image.

[0246] In one example, step 507 can be implemented as follows: The second feature is input into the feature decoder of the initial model, and then feature modeling is performed on the second feature based on the feature decoder, thereby enhancing the second feature in time and space. The second feature is mapped to multiple dimensions to obtain the enhanced second feature. The enhanced second feature includes the second feature on the first dimension q, the second feature on the second dimension k, and the second feature on the third dimension v.

[0247] Then, the enhanced second feature is convolved based on the feature decoder to reduce the number of channels in the enhanced second feature, thus obtaining the decoded feature. In this embodiment, steps 506 and 507 are executed simultaneously, and the enhanced first feature has already been obtained in step 506; the enhanced first feature includes the first feature on the first dimension q, the first feature on the second dimension k, and the first feature on the third dimension v.

[0248] In one example, when performing convolution processing on the enhanced second feature, the fusion feature can be obtained first: the feature of the target object in each frame of the multi-frame images in the storage pool. Then, referring to the steps above, the feature of the target object in each frame is the processed label feature of the segmented image of each frame in the multi-frame images in the storage pool. The processed label features of the segmented image of each frame in the multi-frame images in the storage pool constitute the third feature. The third feature M... E The first feature M and the second feature q are fused together to obtain the fused feature. Then, the second feature q, the first feature k, and the fused feature are convolved to obtain the decoded feature. The second feature belongs to the current frame image, and the first feature belongs to the past frame image. The target object has been labeled in the past frame image, so the temporal context information of the target object in the past frame image is passed to the feature of the current frame image. At the same time, the feature of the target object in the past frame image is combined with the feature of the current frame image to complete the segmentation of the specific target object in the current frame image.

[0249] When implementing step 507, the following methods can be used.

[0250] In one example, the first step of step 507 includes: performing feature modeling processing on the second feature based on the second self-attention model in the feature decoder to map the second feature onto multiple dimensions to obtain the enhanced second feature.

[0251] The second step of step 507 includes: performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain the convolution-processed feature; and performing residual connection processing and normalization processing on the convolution-processed feature to obtain the decoded feature.

[0252] For example, the feature decoder in the initial model includes a second self-attention model SA2, a second cross-attention model CA2, and a third cross-attention model CA3.

[0253] like Figure 4 As shown, the second feature is input into the second self-attention model SA2 of the feature decoder. Based on the second self-attention model SA2, the second feature is mapped to three dimensions: the first dimension q, the second dimension k, and the third dimension v. That is, feature modeling processing (i.e., feature enhancement of the second feature in space and time) is performed on the second feature to obtain the second feature of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v. Then, based on the formula of the second self-attention model, the second features of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v are calculated to obtain the enhanced second feature Q. SA ∈R HW×C Enhanced second feature Q SA A feature map, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels in the feature map. The enhanced first feature is Q. SA It includes the second feature of the first dimension q, the second feature of the second dimension k, and the second feature of the third dimension v.

[0254] The formula for the second self-attention model is as follows: Where Q2 is the second feature of the first dimension q, K2 is the second feature of the second dimension k, and V2 is the second feature of the third dimension v. T d2 is the transpose of K2. k Let K2 be the number of channels. Based on the formula of this attention model, a similarity matrix is ​​calculated using Q2 and K2. Using each element in the similarity matrix as a weight, V2 is mapped to generate a new output. In the formula of the second self-attention model, Q2, K2, and V2 have the same value.

[0255] Following the steps above, the fusion features can be obtained.

[0256] Then, as Figure 5 As shown, the second feature on the first dimension q, the first feature on the second dimension k, and the fused feature are convolved based on the formula of the second cross-attention model CA2 in the feature decoder to obtain the convolved feature. The formula of the second cross-attention model CA2 can be the same as the formula of the second self-attention model described above. However, the values ​​of Q2, K2, and V2 in the formula of the second self-attention model can be different. Then, the convolved feature is subjected to residual connection processing and normalization to obtain the decoded feature Q. out ∈R HW×C .

[0257] The residual connection processing described above involves adding features together.

[0258] In the feature interaction module, the first cross-attention model processes features from past frames and also incorporates features from the current frame; the second cross-attention model processes features from the current frame and also incorporates features from past frames. This enhances the encoded feature M output by the first cross-attention model. out The representation of the target object enhances the decoding features Q output by the second cross-attention model. out The representation of the target object is improved, making the representation of the target object more recognizable and improving the accuracy and robustness of the transmission of target object information in previous frames.

[0259] Through steps 506 and 507, the features of the target object in the current frame image are incorporated into the features of past frames using the first cross-attention model CA1 on the feature encoder and the second cross-attention model CA2 on the feature decoder, resulting in encoded features M. out Furthermore, the temporal context information of the target object in past frames is passed to the features of the current frame to obtain the decoded feature Q. out Then, the encoded features M output by the feature encoder are... out The decoded feature Q output by the feature decoder out It serves as the input to the third cross-attention model (CA3) in the feature decoder, thereby enabling the effective transfer of feature information of the target object from past frames to the current frame.

[0260] 508. Based on the initial model, the encoded and decoded features are processed to obtain the segmented image of the second image. The target object is labeled in the segmented image of the second image.

[0261] Step 508 includes the following steps:

[0262] The first step of step 508 involves convolving the encoded and decoded features to obtain the features to be processed. In one example, the first step of step 508 includes: performing convolution on the encoded and decoded features based on the third cross-attention model in the feature decoder to obtain the features to be processed.

[0263] The second step of step 508 is to perform matrix transformation on the feature to be processed to obtain the transformed feature to be processed; wherein the transformed feature to be processed is a three-dimensional matrix, and the feature to be processed before transformation is a two-dimensional matrix.

[0264] The third step of step 508 involves the segmentation decoder based on the initial model processing the features to be processed to obtain a segmented image of the second image. In one example, the third step of step 508 includes: the segmentation decoder based on the initial model performing convolution processing on the features to be processed and the C3 feature in the hierarchical features of the second image to obtain a segmented image of the second image.

[0265] For example, based on the initial model, the encoded features M out and decoding features Q out The image is then processed to obtain a segmented image of the second image.

[0266] In one example, the feature decoder based on the initial model encodes features M. out and decoding features Q out Convolution processing is performed to obtain the features to be processed. At this point, the encoded features M can be processed based on the formula of the third cross-attention model in the feature decoder. out Sum and decoding features Q out Perform convolution processing to obtain the feature T to be processed. out ∈R HW×C The formula for the third cross-attention model CA3 can be the same as the formula for the second self-attention model described above. However, the values ​​of Q2, K2, and V2 in the third cross-attention model CA3 can be different. Then, the feature T to be processed... out ∈R HW×C Perform matrix transformation to obtain the transformed feature T`. out ∈R H×W×C This transforms the features to be processed from a two-dimensional matrix into the features to be processed from a three-dimensional matrix.

[0267] Then, the segmentation decoder of the initial model is used to process the feature T`. outThe process is performed to output a segmented image of the second image. In one example, since the hierarchical features of the second image were obtained through the previous steps, the C3 feature is extracted from the hierarchical features of the second image. Then, the segmentation decoder of the initial model is used to process the feature T' to be processed. out The C3 feature in the hierarchical features of the second image is processed to output the segmented image of the second image.

[0268] In one example, the third step of step 508 can be implemented in the following way:

[0269] The segmentation decoder based on the initial model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature.

[0270] In one example, "the segmentation decoder based on the initial model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature" includes: the segmentation decoder based on the initial model performs upsampling processing on the feature to be processed to obtain the first upsampled feature; wherein, the dimension of the first upsampled feature is greater than the dimension of the feature to be processed; the first upsampled feature and the C3 feature in the hierarchical features of the second image are fused to obtain the first intermediate feature, and the first intermediate feature is convolved to obtain the first output feature.

[0271] The segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature.

[0272] In one example, "the segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature" includes: the segmentation decoder based on the initial model performs upsampling processing on the first output feature to obtain the second upsampled feature; wherein, the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain the second intermediate feature, and the second intermediate feature is convolved to obtain the second output feature.

[0273] The second output features are subjected to convolution and segmentation processing to obtain the segmented image of the second image.

[0274] In one example, "convolution processing and segmentation processing of the second output features to obtain a segmented image of the second image" includes: performing convolution processing and segmentation processing on the second output features to obtain an initial segmented image; and performing upsampling processing on the initial segmented image to obtain a segmented image of the second image.

[0275] For example, since the hierarchical features of the second image have been obtained through the preceding steps, the C3 feature is extracted from the hierarchical features of the second image; then, the segmentation decoder of the initial model is used to process the feature T' to be processed. out The first output feature is obtained by processing the C3 feature in the hierarchical features of the first and second images. Then, the C2 feature in the hierarchical features of the second image is extracted; the segmentation decoder of the initial model is used to convolve the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature. After the previous two processing steps, the feature T` to be processed has been obtained. out By adjusting the size of the second image and then performing convolution and segmentation processing on the second output features, a segmented image of the second image with the same size as the second image can be obtained.

[0276] In one example, the initial model segmentation decoder can be found in [reference needed]. Figure 7 As shown, Figure 7 As shown, the segmentation decoder includes a first refinement module, a second refinement module, a 2-channel convolutional layer, and a softmax layer. First, the features T` to be processed are... out The C3 feature from the hierarchical features of the second image is input into the first thinning module; based on the feature T' to be processed in the first thinning module... out Perform a 2x upsampling process to further refine the feature T' to be processed. out The size is doubled (the feature T to be processed is increased in the H and W dimensions). out The size of the feature is increased by two times to obtain the first upsampled feature; it can be seen that the dimension of the first upsampled feature is larger than that of the feature T to be processed. out The dimension (i.e., the size of the first upsampled feature is larger than the feature T` to be processed) out (size).

[0277] The first upsampled feature and the C3 feature from the hierarchical features of the second image are then added together to complete feature fusion and obtain the first intermediate feature. The first intermediate feature is then subjected to two convolutional processes based on the two convolutional layers of the first thinning module to obtain the first output feature.

[0278] At this point, the size of the first output feature is the feature T` to be processed. out The size is twice that of the previous one; and the first output feature is fused with the feature T` to be processed. out And the C3 feature in the hierarchical features of the second image.

[0279] The first output feature is input into the second refinement module; the first output feature is upsampled by 2 times based on the second refinement module, thereby increasing the size of the first output feature by 2 times (in the two dimensions of H and W by 2 times) to obtain the second upsampled feature; it can be seen that the dimension of the second upsampled feature is larger than the dimension of the first output feature (that is, the size of the second upsampled feature is larger than the size of the first output feature).

[0280] The second upsampled feature and the C2 feature from the hierarchical features of the second image are then added together to complete feature fusion and obtain the second intermediate feature. The second intermediate feature is then convolved twice by the two convolutional layers of the second thinning module to obtain the second output feature.

[0281] The second output feature is input into a 2-channel convolutional layer of the segmentation decoder for convolution processing, resulting in the processed second output feature. This processed second output feature is then input into the softmax layer of the segmentation decoder for segmentation processing, yielding the initial segmented image. At this point, the size of the initial segmented image is 1 / 4 the size of the second image.

[0282] Finally, the initial segmentation image is upsampled by 4 times, thereby increasing the size of the initial segmentation image by 4 times, and generating the final segmentation prediction image, that is, the segmentation image of the second image.

[0283] 509. Based on the segmented image of the second image and the target objects labeled in the second image, adjust the parameters of the initial model.

[0284] For example, refer to step 406 above, which will not be repeated here. When the number of executions (i.e., the number of iterations) in steps 502-509 exceeds a preset threshold, the execution of the steps in this embodiment is stopped, and a segmentation model can be obtained. Alternatively, when the target object in the segmented image of each second image is consistent with the target object labeled in each second image, the execution of the steps in this embodiment is stopped, and a segmentation model can be obtained.

[0285] In this embodiment, based on the above embodiment, to reduce computational cost, features from all images preceding the second image are not stored in the memory pool. Instead, the 0th frame, the first image (i.e., the (t-1)th image), and N frames between the 0th and first frames are selected. Dynamic sampling is employed, allowing the 0th frame to be replaced by a new 0th frame over time, followed by updates to the first and N frames. This reduces computation and fully utilizes the features of the target object from past frames, reusing the temporal information generated by those past frames. The first feature is input into the feature encoder of the initial model, outputting encoded features. The second feature is input into the feature decoder of the initial model, obtaining decoded features. In subsequent steps, the encoded and decoded features are combined to perform image segmentation on the second image. In the above process, during the segmentation of the second image, a feature encoder and a feature decoder are distinguished. The first features, composed of multiple past frames, are used as input to the feature encoder, thereby enhancing the representation and robustness of the target object in the past frames. Similarly, the second features of the second image to be segmented are used as input to the feature decoder, further enhancing the representation and robustness of the target object in the current frame. Furthermore, during image segmentation of the second image by combining encoded and decoded features, the features of past frames are processed based on a first cross-attention model, which incorporates features from the current frame in addition to those of past frames. The features of the current frame are processed based on a second cross-attention model, which incorporates features from past frames in addition to those of the current frame. This enhances the representation of the target object from the encoded features output by the first cross-attention model and the representation of the target object from the decoded features output by the second cross-attention model. This makes the representation of the target object more discriminative and improves the accuracy and robustness of transmitting information about the target object from past frames compared to previous frames. Through this process, the resulting segmentation model can accurately identify the image.

[0286] Figure 10 A schematic diagram of a video-based image segmentation device provided in this application embodiment is shown below. Figure 10 As shown, the device includes:

[0287] The first acquisition unit 61 is used to acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is before the second image.

[0288] The first processing unit 62 is used to input the first image into the first model of the segmentation model to obtain the image features of the first image.

[0289] Storage unit 63 is used to store image features into the storage pool of the segmentation model to obtain the first feature; wherein, the first feature includes image features of multiple frames of images, all of which are images that are located before the second image in time, and each of the multiple frames of images is labeled with the target object.

[0290] The second processing unit 64 is used to input the second image into the second model of the segmentation model to obtain the second feature of the second image; wherein the first model and the second model are two models under the Siamese network.

[0291] The third processing unit 65 is used to encode and decode the first feature and the second feature based on the segmentation model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object.

[0292] For example, this embodiment can refer to the above method embodiment, and its principle and technical effect are similar, so they will not be repeated here.

[0293] Figure 11 This is a schematic diagram of another video-based image segmentation device provided in an embodiment of this application. Figure 10 Based on the illustrated embodiments, as Figure 11 As shown, the third processing unit 65 includes:

[0294] The encoding module 651 is used to input the first feature into the feature encoder in the segmentation model to obtain the encoded feature.

[0295] The decoding module 652 is used to input the second feature into the feature decoder of the segmentation model to obtain the decoded feature.

[0296] The first processing module 653 is used to process the encoded features and decoded features based on the segmentation model to obtain the segmented image of the second image.

[0297] In one example, the first processing module 653 includes:

[0298] The first convolutional submodule is used to perform convolution processing on the encoded features and decoded features to obtain the features to be processed.

[0299] The segmentation submodule is used to process the features to be processed by the segmentation decoder based on the segmentation model to obtain the segmented image of the second image.

[0300] In one example, encoding module 65l includes:

[0301] The first enhancement submodule is used to input the first feature into the feature encoder in the segmentation model, perform feature modeling processing on the first feature, and map the first feature to multiple dimensions to obtain the enhanced first feature.

[0302] The second convolutional submodule is used to perform convolution processing on the enhanced first feature to obtain the encoded feature; wherein the number of channels of the encoded feature is less than the number of channels of the enhanced first feature.

[0303] In one example, decoding module 652 includes:

[0304] The second enhancement submodule is used to input the second feature into the feature decoder of the segmentation model, perform feature modeling processing on the second feature, and map the second feature onto multiple dimensions to obtain the enhanced second feature.

[0305] The third convolutional submodule is used to perform convolution processing on the enhanced second feature to obtain the decoded feature; wherein the number of channels of the decoded feature is less than the number of channels of the enhanced second feature.

[0306] In one example, the enhanced first feature includes the first feature in the first dimension, the first feature in the second dimension, and the first feature in the third dimension; the enhanced second feature includes the second feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension.

[0307] The second convolutional submodule is specifically used to perform convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded features.

[0308] The third convolutional submodule is specifically used to perform convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein, the fused feature includes the features of the target object in each image in multiple frames, as well as the first feature.

[0309] In one example, the apparatus provided in this embodiment further includes:

[0310] The second acquisition unit 71 is used to acquire the features of the target object in each of the multiple frames of images, and to perform fusion processing on the features of the target object in each of the multiple frames of images and the first feature to obtain fused features.

[0311] In one example, the first enhancement submodule is specifically used to perform feature modeling processing on the first feature based on the first self-attention model in the feature encoder.

[0312] The second convolutional submodule is specifically used to: perform convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention model in the feature encoder to obtain the convolutional features; and perform residual connection processing and normalization processing on the convolutional features to obtain the encoded features.

[0313] In one example, the second enhancement submodule is specifically used to perform feature modeling processing on the second feature based on the second self-attention model in the feature decoder.

[0314] The third convolutional submodule is specifically used to: perform convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain the convolutional feature; and perform residual connection processing and normalization processing on the convolutional feature to obtain the decoded feature.

[0315] In one example, the first convolutional submodule is specifically used to: perform convolution processing on the encoded features and decoded features based on the third cross-attention model in the feature decoder to obtain the features to be processed.

[0316] In one example, the segmentation submodule is specifically used to: perform convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image by the segmentation decoder based on the segmentation model, to obtain the segmented image of the second image.

[0317] In one example, the segmentation submodule, specifically used by the segmentation decoder based on the segmentation model to perform convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the segmented image of the second image, specifically involves: the segmentation decoder based on the segmentation model performing convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature; the segmentation decoder based on the segmentation model performing convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature; and performing convolution processing and segmentation processing on the second output feature to obtain the segmented image of the second image.

[0318] In one example, the segmentation submodule, used by the segmentation decoder based on the segmentation model to perform convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature, specifically performs the following: the segmentation decoder based on the segmentation model performs upsampling processing on the feature to be processed to obtain the first upsampled feature; wherein the dimension of the first upsampled feature is greater than the dimension of the feature to be processed; the first upsampled feature and the C3 feature in the hierarchical features of the second image are fused to obtain the first intermediate feature, and the first intermediate feature is convolved to obtain the first output feature.

[0319] In one example, the segmentation submodule, used by the segmentation decoder based on the segmentation model to perform convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature, specifically performs the following: the segmentation decoder based on the segmentation model performs upsampling processing on the first output feature to obtain the second upsampled feature; wherein the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain the second intermediate feature, and the second intermediate feature is convolved to obtain the second output feature.

[0320] In one example, the segmentation submodule, used to perform convolution and segmentation processing on the second output features to obtain a segmented image of the second image, specifically performs the following: performs convolution and segmentation processing on the second output features to obtain an initial segmented image; and performs upsampling processing on the initial segmented image to obtain a segmented image of the second image.

[0321] In one example, the first processing module 653 further includes: a transformation submodule, used to perform matrix transformation on the features to be processed before the segmentation submodule processes the features to be processed based on the segmentation decoder of the segmentation model to obtain the segmented image of the second image, to obtain the transformed features to be processed; wherein the transformed features to be processed are three-dimensional matrices, and the features to be processed before transformation are two-dimensional matrices.

[0322] In one example, the first processing unit 62 includes:

[0323] The second processing module is used to input the first image into the first model of the segmentation model and output the hierarchical features of the first image.

[0324] The third processing module is used to determine the C4 feature in the hierarchical features of the first image, which is the image feature.

[0325] In one example, the apparatus provided in this embodiment further includes: a transformation module, used to perform transformation mapping processing on the image features in the channel dimension of the image before storing the image features in the storage pool of the segmentation model in the storage unit 63 to obtain the first feature, so as to obtain the processed image features; wherein the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0326] In one example, the second processing unit 64 is specifically used for:

[0327] The second image is input into the second model of the segmentation model, and the hierarchical features of the second image are output. The C4 feature in the hierarchical features of the second image is transformed and mapped in the channel dimension of the image to obtain the second feature. The number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

[0328] For example, this embodiment can refer to the above method embodiment, and its principle and technical effect are similar, so they will not be repeated here.

[0329] Figure 12 This application provides a schematic diagram of the structure of a training device for an image segmentation model, as shown in the embodiments of the present application. Figure 12 As shown, the device includes:

[0330] Execution unit 81 is used to repeat the following steps until the preset conditions are met.

[0331] The first acquisition unit 82 is used to acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is earlier than the second image; the target object is marked on the second image.

[0332] The first processing unit 83 is used to input the first image into the first model of the initial model to obtain the image features of the first image.

[0333] Storage unit 84 is used to store image features into the storage pool of the initial model to obtain the first feature. The first feature includes image features from multiple frames, where all images are located before the second image in time, and each frame in the multiple frames is labeled with the target object.

[0334] The second processing unit 85 is used to input the second image into the second model of the initial model to obtain the second feature of the second image; wherein the first model and the second model are two models under the Siamese network.

[0335] The third processing unit 86 is used to encode and decode the first feature and the second feature based on the initial model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object.

[0336] The adjustment unit 87 is used to adjust the parameters of the initial model based on the segmented image of the second image and the target object labeled in the second image.

[0337] The segmentation model obtained when the preset conditions are met is used to segment the image to obtain the target object.

[0338] For example, this embodiment can refer to the above method embodiment, and its principle and technical effect are similar, so they will not be repeated here.

[0339] Figure 13 This is a schematic diagram of the structure of a training device for a segmentation model for image segmentation provided in an embodiment of this application. Figure 12 Based on the illustrated embodiments, as Figure 13 As shown, the third processing unit 86 includes:

[0340] The encoding module 861 is used to input the first feature into the feature encoder in the initial model to obtain the encoded feature.

[0341] The decoding module 862 is used to input the second feature into the feature decoder of the initial model to obtain the decoded feature.

[0342] The first processing module 863 is used to process the encoded features and decoded features based on the initial model to obtain the segmented image of the second image.

[0343] In one example, the first processing module 863 includes:

[0344] The first convolutional submodule is used to perform convolution processing on the encoded features and decoded features to obtain the features to be processed.

[0345] The segmentation submodule is used to process the features to be processed by the segmentation decoder based on the initial model to obtain the segmented image of the second image.

[0346] In one example, encoding module 861 includes:

[0347] The first enhancement submodule is used to input the first feature into the feature encoder in the initial model, perform feature modeling processing on the first feature, and map the first feature to multiple dimensions to obtain the enhanced first feature.

[0348] The second convolutional submodule is used to perform convolution processing on the enhanced first feature to obtain the encoded feature; wherein the number of channels of the encoded feature is less than the number of channels of the enhanced first feature.

[0349] In one example, decoding module 862 includes:

[0350] The second enhancement submodule is used to input the second feature into the feature decoder of the initial model, perform feature modeling processing on the second feature, and map the second feature onto multiple dimensions to obtain the enhanced second feature.

[0351] The third convolutional submodule is used to perform convolution processing on the enhanced second feature to obtain the decoded feature; wherein the number of channels of the decoded feature is less than the number of channels of the enhanced second feature.

[0352] In one example, the enhanced first feature includes the first feature in the first dimension, the first feature in the second dimension, and the first feature in the third dimension; the enhanced second feature includes the second feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension.

[0353] The second convolutional submodule is specifically used to perform convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded features.

[0354] The third convolutional submodule is specifically used to perform convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein, the fused feature includes the features of the target object in each image in multiple frames, as well as the first feature.

[0355] In one example, the apparatus provided in this embodiment further includes:

[0356] The second acquisition unit 88 is used to acquire the features of the target object in each of the multiple frames of images, and to perform fusion processing on the features of the target object in each of the multiple frames of images and the first feature to obtain fused features.

[0357] In one example, the first enhancement submodule is specifically used to perform feature modeling processing on the first feature based on the first self-attention model in the feature encoder.

[0358] The second convolutional submodule is specifically used to: perform convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention model in the feature encoder to obtain the convolutional features; and perform residual connection processing and normalization processing on the convolutional features to obtain the encoded features.

[0359] In one example, the second enhancement submodule is specifically used to perform feature modeling processing on the second feature based on the second self-attention model in the feature decoder.

[0360] The third convolutional submodule is specifically used to: perform convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain the convolutional feature; and perform residual connection processing and normalization processing on the convolutional feature to obtain the decoded feature.

[0361] In one example, the first convolutional submodule is specifically used to: perform convolution processing on the encoded features and decoded features based on the third cross-attention model in the feature decoder to obtain the features to be processed.

[0362] In one example, the segmentation submodule is specifically used to: perform convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image based on the segmentation decoder of the initial model, to obtain the segmented image of the second image.

[0363] In one example, the segmentation submodule is specifically used to perform convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image based on the initial model segmentation decoder to obtain the first output feature; the segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature; and the second output feature is subjected to convolution processing and segmentation processing to obtain the segmented image of the second image.

[0364] In one example, the segmentation submodule, used to perform convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image based on the segmentation decoder of the initial model to obtain the second output feature, specifically performs the following: the segmentation decoder based on the initial model performs upsampling processing on the feature to be processed to obtain the first upsampled feature; wherein the dimension of the first upsampled feature is greater than the dimension of the feature to be processed; the first upsampled feature and the C3 feature in the hierarchical features of the second image are fused to obtain the first intermediate feature, and the first intermediate feature is convolved to obtain the first output feature.

[0365] In one example, the segmentation submodule, used to perform convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image based on the segmentation decoder of the initial model to obtain the second output feature, specifically performs the following: the segmentation decoder based on the initial model performs upsampling processing on the first output feature to obtain the second upsampled feature; wherein the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain the second intermediate feature, and the second intermediate feature is convolved to obtain the second output feature.

[0366] In one example, the segmentation submodule, used to perform convolution and segmentation processing on the second output features to obtain a segmented image of the second image, specifically performs the following: performs convolution and segmentation processing on the second output features to obtain an initial segmented image; and performs upsampling processing on the initial segmented image to obtain a segmented image of the second image.

[0367] In one example, the first processing module 863 further includes: a transformation submodule, used to perform matrix transformation on the features to be processed before the segmentation submodule processes the features to be processed based on the initial model's segmentation decoder to obtain the segmented image of the second image, to obtain the transformed features to be processed; wherein the transformed features to be processed are three-dimensional matrices, and the features to be processed before transformation are two-dimensional matrices.

[0368] In one example, the first processing unit 83 includes:

[0369] The second processing module is used to input the first image into the first model of the initial model and output the hierarchical features of the first image.

[0370] The third processing module is used to determine the C4 feature in the hierarchical features of the first image, which is the image feature.

[0371] In one example, the apparatus provided in this embodiment further includes: a transformation module, used to perform transformation mapping processing on the image features in the channel dimension of the image before the image features are stored in the storage pool of the initial model in the storage unit 84 to obtain the first feature, so as to obtain the processed image features; wherein the number of channels of the processed image features is less than the number of channels of the image features before processing.

[0372] In one example, the second processing unit 85 is specifically used for:

[0373] The second image is input into the second model of the initial model, and the hierarchical features of the second image are output.

[0374] The C4 feature in the hierarchical features of the second image is transformed and mapped along the channel dimension to obtain the second feature; wherein the number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

[0375] In one example, the adjustment unit 87 is specifically used to: perform calculation processing on the segmented image of the second image and the target objects labeled in the second image based on the loss function to obtain the calculation result; and adjust the parameters of the initial model based on the calculation result.

[0376] In one example, the preset conditions are any of the following: the number of iterations is greater than a preset threshold, and the target object in the segmented image of each second image is the same as the target object labeled in each second image.

[0377] For example, this embodiment can refer to the above method embodiment, and its principle and technical effect are similar, so they will not be repeated here.

[0378] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 14 As shown, the electronic device includes: a memory 91 and a processor 92;

[0379] Memory 91; a memory used to store executable instructions of processor 92;

[0380] The processor 92 is configured to perform the methods provided in the embodiments described above.

[0381] The electronic device also includes a receiver 93 and a transmitter 94. The receiver 93 is used to receive instructions and data sent by external devices, and the transmitter 94 is used to send instructions and data to external devices.

[0382] Figure 15 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0383] The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0384] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0385] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0386] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0387] Multimedia component 808 includes a screen that provides an output interface between device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0388] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0389] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0390] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0391] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0392] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0393] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0394] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the methods described above.

[0395] This application also provides a computer program product, which includes: a computer program stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above embodiments.

[0396] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0397] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video-based image segmentation method, the method comprising: Acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is preceding the second image; The first image is input into the first model of the segmentation model, and the hierarchical features of the first image are output. The C4 feature in the hierarchical features of the first image is determined as the image feature of the first image; The image features are transformed and mapped along the channel dimension of the image to obtain processed image features; wherein the number of channels in the processed image features is less than the number of channels in the unprocessed image features. The processed image features are stored in the storage pool of the segmentation model to obtain the first feature; wherein, the first feature includes image features of multiple frames, all of which are images that are located before the second image in time, and each of the multiple frames is labeled with the target object. The frames in the storage pool are dynamically updated and replaced as follows: the 0th frame image, the first image, and the N frames located between the 0th frame image and the first image are selected from the video and stored in the storage pool, and a dynamic sampling method is adopted. As time continues, the 0th frame image is replaced by a new 0th frame image, and the first image and the N frames image are also updated and replaced. The second image is input into the second model of the segmentation model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network; The first feature and the second feature are encoded and decoded based on the segmentation model to obtain a segmented image of the second image, wherein the target object is marked on the segmented image of the second image.

2. The method according to claim 1, wherein encoding and decoding processing is performed on the first feature and the second feature based on the segmentation model to obtain a segmented image of the second image, comprising: The first feature is input into the feature encoder in the segmentation model to obtain the encoded feature; The second feature is then input into the feature decoder of the segmentation model to obtain the decoded feature; The encoded features and the decoded features are processed based on the segmentation model to obtain a segmented image of the second image.

3. The method according to claim 2, wherein processing the encoded features and the decoded features based on the segmentation model to obtain a segmented image of the second image includes: The encoded features and the decoded features are convolved to obtain the features to be processed; The segmentation decoder based on the segmentation model processes the features to be processed to obtain a segmented image of the second image.

4. The method according to claim 3, wherein the first feature is input into the feature encoder in the segmentation model to obtain encoded features, comprising: The first feature is input into the feature encoder in the segmentation model to perform feature modeling processing on the first feature, so as to map the first feature onto multiple dimensions to obtain the enhanced first feature; The enhanced first feature is convolved to obtain the encoded feature; wherein the number of channels of the encoded feature is less than the number of channels of the enhanced first feature.

5. The method according to claim 4, wherein the second feature is input into the feature decoder of the segmentation model to obtain decoded features, comprising: The second feature is input into the feature decoder of the segmentation model to perform feature modeling processing on the second feature, so as to map the second feature onto multiple dimensions to obtain the enhanced second feature; The enhanced second feature is convolved to obtain the decoded feature; wherein the number of channels of the decoded feature is less than the number of channels of the enhanced second feature.

6. The method according to claim 5, wherein the enhanced first feature includes a first feature in a first dimension, a first feature in a second dimension, and a first feature in a third dimension; and the enhanced second feature includes a second feature in a first dimension, a second feature in a second dimension, and a second feature in a third dimension. The enhanced first feature is convolved to obtain the encoded feature, including: convolving the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature; The enhanced second feature is convolved to obtain the decoded feature, which includes: convolving the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein the fused feature includes the features of the target object in each image of the multi-frame image and the first feature.

7. The method according to claim 6, further comprising: The features of the target object in each of the multi-frame images are obtained, and the features of the target object in each of the multi-frame images and the first feature are fused to obtain the fused feature.

8. The method of claim 6, performing feature modeling processing on the first features, comprising: Based on the first self-attention model in the feature encoder, feature modeling processing is performed on the first feature; The method involves performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature. This includes: performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention model in the feature encoder to obtain the convolution-processed feature; and performing residual connection processing and normalization processing on the convolution-processed feature to obtain the encoded feature.

9. The method of claim 6, performing feature modeling processing on the second features, comprising: Based on the second self-attention model in the feature decoder, feature modeling processing is performed on the second feature; The decoding features are obtained by performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature, including: performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain convolution-processed features; and performing residual connection processing and normalization processing on the convolution-processed features to obtain the decoding features.

10. The method according to claim 3, wherein convolution processing is performed on the encoded features and the decoded features to obtain the features to be processed, comprising: The encoded features and the decoded features are convolved based on the third cross-attention model in the feature decoder to obtain the features to be processed.

11. The method according to claim 3, wherein the segmentation decoder based on the segmentation model processes the features to be processed to obtain a segmented image of the second image, comprising: The segmentation decoder based on the segmentation model performs convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image to obtain the segmented image of the second image.

12. The method according to claim 11, wherein the segmentation decoder based on the segmentation model performs convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image to obtain a segmented image of the second image, comprising: The segmentation decoder based on the segmentation model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature; The segmentation decoder based on the segmentation model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature; The second output features are subjected to convolution and segmentation processing to obtain a segmented image of the second image.

13. The method according to claim 12, wherein the segmentation decoder based on the segmentation model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain a first output feature, comprising: The segmentation decoder based on the segmentation model performs upsampling processing on the feature to be processed to obtain a first upsampled feature; wherein the dimension of the first upsampled feature is greater than the dimension of the feature to be processed; the first upsampled feature and the C3 feature in the hierarchical features of the second image are fused to obtain a first intermediate feature, and the first intermediate feature is convolved to obtain a first output feature; The segmentation decoder based on the segmentation model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature, including: The segmentation decoder based on the segmentation model performs upsampling processing on the first output feature to obtain a second upsampled feature; wherein the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain a second intermediate feature, and the second intermediate feature is convolved to obtain a second output feature.

14. The method according to claim 13, wherein the second output features are subjected to convolution processing and segmentation processing to obtain a segmented image of the second image, comprising: The second output feature is subjected to convolution and segmentation processing to obtain an initial segmented image; The initial segmented image is upsampled to obtain the segmented image of the second image.

15. The method according to claim 3, further comprising, before processing the features to be processed by the segmentation decoder based on the segmentation model to obtain the segmented image of the second image: The feature to be processed is subjected to matrix transformation to obtain the transformed feature to be processed; wherein the transformed feature to be processed is a three-dimensional matrix, and the feature to be processed before transformation is a two-dimensional matrix.

16. The method according to any one of claims 1-15, wherein the second image is input into a second model of the segmentation model to obtain a second feature of the second image, comprising: The second image is input into the second model of the segmentation model, and the hierarchical features of the second image are output. The C4 feature in the hierarchical features of the second image is transformed and mapped along the channel dimension of the image to obtain the second feature; wherein the number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

17. A method for training a segmentation model for image segmentation, the method comprising: Repeat the following steps until the preset conditions are met: Acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image that appears before the second image in time; the target object is marked on the second image; The first image is input into the first model of the initial model, and the hierarchical features of the first image are output. The C4 feature in the hierarchical features of the first image is determined as the image feature of the first image; The image features are transformed and mapped along the channel dimension of the image to obtain processed image features; wherein the number of channels in the processed image features is less than the number of channels in the unprocessed image features. The processed image features are stored in the storage pool of the initial model to obtain the first feature; wherein, the first feature includes image features of multiple frames, all of which are images that are located before the second image in time, and each of the multiple frames is labeled with the target object. The frames in the storage pool are dynamically updated and replaced as follows: the 0th frame image, the first image, and the N frames located between the 0th frame image and the first image are selected from the video and stored in the storage pool, and a dynamic sampling method is adopted. As time continues, the 0th frame image is replaced by a new 0th frame image, and the first image and the N frames image are also updated and replaced. The second image is input into the second model of the initial model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network; Based on the initial model, the first feature and the second feature are encoded and decoded to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object; Based on the segmented image of the second image and the target object labeled in the second image, the parameters of the initial model are adjusted; The segmentation model obtained when the preset conditions are met is used to segment the image to obtain the target object.

18. The method according to claim 17, wherein encoding and decoding processing is performed on the first feature and the second feature based on the initial model to obtain a segmented image of the second image, comprising: The first feature is input into the feature encoder in the initial model to obtain the encoded feature; The second feature is then input into the feature decoder of the initial model to obtain the decoded feature; Based on the initial model, the encoded features and the decoded features are processed to obtain a segmented image of the second image.

19. The method according to claim 18, wherein processing the encoded features and the decoded features based on the initial model to obtain a segmented image of the second image comprises: The encoded features and the decoded features are convolved to obtain the features to be processed; The segmentation decoder based on the initial model processes the features to be processed to obtain a segmented image of the second image.

20. The method according to claim 19, wherein the first feature is input into the feature encoder in the initial model to obtain encoded features, comprising: The first feature is input into the feature encoder in the initial model to perform feature modeling processing on the first feature, so as to map the first feature to multiple dimensions and obtain the enhanced first feature. The enhanced first feature is convolved to obtain the encoded feature; wherein the number of channels of the encoded feature is less than the number of channels of the enhanced first feature.

21. The method according to claim 20, wherein the second feature is input into the feature decoder of the initial model to obtain decoded features, comprising: The second feature is input into the feature decoder of the initial model to perform feature modeling processing on the second feature, so as to map the second feature onto multiple dimensions to obtain the enhanced second feature; The enhanced second feature is convolved to obtain the decoded feature; wherein the number of channels of the decoded feature is less than the number of channels of the enhanced second feature.

22. The method according to claim 21, wherein the enhanced first feature includes a first feature in a first dimension, a first feature in a second dimension, and a first feature in a third dimension; and the enhanced second feature includes a second feature in a first dimension, a second feature in a second dimension, and a second feature in a third dimension; The enhanced first feature is convolved to obtain the encoded feature, including: convolving the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature; The enhanced second feature is convolved to obtain the decoded feature, which includes: convolving the second feature in the first dimension, the first feature in the second dimension, and the fused feature to obtain the decoded feature; wherein the fused feature includes the features of the target object in each image of the multi-frame image and the first feature.

23. The method according to claim 22, further comprising: The features of the target object in each of the multi-frame images are obtained, and the features of the target object in each of the multi-frame images and the first feature are fused to obtain the fused feature.

24. The method of claim 22, performing feature modeling processing on the first features, comprising: Based on the first self-attention model in the feature encoder, feature modeling processing is performed on the first feature; The method involves performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension to obtain the encoded feature. This includes: performing convolution processing on the first feature in the first dimension, the second feature in the second dimension, and the second feature in the third dimension based on the first cross-attention model in the feature encoder to obtain the convolution-processed feature; and performing residual connection processing and normalization processing on the convolution-processed feature to obtain the encoded feature.

25. The method of claim 22, performing feature modeling processing on the second features, comprising: Based on the second self-attention model in the feature decoder, feature modeling processing is performed on the second feature; The decoding features are obtained by performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature, including: performing convolution processing on the second feature in the first dimension, the first feature in the second dimension, and the fused feature based on the second cross-attention model in the feature decoder to obtain convolution-processed features; and performing residual connection processing and normalization processing on the convolution-processed features to obtain the decoding features.

26. The method according to claim 19, wherein convolution processing is performed on the encoded features and the decoded features to obtain the features to be processed, comprising: The encoded features and the decoded features are convolved based on the third cross-attention model in the feature decoder to obtain the features to be processed.

27. The method according to claim 19, wherein the segmentation decoder based on the initial model processes the features to be processed to obtain a segmented image of the second image, comprising: The segmentation decoder based on the initial model performs convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image to obtain a segmented image of the second image.

28. The method according to claim 27, wherein the segmentation decoder based on the initial model performs convolution processing on the features to be processed and the C3 features in the hierarchical features of the second image to obtain a segmented image of the second image, comprising: The segmentation decoder based on the initial model performs convolution processing on the feature to be processed and the C3 feature in the hierarchical features of the second image to obtain the first output feature; The segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature; The second output features are subjected to convolution and segmentation processing to obtain a segmented image of the second image.

29. The method according to claim 28, wherein the segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature, comprising: The segmentation decoder based on the initial model performs upsampling on the feature to be processed to obtain a first upsampled feature; wherein the dimension of the first upsampled feature is greater than the dimension of the feature to be processed; the first upsampled feature and the C3 feature in the hierarchical features of the second image are fused to obtain a first intermediate feature, and the first intermediate feature is convolved to obtain a first output feature; The segmentation decoder based on the initial model performs convolution processing on the first output feature and the C2 feature in the hierarchical features of the second image to obtain the second output feature, including: The segmentation decoder based on the initial model performs upsampling processing on the first output feature to obtain a second upsampled feature; wherein the dimension of the second upsampled feature is greater than the dimension of the first output feature; the second upsampled feature and the C2 feature in the hierarchical features of the second image are fused to obtain a second intermediate feature, and the second intermediate feature is convolved to obtain a second output feature.

30. The method according to claim 29, wherein the second output features are subjected to convolution processing and segmentation processing to obtain a segmented image of the second image, comprising: The second output feature is subjected to convolution and segmentation processing to obtain an initial segmented image; The initial segmented image is upsampled to obtain the segmented image of the second image.

31. The method according to claim 19, further comprising, before processing the features to be processed by the segmentation decoder based on the initial model to obtain the segmented image of the second image: The feature to be processed is subjected to matrix transformation to obtain the transformed feature to be processed; wherein the transformed feature to be processed is a three-dimensional matrix, and the feature to be processed before transformation is a two-dimensional matrix.

32. The method according to any one of claims 17-31, wherein the second image is input into a second model of the initial model to obtain a second feature of the second image, comprising: The second image is input into the second model of the initial model, and the hierarchical features of the second image are output. The C4 feature in the hierarchical features of the second image is transformed and mapped along the channel dimension of the image to obtain the second feature; wherein the number of channels of the second feature is less than the number of channels of the C4 feature in the hierarchical features of the second image.

33. The method according to any one of claims 17-31, wherein, based on the segmented image of the second image and the target object labeled in the second image, parameter adjustment of the initial model is performed, comprising: The segmented image of the second image and the labeled target objects in the second image are calculated based on the loss function to obtain the calculation results; The parameters of the initial model are adjusted based on the calculation results.

34. The method according to any one of claims 17-31, wherein the preset condition is any one of the following: the number of iterations is greater than a preset number threshold, and the target object in the segmented image of each second image is consistent with the target object labeled in each second image.

35. A video-based image segmentation apparatus, the apparatus comprising: The first acquisition unit is used to acquire a first image and a second image, wherein the second image is the current frame image and the first image is an image whose time is preceding the second image; The first processing unit is configured to input the first image into the first model of the segmentation model, output the hierarchical features of the first image; determine the C4 feature in the hierarchical features of the first image as the image feature of the first image; perform transformation mapping processing on the image feature in the channel dimension of the image to obtain the processed image feature; wherein the number of channels of the processed image feature is less than the number of channels of the image feature before processing. A storage unit is used to store the processed image features into the storage pool of the segmentation model to obtain a first feature; wherein, the first feature includes image features of multiple frames, all of which are images located before the second image in time, and each of the multiple frames is labeled with a target object. The frames in the storage pool are dynamically updated and replaced in the following manner: the 0th frame image, the first image, and N frames located between the 0th frame image and the first image are selected and stored in the storage pool, and a dynamic sampling method is adopted. As time continues, the 0th frame image is replaced by a new 0th frame image, and the first image and the N frames are also updated and replaced. The second processing unit is used to input the second image into the second model of the segmentation model to obtain the second feature of the second image; wherein, the first model and the second model are two models under a Siamese network; The third processing unit is used to encode and decode the first feature and the second feature based on the segmentation model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object.

36. A training apparatus for a segmentation model for image segmentation, the apparatus comprising: An execution unit is used to repeat the following steps until a preset condition is met: The first acquisition unit is used to acquire a first image and a second image, wherein the second image is the current frame image, and the first image is an image that occurs before the second image in time; the target object is marked on the second image; The first processing unit is configured to input the first image into the first model of the initial model, output the hierarchical features of the first image; determine the C4 feature in the hierarchical features of the first image as the image feature of the first image; perform transformation mapping processing on the image feature in the channel dimension of the image to obtain the processed image feature; wherein the number of channels of the processed image feature is less than the number of channels of the image feature before processing. A storage unit is used to store the processed image features into the storage pool of the initial model to obtain a first feature; wherein, the first feature includes image features of multiple frames, all of which are images located before the second image in time, and each of the multiple frames is labeled with a target object. The frames in the storage pool are dynamically updated and replaced in the following manner: the 0th frame image, the first image, and N frames located between the 0th frame image and the first image are selected and stored in the storage pool, and a dynamic sampling method is adopted. As time continues, the 0th frame image is replaced by a new 0th frame image, and the first image and the N frames are also updated and replaced. The second processing unit is used to input the second image into the second model of the initial model to obtain the second feature of the second image; wherein, the first model and the second model are two models under the Siamese network; The third processing unit is used to perform encoding and decoding processing on the first feature and the second feature based on the initial model to obtain a segmented image of the second image, wherein the segmented image of the second image is labeled with the target object; An adjustment unit is used to adjust the parameters of the initial model based on the segmented image of the second image and the target object labeled in the second image; The segmentation model obtained when the preset conditions are met is used to segment the image to obtain the target object.

37. An electronic device, comprising: Memory, processor; Memory; Memory used to store the processor's executable instructions; The processor is configured to perform the method as described in any one of claims 1-16, or the processor is configured to perform the method as described in any one of claims 17-34.

38. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as claimed in any one of claims 1-16, or, when executed by a processor, are used to implement the method as claimed in any one of claims 17-34.

39. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-16, or, when executed by a processor, implements the method of any one of claims 17-34.

Images