Image segmentation, image segmentation model processing method, device and computer equipment

Abstract

The application relates to an image segmentation method, an image segmentation model processing method, an image segmentation model processing device and a computer device. The method comprises the following steps: extracting initial feature maps based on a target image and a target interaction information map respectively, and fusing the extracted initial feature maps; performing at least one level of encoding processing on the fused feature map to obtain an encoded feature map output by encoding processing; in each level of encoding processing, for a pixel in a feature map input at the level, determining pixels adjacent to the pixel from the feature map input at the level, performing attention feature extraction on each adjacent pixel based on a position deviation between the pixel and each adjacent pixel, updating the pixel according to the extracted attention feature, and determining a feature map output at the level according to the updated pixel; and decoding the encoded feature map to obtain an image segmentation result. The method can improve the accuracy of the image segmentation result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image segmentation method, apparatus, computer equipment, storage medium and computer program product, as well as an image segmentation model processing method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] With the development of computer technology, image segmentation technology has emerged. This technology can extract target object regions from images, which can be living or non-living objects. For example, it can extract the area where the dog is located from an image of a dog.

[0003] In related technologies, image segmentation can be achieved by encoding and decoding images to obtain image segmentation results. However, the accuracy of image segmentation results is often low. Summary of the Invention

[0004] Therefore, it is necessary to provide an image segmentation method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of image segmentation results, as well as an image segmentation model processing method, apparatus, computer device, storage medium, and computer program product, in response to the above-mentioned technical problems.

[0005] On one hand, this application provides an image segmentation method. The method includes:

[0006] A target image and a corresponding target interaction information map are acquired. Initial feature maps are extracted based on the target image and the target interaction information map, and the extracted initial feature maps are fused. The target interaction information map records the position of the pixels affected by the interaction performed on the target image.

[0007] At least one level of encoding processing is performed on the feature map obtained by fusion to obtain an encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, for a pixel in the feature map input at the current level, at least one neighboring pixel is determined from the feature map input at the current level, attention features are extracted for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, the target pixel is updated according to the extracted attention features, and the feature map output at the current level is determined according to the feature map of the updated pixel.

[0008] The encoded feature map is decoded to obtain the image segmentation result corresponding to the target image.

[0009] On the other hand, this application also provides an image segmentation apparatus. The apparatus includes:

[0010] An image acquisition module is used to acquire a target image and a corresponding target interaction information map, extract initial feature maps based on the target image and the target interaction information map respectively, and fuse the extracted initial feature maps; the target interaction information map records the position of the pixels affected by the interaction performed on the target image;

[0011] An encoding module is configured to perform at least one level of encoding processing based on the fused feature map to obtain an encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module is further configured to, for a pixel in the feature map input at the current level, determine at least one neighboring pixel from the feature map input at the current level, extract attention features from each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, update the target pixel according to the extracted attention features, and determine the feature map output at the current level based on the feature map of the updated pixel;

[0012] The decoding module is used to decode the encoded feature map to obtain the image segmentation result corresponding to the target image.

[0013] On the other hand, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the image segmentation method described above.

[0014] On the other hand, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described image segmentation method.

[0015] On the other hand, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the image segmentation method described above.

[0016] The aforementioned image segmentation method, apparatus, computer equipment, storage medium, and computer program product acquire a target image and a corresponding target interaction information map, extract initial feature maps based on the target image and the target interaction information map respectively, fuse the extracted initial feature maps, and perform at least one level of encoding processing based on the fused feature map to obtain an encoded feature map output by at least one level of encoding processing. In each level of encoding processing, for a pixel in the input feature map of the current level, at least one neighboring pixel is determined from the input feature map of the current level, and the relationship between the target pixel and its neighboring pixels is established. The positional deviation is used to extract attention features from each adjacent pixel, and the pixel is updated based on the extracted attention features. The feature map of the layer is determined based on the feature map of the updated pixel. Since attention features are extracted based on positional deviation and the pixels are updated based on the extracted attention features during the encoding process of obtaining the encoded feature map, the relationship between pixels can be used to determine the path of interactive information propagation and perceive the long-distance dependency between pixels. This makes the obtained encoded feature map have better feature representation ability, so when the encoded feature map is decoded, a more accurate image segmentation result can be obtained.

[0017] On the other hand, this application also provides an image segmentation model processing method. The method includes:

[0018] Using an image segmentation model, initial feature maps of samples are extracted based on the sample images and the corresponding sample interaction information maps, and the extracted initial feature maps are then fused together; the sample interaction information map records the positions of the pixels affected by the interactions performed on the sample images.

[0019] The image segmentation model performs at least one level of encoding processing based on the fused sample feature map to obtain a sample encoded feature map output by the at least one level of encoding processing. In each level of encoding processing, for a pixel in the sample feature map input to the current level, at least one neighboring pixel is determined from the sample feature map input to the current level. Attention features are extracted from each neighboring pixel based on the positional deviation between the neighboring pixels. The target pixel is updated according to the extracted attention features. The sample feature map output by the current level is determined based on the feature map of the updated pixel.

[0020] The image segmentation model is used to decode the sample encoded feature map to obtain the sample image segmentation result corresponding to the sample image.

[0021] The image segmentation model is updated based on the sample image segmentation results, and training continues until the training ends.

[0022] On the other hand, this application also provides an image segmentation model processing apparatus. The apparatus includes:

[0023] The image acquisition module is used to extract initial feature maps of samples based on the sample image and the corresponding sample interaction information map using an image segmentation model, and to fuse the extracted initial feature maps; the sample interaction information map records the position of the pixels affected by the interaction performed on the sample image;

[0024] An encoding module is configured to perform at least one level of encoding processing based on the fused sample feature map using the image segmentation model to obtain a sample encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module is further configured to, for a pixel in the sample feature map input at the current level, determine at least one neighboring pixel from the sample feature map input at the current level, extract attention features from each neighboring pixel based on the positional deviation between each neighboring pixel, update the target pixel according to the extracted attention features, and determine the sample feature map output at the current level based on the feature map of the updated pixel;

[0025] The decoding module is used to decode the sample encoded feature map through the image segmentation model to obtain the sample image segmentation result corresponding to the sample image;

[0026] The model update module is used to update the image segmentation model based on the sample image segmentation results and continue training until the training ends.

[0027] On the other hand, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the image segmentation model processing method described above.

[0028] On the other hand, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described image segmentation model processing method.

[0029] On the other hand, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the image segmentation model processing method described above.

[0030] The aforementioned image segmentation model processing method, apparatus, computer equipment, storage medium, and computer program product, through an image segmentation model, extract initial sample feature maps based on sample images and corresponding sample interaction information maps, respectively, and fuse the extracted initial sample feature maps. Based on the fused sample feature maps, at least one level of encoding processing is performed to obtain sample encoded feature maps output by at least one level of encoding processing. In each level of encoding processing, for a pixel in the input sample feature map of the current level, at least one neighboring pixel is determined from the input sample feature map of the current level. Attention is then applied to each neighboring pixel based on the positional deviation between the neighboring pixels of the target pixel. Feature extraction is performed, and the target pixels are updated based on the extracted attention features. The output sample feature map of the layer is determined based on the feature map of the updated pixels. Since the image segmentation model extracts attention features based on positional bias during the encoding process of obtaining the sample encoding feature map, and updates pixels based on the extracted attention features, it can use the relationship between pixels to determine the path of interactive information propagation, thereby achieving a large receptive field. This makes the obtained sample encoding feature map have better feature representation ability, so when decoding the sample encoding feature map, a more accurate image segmentation result can be obtained. By updating the image segmentation model with this image segmentation result, a higher-performance image segmentation model can be obtained. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of the image segmentation method and image segmentation model processing method in some embodiments;

[0032] Figure 2 This is a flowchart illustrating the image segmentation method in some embodiments;

[0033] Figure 3 This is a schematic diagram of the interactive information graph in some embodiments;

[0034] Figure 4 This is a schematic diagram illustrating the adjacency relationship between pixels in the feature map in some embodiments;

[0035] Figure 5 This is a flowchart illustrating the image segmentation method in some specific embodiments;

[0036] Figure 6 This is a schematic diagram of the image segmentation model processing method in some embodiments;

[0037] Figure 7 This is a schematic diagram of the overall process of the image segmentation method in some embodiments;

[0038] Figure 8 This is a schematic diagram illustrating image segmentation using image segmentation methods in some embodiments;

[0039] Figure 9 This is a diagram of the overall network framework of the image segmentation model in some embodiments;

[0040] Figure 10 This is a schematic diagram of the DiNA module in some embodiments;

[0041] Figure 11 This is a schematic diagram of the training and testing process of the image segmentation model in some embodiments;

[0042] Figure 12 This is a visual illustration of the propagation of click information in some embodiments;

[0043] Figure 13 This is a structural block diagram of the image segmentation apparatus in some embodiments;

[0044] Figure 14 This is a structural block diagram of the image segmentation model processing device in some embodiments;

[0045] Figure 15 These are internal structural diagrams of the computer device in some embodiments;

[0046] Figure 16 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0050] With the research and advancement of artificial intelligence (AI) technology, AI is being researched and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role. The solutions provided in this application involve AI technologies such as computer vision and machine learning, which are specifically illustrated through the following embodiments.

[0051] The image segmentation method and image segmentation model processing method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc.

[0052] The image segmentation method provided in this application embodiment can be executed by the server or the terminal alone, or by the server and the terminal working together. Taking server-terminal collaborative execution as an example, the terminal can send an image segmentation request to the server. After receiving the image segmentation request, the server can obtain the target image and the corresponding target interaction information map from the image segmentation request, extract initial feature maps based on the target image and the target interaction information map respectively, and send the target image and the target interaction information map to the server. The server can extract initial feature maps based on the target image and the target interaction information map, fuse the extracted initial feature maps, and perform at least one level of encoding processing based on the fused feature map to obtain at least one level of encoded feature map output. In each level of encoding processing, for a pixel in the input feature map of the current level, at least one neighboring pixel is determined from the input feature map of the current level. Attention features are extracted for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, and the target pixel is updated according to the extracted attention features. The feature map output of the current level is determined according to the feature map of the updated pixel. Finally, the encoded feature map is decoded to obtain the image segmentation result corresponding to the target image, and the image segmentation result is returned to the terminal.

[0053] The image segmentation model processing method provided in this application embodiment can be executed by a server or a terminal alone, or by a server and a terminal working together. Taking server-only execution as an example, the server can acquire sample images and corresponding sample interaction information maps, input the sample images and corresponding sample interaction information maps into the image segmentation model, and perform the following steps through the image segmentation model: extract initial sample feature maps based on the sample images and corresponding sample interaction information maps respectively, fuse the extracted initial sample feature maps, perform at least one level of encoding processing based on the fused sample feature maps, and obtain sample encoded feature maps output by at least one level of encoding processing. In each level of encoding processing, for pixels in the sample feature map input at the current level, the encoding process is performed by extracting the initial sample feature map from the sample feature map input at the current level. The process involves identifying at least one neighboring pixel of the target pixel, extracting attention features from each neighboring pixel based on the positional deviation between them, updating the target pixel based on the extracted attention features, determining the output sample feature map of the layer based on the updated pixel's feature map, decoding the sample encoded feature map to obtain the sample image segmentation result corresponding to the sample image, and further updating the image segmentation model based on the sample image segmentation result. The training continues until the training is complete, resulting in a trained image segmentation model that can be used for image segmentation processing of the target image.

[0054] In some embodiments, such as Figure 2 As shown, an image segmentation method is provided. This method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0055] Step 202: Obtain the target image and the corresponding target interaction information map, extract initial feature maps based on the target image and the target interaction information map respectively, and fuse the extracted initial feature maps.

[0056] The interaction infographic records the positions of pixels affected by interactions performed on the target image. Interactions can take various forms, such as clicks, scribbles, and boxes. Interactions on the target image can be actual user actions or simulated interactions. Actual user actions refer to the user's actual interactions with the target image, while simulated interactions mimic user actions. Each interaction can affect one or more pixels, with "multiple" meaning at least two. The position of the affected pixel can be within at least one region of the foreground or background area. The foreground area is the location of the target object to be segmented within the target image. When the affected pixel is in the foreground area, the interaction is positive; when it is in the background area, the interaction is negative. In specific applications, the target interaction infographic can also record the interaction type of the affected pixels, indicating whether the interaction is positive or negative.

[0057] Specifically, when image segmentation is required, the server can obtain the target image to be segmented and the target interaction information map corresponding to the target image. Initial feature maps are extracted based on the target image and the target interaction information map respectively, and the extracted initial feature maps are fused. The fused feature map can contain the interaction information recorded in the target interaction information map. In subsequent encoding processing, through attention feature extraction, the interaction information recorded in the target interaction information map can be propagated in the feature map.

[0058] In some specific embodiments, the server can downsample based on the target image and the target interaction information map respectively to obtain the initial feature map corresponding to the target image and the initial feature map corresponding to the target interaction information map, and add the two initial feature maps together to obtain the fused feature map.

[0059] In some specific embodiments, the interaction with the target image is a user click operation. The target interaction information map can be generated through the following steps: converting the user's click into a Click map, where its two channels represent the foreground and background classes respectively. A circular area with a radius of a preset number of pixels, centered on each foreground or background click, has a value of 1 inside and 0 outside. The preset number could be, for example, 5. Figure 3 Assuming the user clicks on the target image at a location such as... Figure 3 As shown in Figure (a), where 1 and 2 represent positions within the foreground region and 3 and 4 represent positions within the background region, a result can be generated as shown in Figure (a). Figure 3The target interaction information diagram shown in Figure (b) is shown on the left, where the left side is the foreground channel diagram and the right side is the background channel diagram.

[0060] Step 204: Perform at least one level of encoding processing based on the fused feature map to obtain an encoded feature map output by at least one level of encoding processing; wherein, in each level of encoding processing, for a pixel in the input feature map of the current level, determine at least one neighboring pixel from the input feature map of the current level, extract attention features for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, update the target pixel according to the extracted attention features, and determine the output feature map of the current level based on the feature map of the updated pixel.

[0061] The encoding process, which involves at least one level, can be either single-level or multi-level, with multi-level referring to at least two levels. In multi-level encoding, the input and output of each level is a feature map. In the first level, the encoding can be directly based on the fused feature map. Starting from the second level, the feature map output from the previous level can be used as the input feature map for that level, allowing for multiple feature processing steps until the final level is completed and the encoded feature map is output. For a pixel in the input feature map of a given level, the adjacent pixels can be pixels within the pixel's neighborhood that meet certain conditions. For example, these could be all pixels in the pixel's neighborhood, or pixels within the pixel's neighborhood that meet the hole adjacency condition. The pixels meeting the hole adjacency condition are determined by the hole coefficient of the given level, which is a positive integer greater than or equal to 1.

[0062] Attention feature extraction is a feature extraction process based on the attention mechanism. The target pixel in the input feature map of a given layer has a certain correlation with its neighboring pixels. The attention mechanism can capture this correlation, determine attention weights based on these correlations, and then extract attention features based on these weights. Updating the target pixel based on the extracted attention features allows for long-distance propagation of pixel features across the feature map, resulting in a more representative feature map. The positional deviation between the target pixel and its neighboring pixels refers to the magnitude of the deviation of each neighboring pixel's position relative to the target pixel's position. Generally, a smaller deviation indicates a stronger correlation between the target and neighboring pixels, while a larger deviation indicates a weaker correlation. Therefore, positional deviation and the extracted attention features are negatively correlated; that is, a larger positional deviation results in fewer extracted attention features, and vice versa.

[0063] Specifically, the server performs at least one level of encoding processing based on the fused feature map to obtain an encoded feature map output from at least one level of encoding processing. In specific applications, when the at least one level includes multiple levels, in the encoding processing of each level, for each pixel in the input feature map of the current level, at least one neighboring pixel is determined from the input feature map of the current level. The correlation between the target pixel and each neighboring pixel is determined, and the deviation parameters of each neighboring pixel are determined based on the positional deviation between the target pixel and each neighboring pixel. The correlation and deviation parameters of each neighboring pixel are added together to obtain the attention weight of each pixel. Feature weighting processing is performed on each neighboring pixel based on the attention weight of each neighboring pixel. The pixel features obtained from the feature weighting processing are used to update the target pixel. After each pixel is updated, the feature map of the updated pixel is obtained. The server can determine the output feature map of the current level based on the feature map of the updated pixel, thereby realizing the encoding processing of this level. After the feature map is output by the current level of encoding, the next level of encoding can continue to encode based on the output feature map, thereby realizing multi-level encoding and obtaining multi-level output encoded feature maps.

[0064] During the encoding process, since pixel updates can be performed based on attention features, interaction information can propagate over long distances in the feature map, and the resulting feature map can better represent the location of the target object that needs to be segmented.

[0065] In some specific embodiments, after obtaining the feature map of the updated pixel, the server can directly use the feature map of the updated pixel as the input feature map of the current layer.

[0066] In other embodiments, after obtaining the feature map of the updated pixel, the server can further perform pixel optimization processing on the feature map of the updated pixel, such as pixel normalization processing, MLP (Multilayer Perceptron) processing, etc.

[0067] In other embodiments, the server can combine two methods for determining neighboring pixels to obtain the feature map output at the current level. For example, in the encoding process of each level, for a pixel in the input feature map of the current level, the server can determine the neighborhood of the target pixel from the input feature map of the current level, and determine the pixels that meet the hole adjacency condition within the neighborhood as neighboring pixels. After obtaining the feature map of the updated pixel based on these neighboring pixels, attention feature extraction can be performed on the feature map of the updated pixel to update the pixel. In the second attention feature extraction process, for each pixel, the server can take each pixel in the neighborhood of the pixel as the neighboring pixel of the pixel, and then perform attention feature extraction based on these neighboring pixels to update the pixel. In this way, by selecting different neighboring pixels to perform attention feature extraction twice, information propagation can be better carried out, and the expressive power of the obtained feature map can be further improved.

[0068] In some specific embodiments, in the encoding process of each level, before performing attention feature extraction, the server can also perform pixel normalization processing on the feature map of the input at that level, thereby improving the efficiency of the encoding process at that level.

[0069] Step 206: Decode the encoded feature map to obtain the image segmentation result corresponding to the target image.

[0070] Specifically, after obtaining the encoded feature map output by the encoding process, the server can further decode the encoded feature map to obtain the image segmentation result corresponding to the target image. Optionally, the image segmentation result can be a mask image corresponding to the target image, which can indicate the location of the target object to be segmented in the target image.

[0071] In practical applications, the image segmentation method of this application can be implemented by a trained image segmentation model. After obtaining the target image and the corresponding target interaction information map, the target image and the corresponding target interaction information map can be input into the trained image segmentation model. The image segmentation model extracts initial feature maps based on the target image and the target interaction information map respectively, and fuses the extracted initial feature maps. The image segmentation model performs at least one level of encoding processing based on the fused feature maps to obtain an encoded feature map output by at least one level of encoding processing. The image segmentation model decodes the encoded feature map to obtain the image segmentation result corresponding to the target image.

[0072] In the aforementioned image segmentation method, the target image and the corresponding target interaction information map are acquired. Initial feature maps are extracted based on the target image and the target interaction information map, respectively. The extracted initial feature maps are then fused, and at least one level of encoding processing is performed based on the fused feature map to obtain an encoded feature map output by at least one level of encoding processing. In each level of encoding processing, for a pixel in the input feature map of the current level, at least one neighboring pixel is determined from the input feature map of the current level. Attention features are extracted for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel. The target pixel is then updated based on the extracted attention features, and the output feature map of the current level is determined based on the updated pixel feature map. Since attention features are extracted based on positional deviations and pixels are updated based on the extracted attention features during the encoding process of obtaining the encoded feature map, the relationship between pixels can be used to determine the propagation path of interaction information and perceive the long-distance dependency between pixels. This results in the encoded feature map having better feature representation capabilities, thus leading to more accurate image segmentation results when decoding the encoded feature map.

[0073] In some embodiments, determining at least one pixel adjacent to the target pixel from the feature map input at the current level includes: determining the size of a neighborhood window corresponding to the current level; generating at least one neighborhood window matching the size of the neighborhood window in the feature map input at the current level; and determining at least one of the pixels covered by the maximum number of directly adjacent pixels of the target pixel in the feature map input at the current level as the pixel adjacent to the target pixel.

[0074] The neighborhood window indicates the range of a pixel's neighborhood. The neighborhood window size describes the size of the neighborhood window. Optionally, the neighborhood window size can be preset; for example, it can be preset to 5*5. In other embodiments, the neighborhood window size can also be determined based on a preset number of adjacent pixels, and is positively correlated with the preset number of adjacent pixels. That is, the more preset adjacent pixels, the larger the neighborhood window size, and vice versa. In specific applications, if adjacent pixels are configured as each pixel within the neighborhood, the neighborhood window size can be directly determined based on the preset number of adjacent pixels. For example, assuming there are 48 adjacent pixels, the neighborhood window size is 7*7. The maximum number of directly adjacent pixels for a targeted pixel is determined by the pixel's position. When the targeted pixel is located at the edge of the feature map, the maximum number of directly adjacent pixels can be two or three. When the targeted pixel is located in the middle of the feature map, the maximum number of directly adjacent pixels is four, meaning there are directly adjacent pixels in all four directions (up, down, left, and right). See the example below. Figure 4 , Figure 4 Pixel A is located at the edge of the feature map. Pixel A has two directly adjacent pixels, namely pixels B and C. Pixel C is located at the edge of the feature map. Pixel C has three directly adjacent pixels, namely pixels A, D, and E. Pixel D is located in the middle of the feature map. Pixel D has four directly adjacent pixels, namely pixels B, C, F, and G.

[0075] Specifically, the server can determine the size of the neighborhood window corresponding to its current layer, and then generate one or more neighborhood windows matching the size in the feature map input at that layer. This allows the server to determine adjacent pixels based on the neighborhood window. In some specific embodiments, a single neighborhood window can be generated and slid across the feature map input at the current layer. During this sliding process, when the neighborhood window covers the maximum number of directly adjacent pixels of the target pixel, the pixels currently covered by the neighborhood window are identified as the pixels adjacent to the target pixel. In other specific embodiments, multiple neighborhood windows can be generated. For example, a neighborhood window can be generated for each pixel in the feature map, ensuring that the neighborhood window covers the maximum number of directly adjacent pixels of that pixel. This allows the pixels covered by the neighborhood window to be identified as the pixels adjacent to that pixel.

[0076] In the above embodiments, neighboring pixels can be determined quickly by using a neighborhood window, thereby improving the efficiency of encoding processing.

[0077] In some embodiments, determining the neighborhood window size corresponding to the current level includes: obtaining a preset number of adjacent pixels and determining an initial window size based on the number of adjacent pixels; obtaining a hole coefficient set for the current level and determining a window expansion size based on the hole coefficient; and determining the neighborhood window size corresponding to the current level based on the initial window size and the window expansion size.

[0078] The hole coefficient determines the pixel distance between selected adjacent pixels. When the hole coefficient is 1, the pixel distance between selected adjacent pixels is 0, meaning the selected adjacent pixels are directly adjacent. In this case, adjacent pixels are all pixels within the neighborhood window. When the hole coefficient is 2, the pixel distance between selected adjacent pixels is 1, and so on. The initial window size describes the window size when all pixels within the neighborhood are considered as adjacent pixels. For example, assuming there are 8 adjacent pixels, the initial window size is 3*3. The window expansion size refers to the size by which the initial window size needs to be expanded when adjacent pixels are selected within the neighborhood based on the hole coefficient. For example, for an initial window size of 3*3, assuming the hole coefficient is 2, the window expansion size is 2*2.

[0079] Specifically, after obtaining the preset number of adjacent pixels, the server can determine the initial window size based on this number. It then obtains the hole coefficient set for the current layer, determines the window expansion size based on the hole coefficient, and adds the initial window size and the window expansion size to obtain the corresponding neighborhood window size for that layer. For example, assuming the initial window size is 3*3 and the window expansion size is 2*2, the final neighborhood window size will be 5*5.

[0080] It is understandable that if the hole coefficient is 1, the window expansion size is 0, and the initial window size is the size of the neighboring window corresponding to the current level.

[0081] In the above embodiments, by setting a hole coefficient to expand the neighborhood window, the receptive field of the neighborhood window can be further increased, thereby further improving the propagation range of interactive information.

[0082] In some embodiments, the hole coefficient is greater than 1; determining at least one of the pixels covered by the maximum number of neighborhood windows as the pixel adjacent to the target pixel includes: performing a modulo operation on the position parameters of the target pixel in the feature map of the input layer according to the hole coefficient to obtain the modulo operation result corresponding to the target pixel; performing a modulo operation on the position parameters of each pixel covered by the maximum number of neighborhood windows according to the hole coefficient to obtain the modulo operation result corresponding to each covered pixel; selecting at least one pixel from the pixels covered by the neighborhood windows whose modulo operation result matches the modulo operation result corresponding to the target pixel, and determining the selected pixel as the pixel adjacent to the target pixel.

[0083] The position parameter describes the order of pixel positions in the feature map. For an example, please refer to [link / reference needed]. Figure 4 The position parameters of pixels A, C, and E are 1, 2, and 3, respectively.

[0084] Specifically, for a pixel with position parameter i in the feature map of the input layer, the server can perform a modulo operation on the pixel according to the hole coefficient δ of the layer to obtain the modulo operation result imodδ. If the modulo operation result jmodδ=imodδ of a pixel with position parameter j in the neighborhood covered by the maximum number of neighborhood windows, then the server can select the pixel with position parameter j as the pixel adjacent to the pixel with position parameter i.

[0085] In the above embodiments, adjacent pixels are determined by modulo operation, which can accurately and quickly determine adjacent pixels, thereby improving the efficiency of encoding processing.

[0086] In some embodiments, at least one pixel adjacent to the targeted pixel includes each pixel in the neighborhood corresponding to the targeted pixel; determining the feature map output of the level based on the feature map of the updated pixel includes: performing pixel value optimization processing on the feature map of the updated pixel to obtain an intermediate feature map; for each intermediate pixel in the intermediate feature map, selecting an intermediate pixel from the neighborhood of the targeted intermediate pixel at a preset pixel distance as the intermediate pixel adjacent to the targeted intermediate pixel; the preset pixel distance is matched with the hole coefficient set for the level; attention features are extracted from each adjacent intermediate pixel based on the positional deviation between the targeted intermediate pixel and each of the adjacent intermediate pixels; the targeted intermediate pixel is updated based on the extracted attention features; and the feature map output of the level is determined based on the intermediate feature map of the updated pixel.

[0087] The pixel value optimization processing includes at least one of pixel value normalization, MLP processing, or cross-connection fusion. The preset pixel distance is matched with the hole coefficient set for the layer, that is, the preset pixel distance is consistent with the pixel distance determined by the hole coefficient. For example, if the hole coefficient is 2, the preset pixel distance is 1; if the hole coefficient is 3, the preset pixel distance is 2, and so on.

[0088] Specifically, the server can combine two methods for determining neighboring pixels to obtain the feature map output at its current level. In the encoding process of each level, for a pixel in the input feature map of that level, the server can determine the neighborhood of that pixel, treating each pixel within that neighborhood as a neighboring pixel. After obtaining the feature map of the updated pixel based on these neighboring pixels, pixel value optimization can be performed on the updated pixel's feature map to obtain an intermediate feature map. Then, attention feature extraction is performed based on the intermediate feature map to continue pixel updates. In the second attention feature extraction... During the extraction process, for each intermediate pixel in the intermediate feature map, an intermediate pixel is selected from the neighborhood of the targeted intermediate pixel at a preset pixel distance as the intermediate pixel adjacent to the targeted intermediate pixel. The preset pixel distance matches the hole coefficient set for the level. That is, the selected pixels as the intermediate pixels adjacent to the targeted intermediate pixel are pixels in the neighborhood of the targeted intermediate pixel that meet the hole adjacency condition. Attention feature extraction is then performed based on these adjacent intermediate pixels to update the intermediate pixel. The server can further determine the feature map output for the level based on the intermediate feature map of the updated pixel.

[0089] Optionally, the server can determine the intermediate feature map of the updated pixel as the feature map output at the current level, or the server can update each pixel in the intermediate feature map of the updated pixel to obtain the feature map output at the current level.

[0090] In the above embodiments, by selecting different neighboring pixels to perform attention feature extraction twice, information propagation can be better carried out, and the expressive power of the obtained feature map can be further improved.

[0091] In some embodiments, updating the intermediate pixel targeted by the extracted attention features and determining the feature map of the output layer based on the intermediate feature map of the updated pixel includes: obtaining the intermediate feature map of the updated pixel by updating the intermediate pixel targeted by the extracted attention features; performing cross-connection fusion on the intermediate feature map of the updated pixel and the intermediate feature map obtained by pixel value optimization processing to obtain a fused feature map; and determining the feature map of the output layer based on the fused feature map.

[0092] Specifically, after obtaining the intermediate feature map of the updated pixels, the server performs cross-connection fusion between the intermediate feature map of the updated pixels and the intermediate feature map obtained through pixel value optimization, resulting in a fused feature map. The fusion method can be either adding corresponding positions or averaging. The server further determines the output feature map for its current layer based on the fused feature map. In practical applications, the server can further perform pixel value optimization processing on the fused feature map, such as pixel value normalization and MLP processing, and determine the feature map obtained from the pixel value optimization processing as the output feature map for its current layer.

[0093] In the above embodiments, feature maps obtained from two attention feature extractions are fused through cross-connection fusion, so that the feature map output at the same level has better feature representation ability.

[0094] In some embodiments, attention features are extracted from each neighboring pixel based on the positional deviation between the target pixel and each of its neighboring pixels, and the target pixel is updated according to the extracted attention features. This includes: determining the correlation between the target pixel and each of its neighboring pixels under various attention feature extraction methods, and determining the deviation parameters of each of its neighboring pixels based on the positional deviation between the target pixel and each of its neighboring pixels; summing the correlation and deviation parameters of each of its neighboring pixels under each attention feature extraction method to obtain the attention weight of each of its neighboring pixels under each attention feature extraction method; performing feature weighting processing on each of its neighboring pixels based on the attention weights of each of its neighboring pixels under each attention feature extraction method; fusing the pixel features obtained from the feature weighting processing under each attention feature extraction method, and updating the target pixel according to the fused pixel features.

[0095] Specifically, the server extracts attention features using various attention feature extraction methods. Each method corresponds to a different query parameter matrix, key parameter matrix, and value parameter matrix. Under each method, the server can convert the target pixel into a query representation using the query parameter matrix, convert adjacent pixels into keys using the key parameter matrix, and convert adjacent pixels into values ​​using the value parameter matrix. The relevance between the query and each key is then calculated, for example, by calculating the dot product between the query and the key, thus determining the relevance between the target pixel and its adjacent pixels. Furthermore, the server can query a pre-built index table based on the positional deviation between the target pixel and its adjacent pixels. Based on the index value in the retrieved index table, the server determines the deviation parameter from the deviation parameter table for that attention extraction method. For example, continuing to refer to... Figure 4 The positional deviation between pixel A and pixel C is 1. The index value obtained by querying a pre-built index table is 4. Therefore, the deviation parameter is the 4th parameter in the deviation parameter table. The deviation parameter table is pre-built and can be set based on experience or obtained through machine learning training.

[0096] The role of the bias parameter is to make the attention weights vary according to the different pixel positions between pixels. Therefore, under each attention feature extraction method, the server can also add the relevance and bias parameter of each adjacent pixel to obtain the attention weight of each adjacent pixel under each attention feature extraction method. Based on the attention weight of each adjacent pixel, the server performs feature weighting processing on the adjacent pixels. Finally, the server fuses the pixel features obtained by feature weighting processing under each attention feature extraction method and updates the target pixel with the fused pixel features.

[0097] In the above embodiments, attention features are extracted through multiple feature extraction methods, which can extract features from multiple different dimensions. The attention features extracted by multiple feature extraction methods are fused, and the pixels obtained by fusion are used to update the target pixels, which can improve the accuracy of attention feature extraction.

[0098] In some embodiments, acquiring a target image and a corresponding target interaction information map, and extracting initial feature maps based on the target image and the target interaction information map respectively, includes: acquiring a target image and a corresponding target interaction information map, and acquiring a historical segmentation image obtained by performing historical segmentation processing on the target image; overlaying the historical segmentation image and the target interaction information map by channels to obtain a channel-overlayed target interaction information map; and extracting initial feature maps from the channel-overlayed target interaction information map and the target image respectively.

[0099] The historical segmentation image is a mask image obtained by performing at least one historical segmentation process on the target image. The target interaction information map after channel overlay is the image obtained by overlaying the historical segmentation image and the target interaction information map in the channel direction.

[0100] Specifically, the server can overlay the historical segmented image and the target interaction information map through channels to obtain the overlaid target interaction information map. Then, the server can downsample the overlaid target interaction information map and the target image in the same way to obtain the initial feature map corresponding to the overlaid target interaction information map and the initial feature map corresponding to the target image.

[0101] In a specific implementation, the historical interaction information map and the initialized mask image can be channel-overlayed and then input together with the target image into a trained image segmentation model. The trained image segmentation model performs image segmentation to obtain the historical segmented image. In other embodiments, the historical segmented image obtained by channel overlaying the target interaction information map can also be obtained through multiple historical segmentations. That is, after obtaining the historical segmented image based on the initialized mask image, the historical interaction information map and the historical segmented image can be channel-overlayed and then input together with the target image into a trained image segmentation model. The trained image segmentation model performs image segmentation again to obtain the historical segmented image.

[0102] In the above embodiments, by overlaying the historical segmentation image and the target interaction information map through channels, the target interaction information map after channel overlay can be obtained. This can be combined with the historical segmentation image for image segmentation, further improving the accuracy of the image segmentation results.

[0103] In some embodiments, at least one level includes multiple levels; encoding processing at least one level is performed based on the fused feature map to obtain an encoded feature map output by the encoding processing at least one level, including: in the encoding processing of the first level, for a pixel in the fused feature map, determining at least one pixel adjacent to the target pixel in the fused feature map, extracting attention features from each adjacent pixel based on the positional deviation between the target pixel and each adjacent pixel, updating the target pixel according to the extracted attention features, and determining the feature map output by the first level based on the feature map of the updated pixel; at each level starting from the second level, encoding processing of the current level is performed based on the feature map output by the previous level to obtain the feature map output by the current level; when the encoding processing of the last level is completed, encoded feature maps of multiple levels are obtained.

[0104] Specifically, at least one level includes multiple levels, where "multiple" refers to at least two. These levels are connected in series. In the encoding process of the first level, the server directly encodes the fused feature map, determines at least one neighboring pixel of the target pixel from the fused feature map, extracts attention features from each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, updates the target pixel based on the extracted attention features, and determines the output feature map of the first level based on the updated pixel's feature map. From the second level onwards, at each level, the server can perform encoding processing based on the feature map output of the previous level to obtain the output feature map of the current level. When the encoding processing of the last level is completed, the output feature map of the last level is the encoded feature map output from multiple levels of encoding processing. That is, for each level starting from the second level, the output feature map of the previous level can be used as the input feature map for encoding processing, and the output feature map can be used as the input feature map of the next level, until the encoding processing of the last level is completed, obtaining the encoded feature map.

[0105] In this embodiment, the server can perform multi-level encoding processing to extract deep features from the target image, which is beneficial to obtain features that can accurately express the characteristics of the target object to be segmented, thereby improving the accuracy of image decoding processing and obtaining more accurate image segmentation results.

[0106] In some embodiments, for each level starting from the second level, an encoded feature map of the current level is generated based on the feature map output by the previous level to obtain the feature map output by the current level. This includes: for each level starting from the second level, when the current level satisfies the downsampling condition, dividing the feature map output by the previous level into multiple feature blocks of the same size; extracting pixels at the same position between the multiple feature blocks of the same size, and forming pixel matrices corresponding to each position by the pixels extracted at each position; performing channel stacking on the pixel matrices, and optimizing the pixel values ​​of the feature map obtained by channel stacking to obtain a sampled feature map; and performing encoding processing on the sampled feature map to obtain the feature map output by the current level.

[0107] The downsampling condition is used to determine whether the feature map output from the previous level of encoding needs to be downsampled to encode features of different size ranges. The downsampling condition can be set according to actual needs. Specifically, it can be set to determine whether the number of encoding operations on a feature map of a certain size has reached a threshold. If the threshold is reached, the downsampling condition is considered met, and downsampling of the feature map is triggered.

[0108] Specifically, starting from the second level, at each level where the downsampling condition is met, the server performs sampling. During the downsampling process, the server can first divide the feature map output from the previous level into multiple feature blocks of the same size. By dividing the size, the feature map output from the previous level is reduced in size. Next, the multiple feature blocks obtained by the division can be merged by channel merging. Specifically, pixels with the same position between multiple feature blocks of the same size are extracted from each feature block. The pixels extracted at each position are used to form the pixel matrix corresponding to each position. The pixel matrices are then superimposed by channel, and the pixel values ​​of the feature map obtained by channel superposition are optimized to achieve downsampling and obtain the sampled feature map.

[0109] For example, suppose the feature map output from the previous layer is 4*4. Divide the feature map into 4 feature blocks using a 2*2 window. Extract the pixels at the same position in each feature block to obtain four pixel matrices. Stack these four pixel matrices along the depth direction, normalize them along the channel direction, and then perform pixel value optimization processing through a fully connected layer (mapping in each depth direction) to obtain the sampled feature map.

[0110] The final sampled feature map obtained by the server is smaller in length and width and doubled in channel dimension compared to the feature map output by the previous level. The server can use the sampled feature map as the input feature map of the current level, and then encode the feature map to obtain the output feature map of the current level.

[0111] In the above embodiments, by dividing the feature map output from the previous level into multiple feature blocks of the same size, extracting pixels at the same position between multiple feature blocks of the same size, forming pixel matrices corresponding to each position by the pixels extracted at each position, and superimposing the pixel matrices by channels, the number of channels can be adjusted while reducing the resolution, so that the feature map obtained by downsampling forms a hierarchy, with stronger feature expression capability, and can reduce the amount of computation in the decoding process.

[0112] In a specific embodiment, such as Figure 5 As shown, an image segmentation method is provided, which specifically includes the following steps:

[0113] Step 502: Obtain the target image and the corresponding target interaction information map, and obtain the historical segmentation image obtained by performing historical segmentation processing on the target image.

[0114] The target interaction information graph records the position of the pixels affected by the interaction performed on the target image.

[0115] Step 504: Overlay the historical segmented image and the target interaction information map by channels to obtain the target interaction information map after channel overlay.

[0116] Among them, the target interaction information map after channel overlay is the image obtained by overlaying the historical segmentation image and the target interaction information map in the channel direction.

[0117] Step 506: Extract initial feature maps from the target interaction information map and the target image after channel overlay, respectively, and fuse the extracted initial feature maps.

[0118] Specifically, the server can downsample the target interactive information map and the target image after channel overlay in the same way to extract two initial feature maps, and then fuse the extracted initial feature maps.

[0119] The server performs multi-level encoding processing based on the feature map obtained by fusion, and obtains encoded feature maps output by at least one level of encoding processing. The first level of encoding processing steps are described in detail below.

[0120] Step 508: Perform first-level encoding processing based on the feature map obtained by fusion. In the first-level encoding processing, for each pixel in the feature map obtained by fusion, determine the neighborhood of the target pixel and determine each pixel in the neighborhood as the pixel adjacent to the target pixel.

[0121] Specifically, the server can determine the size of the neighborhood window corresponding to the current level, and generate at least one neighborhood window that matches the size of the neighborhood window in the feature map input at the current level. When any neighborhood window covers the directly adjacent pixels of the target pixel in the feature map input at the current level and reaches the maximum number, the area covered by the neighborhood window that reaches the maximum number is determined as the neighborhood of the target pixel.

[0122] The size of the neighborhood window corresponding to the level is directly determined based on the preset number of adjacent pixels. It describes the window size when all pixels in the neighborhood are considered as adjacent pixels. For example, if the number of adjacent pixels is 8, the initial window size is 3*3.

[0123] Step 510: Under various attention feature extraction methods, determine the correlation between the target pixel and its neighboring pixels, and determine the deviation parameters of each neighboring pixel based on the positional deviation between the target pixel and its neighboring pixels.

[0124] Step 512: Under each attention feature extraction method, the correlation and deviation parameters of each adjacent pixel are added together to obtain the attention weight of each adjacent pixel under each attention feature extraction method.

[0125] Step 514: Under each attention feature extraction method, perform feature weighting processing on each neighboring pixel based on the attention weight of each neighboring pixel.

[0126] Step 516: Fuse the pixel features obtained by feature weighting under each attention feature extraction method, update the target pixel according to the fused pixel features, and obtain the feature map of the pixel.

[0127] Step 518: Perform pixel value optimization processing on the feature map of the updated pixels to obtain an intermediate feature map.

[0128] The pixel value optimization process includes at least one of pixel value normalization, MLP processing, and cross-connection fusion processing.

[0129] Step 520: For each intermediate pixel in the intermediate feature map, select an intermediate pixel from the neighborhood of the targeted intermediate pixel at a preset pixel distance as the intermediate pixel adjacent to the targeted intermediate pixel.

[0130] The preset pixel distance is matched with the hole coefficient set for the layer, that is, the selected pixel that is adjacent to the middle pixel of the target middle pixel is the pixel in the neighborhood of the target middle pixel that meets the hole adjacency condition.

[0131] Step 522: Based on the positional deviation between the targeted intermediate pixel and each of its neighboring intermediate pixels, attention features are extracted for each of the neighboring intermediate pixels.

[0132] Step 524: Update the targeted intermediate pixel based on the extracted attention features to obtain the intermediate feature map of the updated pixel.

[0133] Step 526: Perform cross-connection fusion on the intermediate feature map of the updated pixel and the intermediate feature map obtained by pixel value optimization, and determine the first-level output feature map based on the obtained fused feature map.

[0134] Step 528: Starting from the second level, for each level, perform encoding processing based on the feature map output by the previous level to obtain the feature map output by the current level. When the encoding processing of the last level is completed, the encoded feature maps of multiple levels are obtained.

[0135] Specifically, starting from the second level, at each level where the current level meets the downsampling condition, the feature map output from the previous level is divided into multiple feature blocks of the same size. Pixels at the same position between these multiple feature blocks are extracted. The pixels extracted at each position are used to form a pixel matrix corresponding to that position. The pixel matrices are then stacked by channels, and the pixel values ​​of the feature map obtained by channel stacking are optimized to obtain a sampled feature map. The encoding process for the current level is then performed based on the sampled feature map to obtain the feature map output for that level.

[0136] When the current layer does not meet the downsampling condition, the encoding process of the current layer is directly performed based on the feature map output by the previous layer to obtain the feature map output by the current layer.

[0137] It is understandable that the encoding process for each level starting from the second level is the same as that for the first level, as can be seen in steps 508 to 526.

[0138] Step 530: Decode the encoded feature map to obtain the image segmentation result corresponding to the target image.

[0139] In practical applications, steps 506 to 530 above can be implemented by a trained image segmentation model. That is, the target interaction information map after channel overlay and the target image are input into the trained image segmentation model. The image segmentation model implements steps 506 to 530 above and outputs the image segmentation result corresponding to the target image.

[0140] In the above embodiments, since attention features are extracted based on positional deviation during the encoding process of obtaining the encoded feature map, and pixels are updated according to the extracted attention features, the relationship between pixels can be used to determine the path of interactive information propagation and perceive the long-distance dependency between pixels, so that the obtained encoded feature map has better feature expression ability. Therefore, when decoding the encoded feature map, a more accurate image segmentation result can be obtained.

[0141] In some embodiments, such as Figure 6 As shown, an image segmentation model processing method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0142] Step 602: Using an image segmentation model, extract initial feature maps of samples based on the sample images and the corresponding sample interaction information maps, and then fuse the extracted initial feature maps; the sample interaction information map records the positions of the pixels affected by the interactions performed on the sample images.

[0143] The sample images are the data used to train the image segmentation model. Each sample image has a corresponding sample label, and the image segmentation model can be trained in a supervised manner based on these sample images. The image segmentation model is used to perform image segmentation processing on the input image. The image segmentation model can be constructed with different structures according to actual needs; for example, image segmentation models with different layer structures can be constructed based on the Transformer algorithm.

[0144] Specifically, the server can acquire sample images and corresponding sample interaction information maps, and then extract initial sample feature maps based on the sample images and corresponding sample interaction information maps using an image segmentation model. Specifically, the sample images and corresponding sample interaction information maps can be input into the image segmentation model, and the feature extraction structure in the image segmentation model can extract features from the sample images and corresponding sample interaction information maps to obtain initial sample feature maps. Feature extraction can be achieved, for example, through downsampling. After obtaining the initial sample feature maps, the image segmentation model can fuse the extracted initial sample feature maps.

[0145] Step 604: Using an image segmentation model, perform at least one level of encoding processing based on the fused sample feature map to obtain a sample encoded feature map output by at least one level of encoding processing; wherein, in each level of encoding processing, for a pixel in the sample feature map input at the current level, determine at least one neighboring pixel from the sample feature map input at the current level, extract attention features from each neighboring pixel based on the positional deviation between each neighboring pixel, update the target pixel according to the extracted attention features, and determine the sample feature map output at the current level based on the feature map of the updated pixel.

[0146] Specifically, after fusing the extracted initial feature maps of the samples, the image segmentation model can further perform at least one level of encoding processing based on the fused sample feature maps to obtain a sample encoded feature map output by at least one level of encoding processing. The at least one level can be one level or at least two levels. When the at least one level includes at least two levels, the levels are connected in a concatenated manner. In the first level of encoding processing, the fused sample feature maps are directly encoded. From the second level onwards, each level performs encoding processing based on the sample feature maps output by the previous level. When the encoding processing of the last level is completed, the sample encoded feature map output by the encoding processing is obtained.

[0147] Step 606: Decode the sample encoding feature map using an image segmentation model to obtain the sample image segmentation result corresponding to the sample image.

[0148] Step 608: Update the image segmentation model based on the sample image segmentation results and continue training until the training ends.

[0149] Specifically, after obtaining the sample image segmentation results, the server can calculate the loss based on the difference between the sample image segmentation results and the sample labels. Based on the calculated loss, the server adjusts the model parameters of the image segmentation model and continues training the adjusted model until the training termination condition is met. The training termination condition can be any one of the following: the training duration reaches a preset duration, the loss reaches a minimum, or the number of iterations reaches a preset number. The image segmentation model obtained at the end of training can be used to segment the target image.

[0150] The aforementioned image segmentation model processing method extracts initial sample feature maps based on the sample image and the corresponding sample interaction information map through an image segmentation model. The extracted initial sample feature maps are then fused. At least one level of encoding processing is performed based on the fused sample feature maps to obtain sample encoded feature maps output by at least one level of encoding processing. In each level of encoding processing, for a pixel in the input sample feature map of that level, at least one neighboring pixel is determined from the input sample feature map of that level. Attention features are extracted from each neighboring pixel based on the positional deviation between the neighboring pixels, and then... The attention feature update targets the pixel, and the output sample feature map of the layer is determined based on the feature map of the updated pixel. Since the image segmentation model extracts attention features based on positional deviation during the encoding process of obtaining the sample encoding feature map, and updates pixels based on the extracted attention features, it can use the relationship between pixels to determine the path of interactive information propagation, achieving a large receptive field. This makes the obtained sample encoding feature map have better feature representation ability, so when decoding the sample encoding feature map, a more accurate image segmentation result can be obtained. By updating the image segmentation model with this image segmentation result, a better performing image segmentation model can be obtained.

[0151] In some embodiments, initial feature maps of samples are extracted based on sample images and corresponding sample interaction information maps, including: acquiring sample images and corresponding sample interaction information maps, and acquiring historical sample segmentation images obtained by performing historical segmentation processing on sample images; overlaying the historical sample segmentation images and sample interaction information maps by channels to obtain a channel-overlayed sample interaction information map; and extracting initial feature maps of samples from the channel-overlayed sample interaction information map and the sample images, respectively.

[0152] The historical sample segmentation image is obtained by performing historical segmentation processing on the sample image at least once. The sample interaction information map after channel overlay is the image obtained by overlaying the historical sample segmentation image and the sample interaction information map in the channel direction.

[0153] Specifically, the server can overlay the historical sample segmentation image and the sample interaction information map through channels to obtain the channel-overlayed sample interaction information map. Then, the server can downsample the channel-overlayed sample interaction information map and the sample image in the same way to obtain two initial feature maps of the samples.

[0154] In the above embodiments, by overlaying the historical sample segmentation image and the sample interaction information map through channels, a sample interaction information map after channel overlay is obtained. The image segmentation model can combine the historical sample segmentation image to perform image segmentation, which further improves the accuracy of the image segmentation results.

[0155] In some embodiments, historical sample segmentation images are obtained through a sample segmentation step, which includes the following steps:

[0156] 1. Obtain the first historical interaction information map and the initial sample segmentation image corresponding to the sample image.

[0157] The first historical interaction information graph records the position of the pixel affected by the first interaction with the sample image. The initial sample segmentation image is obtained by initializing the pixel values ​​of the sample image.

[0158] In practical applications, initializing the pixel values ​​of a sample image can be achieved by setting each pixel in the sample image to 0 to obtain an initial sample segmentation image.

[0159] 2. Using an image segmentation model, perform initial image segmentation based on the sample image, the first historical interaction information map, and the initial sample segmentation image to obtain the intermediate sample segmentation image.

[0160] Specifically, the server can input the sample image, the first historical interaction information map, and the initial sample segmentation image into the image segmentation model. The image segmentation model will overlay the first historical interaction information map and the initial sample segmentation image through channels, and then downsample the image obtained by channel overlay and the sample image in the same way. The sample feature maps obtained by downsampling will be fused, and multi-level encoding processing will be performed based on the fused sample feature maps to obtain sample encoded feature maps. The sample encoded feature maps will be decoded to complete the first segmentation process and obtain the intermediate sample segmentation image.

[0161] 3. In each image segmentation process after the first image segmentation process, the intermediate sample segmentation image obtained from the previous image segmentation process, the second historical interaction information map corresponding to the sample image, and the sample image are input into the image segmentation model to obtain the intermediate sample segmentation image for each image segmentation process.

[0162] 4. When the image segmentation process reaches the preset number of times, the intermediate sample segmentation image obtained from the last image segmentation process is used as the historical sample segmentation image.

[0163] The second historical interaction information map records the positions of pixels affected by subsequent interactions with the sample image. The preset number of interactions can be set as needed, for example, it can be 3 times.

[0164] Specifically, after the initial image segmentation is completed, the server can continue to perform one or more image segmentation processes using the image segmentation model. In each image segmentation process, the intermediate sample segmentation image obtained from the previous image segmentation process, the second historical interaction information map corresponding to the sample image, and the sample image are input into the image segmentation model to obtain the intermediate sample segmentation image for each image segmentation process. When the image segmentation process reaches a preset number of times, the server can use the intermediate sample segmentation image obtained from the last image segmentation process as the historical sample segmentation image.

[0165] In the above embodiments, historical sample segmentation images are obtained through multiple image segmentation processes, which can yield more accurate historical sample segmentation images. Model training is performed using historical sample segmentation images with higher accuracy, and the trained model has better performance.

[0166] In some embodiments, the first historical interaction information map is generated through a first simulated interaction step, which includes: acquiring a mask image of a sample image; the mask image contains the classification labels of each pixel in the sample image; determining the region of the target to be segmented in the sample image from the sample image based on the classification labels of each pixel in the mask image; selecting at least one pixel from the pixel set; and determining the first historical interaction information map based on the position of the selected pixel and its corresponding classification label.

[0167] The mask image of the sample image serves as its sample label, containing the classification label for each pixel. These labels fall into two categories: foreground, indicating that the pixel belongs to the target to be segmented, and background, indicating that the pixel does not belong to the target. The target to be segmented is the object in the sample image that needs to be segmented; it can be a living or non-living object. Living objects include animals and plants, while non-living objects include cars and houses.

[0168] Specifically, the server can determine the region of the target to be segmented in the sample image from the sample image based on the classification labels of each pixel recorded in the mask image, select at least one pixel in the region of the target to be segmented to simulate positive interaction, and determine the first historical interaction information map based on the position of the selected pixel and the corresponding classification label.

[0169] In practical applications, the server can select pixels from the center of the region containing the target to be segmented with a first probability, and from the edge of the region with a second probability, where the first probability is greater than the second probability. Optionally, the server can also select pixels from a pixel region outside the region containing the target to be segmented with a third probability to simulate negative interactions.

[0170] In some specific embodiments, after selecting a pixel, the server can take a circular area centered on each pixel with a radius of a preset number of pixels, set the value to 1 inside the area, and set the value to 0 in the rest, to generate a first historical interactive information map composed of a foreground channel and a background channel. If the selected pixels include pixels with the category label of foreground, the image content of the foreground channel is determined based on that pixel. If the selected pixels include pixels with the category label of background, the image content of the background channel is determined based on that pixel.

[0171] In the above embodiments, by obtaining the mask image of the sample image, the region where the target to be segmented is located can be determined according to the classification label of each pixel recorded in the mask image. At least one pixel is selected in the region where the target to be segmented is located, simulating the interaction with the sample image, quickly generating the interaction information map, and improving the generation efficiency of the interaction information map.

[0172] In some embodiments, the second historical interaction information map is generated through a second simulated interaction step, which includes: acquiring a mask image of the sample image; the mask image recording the classification label of each pixel in the sample image; comparing the classification result of each pixel in the intermediate sample segmentation image obtained from the previous image segmentation process with the classification label of each pixel in the mask image, and determining the error region in the intermediate sample segmentation image obtained from the previous image segmentation process based on the comparison result; selecting at least one pixel from the center position of the error region, and determining the second historical interaction information map based on the position of the selected pixel and the corresponding classification label.

[0173] The classification results include two categories: one indicating pixels as foreground and the other indicating pixels as background. The comparison results include matching and mismatching pixels; the region formed by mismatched pixels is the error region.

[0174] Specifically, the intermediate sample segmentation image is also a mask image, where each pixel has a corresponding classification result. The server can compare the classification result of each pixel in the intermediate sample segmentation image with the classification label of each pixel in the mask image. If the classification result matches the classification label, the comparison is consistent; if the classification result does not match the classification label, the comparison is inconsistent. Based on the pixels with inconsistent comparisons, the error region is determined from the intermediate sample segmentation image. At least one pixel is selected from the center of the error region, and then the second historical interaction information map can be determined based on the position of the selected pixel and the corresponding classification label.

[0175] In some embodiments, the sample interaction information map can be obtained in the same way as the second historical interaction information map, and the intermediate sample segmentation image used in the process of obtaining the sample interaction information map is the intermediate sample segmentation image obtained in the last image segmentation process in the sample segmentation step.

[0176] In the above embodiments, by determining the error region, selecting at least one pixel from the center of the error region, and determining the second historical interaction information map based on the position of the selected pixel and the corresponding classification label, the interaction with the sample image can be better simulated, thereby obtaining a more accurate historical sample segmentation image.

[0177] In some specific embodiments, the image segmentation method and image segmentation model processing method of this application can be applied to image matting scenarios. The terminal can be equipped with an image processing application capable of image matting, through which the user can fine-tune the matting results. (Reference) Figure 7 The terminal can obtain the target image and user click data through the image processing application, and send the target image and user click data to the server. The server deploys an image segmentation model trained using the image segmentation model processing method of this application. The server uses the image segmentation model to implement the image segmentation method provided in the embodiments of this application to obtain the image segmentation result, and then returns the image segmentation result to the terminal. The terminal uses the image processing application to perform image matting based on the image segmentation result. (Reference) Figure 8 Figure (a) in Figure (8) is an example of image segmentation performed using the image segmentation method of this application. Figure (a) shows the target image and the positive class points (black) and negative class points (white) given by the user. Figure (b) in Figure (8) shows the image segmentation result. The image segmentation result contains the object clicked by the positive class points, but does not contain the background clicked by the negative class points.

[0178] In this embodiment, the image segmentation method and image segmentation model processing method are implemented based on the dilated adjacency attention mechanism and interactive segmentation task. Interactive segmentation task can extract user-labeled target object regions from images. A common user interaction type is click interaction, where the user clicks positive points in the target region and negative points in the background region, aiming to achieve effective segmentation of the target object with the fewest clicks. Currently, most deep learning-based methods adopt existing semantic segmentation model frameworks. These frameworks generally include two parts: an encoder and a decoder. The encoder typically uses a Convolutional Neural Network (CNN) to extract image features for the decoder to perform segmentation. However, while the CNN framework is suitable for general semantic segmentation tasks, it has limitations for interactive segmentation tasks. It cannot propagate the semantic information contained in user clicks to distant parts of the image. That is, CNN can only determine the propagation range of semantic information contained in the click map by detecting object boundaries. Generally, it can only propagate a small range and cannot perceive or utilize image information outside this range. The segmentation results at the boundaries are usually not refined enough.

[0179] To address this shortcoming of CNNs, this application proposes using a Dilated Neighbour Attention (DiNA) mechanism to replace the encoder framework. This mechanism utilizes the relationships between pixels to determine the information propagation path, achieving a large receptive field and superior segmentation performance with fast processing and low memory consumption. The structure and training process of the image segmentation model are described below.

[0180] I. Structure of Image Segmentation Model

[0181] In this embodiment, the overall network framework of the image segmentation model is as follows: Figure 9 As shown. First, the image is downsampled to obtain the result, and then added to the concatenated Click map (interactive infographic) and Prev.mask (historical segmentation image) after being downsampled in the same way. This is then processed through N1 DiNA modules. Finally, it undergoes three more downsampling steps (N... i The final segmentation prediction result is obtained by using one DiNA module (i = 2, 3, 4) layer and one decoder. Within each DiNA module, one level of encoding processing as described in the previous embodiment is performed.

[0182] For each DiNA module, its structure is as follows: Figure 10As shown, the input passes through LayerNorm (layer normalization), NA (Neighbor Attention), cross-connection summation, LayerNorm, MLP, cross-connection summation, LayerNorm, DiNA (Dilated NA), cross-connection summation, LayerNorm, and MLP in sequence. Attention feature extraction can be performed in both the DiNA and NA layers. Taking the DiNA layer as an example, the input feature X∈R... hw×d Where h and w are the length and width of the input feature, respectively, and d is the number of feature channels; X is passed through three fully connected layers to obtain Q, K, V∈R. hw×c For pixels i and j, their deviation parameter is B(i,j). Let k be the pre-defined number of neighboring pixels, and δ be the hole coefficient. Then the i-th position of the output of a DiNA layer is:

[0183]

[0184] in:

[0185]

[0186] in, Let represent the pixels adjacent to pixel i, where pixels i and j satisfy j mod δ = i mod δ. In practical applications, k = 48 can be chosen. NA is a degenerate version of DiNA where δ = 1. In NA, adjacent pixels are every pixel in the neighborhood, while in DiNA, adjacent pixels are pixels that satisfy the hole adjacency condition. Similar to the multi-head self-attention mechanism, different numbers of heads (num_heads) can be set for DiNA and NA modules.

[0187] For downsampling, the initial downsampling is a convolution operation with a kernel size of 4 and a stride of 4. Other downsampling operations are Patch Merging operations, which halve the length and width of the input while quadrupling the channel dimension. The decoder can use a UPerHead structure.

[0188] In implementation, different network layer numbers and other parameters can be selected according to the desired model size. In a smaller network used in the experiment, N1=2, N2=2, N3=6, N4=2. For each DiNA in the four layers, the hole coefficients are set to [1,16], [1,8], [1,2,1,3,1,4], [1,2], respectively. The num_heads in the four layers are set to 3, 6, 12, 24, respectively.

[0189] II. Training Process of Image Segmentation Model

[0190] First, the training dataset and data preprocessing were established. 8498 images and 20172 foreground annotations from the publicly available SBD dataset [3] were used as training data, some of which contained multiple objects, i.e., multiple foreground annotations. The pixel value range of the training data was normalized to the threshold range [0,1]. Each training image was randomly cropped into 256x256 image blocks, and then randomly horizontally and vertically mirrored with a probability of 0.3.

[0191] At the start of training, the main network portion (excluding initial downsampling and the decoder) is initialized using weights pre-trained on ImageNet. The loss function is as follows:

[0192]

[0193] in, For the final output of the network, y gt The true labels of the training samples; This is the normalized focal loss.

[0194] During training, the network parameters are updated and optimized using the Adam algorithm (Adaptive Moment Estimation). In each iteration, the prediction error is calculated and backpropagated to the convolutional neural network model, the gradient is calculated, and the model parameters are updated. Training is performed end-to-end, with an initial learning rate of 5 × 10⁻⁶. -5 The training consists of 100 epochs, with the learning rate decreasing by a factor of 10 at epochs 80 and 90. In each iteration, the batch size is 16, and the patch size is 256x256. Generating training samples requires simulating clicks, i.e., generating the next click at the center of the region of maximum error.

[0195] After training is complete, the trained image segmentation model can be tested. For detailed training and testing procedures, please refer to [link / reference needed]. Figure 11 After the process begins, it is determined whether it is training or testing. If it is training, the model parameters are initialized, then sample images are loaded, and preprocessing and data augmentation are performed on the sample images. Several simulated clicks are generated from the sample images obtained from the preprocessing to obtain click maps (interactive information maps). Then, the segmentation result is predicted by the image segmentation model. After obtaining the predicted segmentation result, the model parameters in the image segmentation model are updated according to the loss function and the Adam algorithm. It is then determined whether the preset number of iterations has been reached. If the preset number of iterations has not been reached, iterative training continues. If the preset number of iterations has been reached, training ends, and the trained image segmentation model is saved.

[0196] Continue to refer to Figure 11 If it is determined to be a test, the trained model is loaded, and then the image to be tested is loaded. The user clicks on the foreground or background class on the image. The image segmentation model outputs the current segmentation result based on the image to be tested, the user's click, and the segmentation result of the previous step. It determines whether the foreground target object has been segmented. If yes, the final segmentation result is obtained. If no, the user continues to click on the foreground or background class on the image, and the image segmentation model continues to perform image segmentation until the foreground target object is segmented.

[0197] The above embodiments address the shortcomings of traditional convolutional methods for interactive segmentation applications—the inability to perceive long-distance pixels and poor information flow. This can be seen from... Figure 12 This can be proven in the text. Figure 12 The text provides a visual representation of the spread of click information. Figure 12 In Figure (a), using the traditional convolutional ResNet, the information at the click point is not propagated in the first layer (L1) of the network, and in the first layer (L4) of the network, the information at the click point can only be propagated to a very small area (the white shaded area in the figure). However, the DiNAT (DiNATransformer) model composed of the dilated adjacency attention module provided in this application embodiment can propagate the information at the click point to the entire figure in the shallow layers of the network. For example, in the L1 layer, the information at the click point can be propagated to the lower half of the figure in the figure, and in the L1 layer, the information at the click point can be propagated to almost the entire body of the figure in the figure.

[0198] In some specific embodiments, the image segmentation method and image segmentation model processing method of this application can also be applied to the dataset annotation task that provides the image segmentation task. The annotation of the target object can be completed efficiently with just a click. This can collect high-quality training data annotation more efficiently for tasks such as medical images where it is difficult to obtain high-quality annotations. Specifically, the image segmentation model is first trained using the image segmentation model processing method of this application. Then, for the target image in the dataset to be labeled, a corresponding target interaction information map is generated by clicking. The trained image segmentation model then performs the following steps: initial feature maps are extracted from the target image and the target interaction information map respectively, and the extracted initial feature maps are fused; the target interaction information map records the positions of pixels affected by the interaction with the target image; at least one level of encoding processing is performed based on the fused feature map to obtain an encoded feature map output from at least one level of encoding processing; in each level of encoding processing, for a pixel in the input feature map of the current level, at least one neighboring pixel is determined from the input feature map of the current level; attention features are extracted from each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel; the target pixel is updated based on the extracted attention features; the output feature map of the current level is determined based on the updated pixel's feature map; the encoded feature map is decoded to obtain the image segmentation result corresponding to the target image. Finally, pixel-level labeling is achieved based on the classification results of each pixel in the image segmentation result. For details on the specific implementation of image segmentation methods and image segmentation model processing methods, please refer to the above embodiments, which will not be repeated here.

[0199] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0200] Based on the same inventive concept, this application also provides an image segmentation apparatus for implementing the image segmentation method described above, and an image segmentation model processing apparatus for implementing the image segmentation model processing method described above. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more image segmentation apparatuses and image segmentation model processing apparatus embodiments provided below can be found in the limitations of the image segmentation method and image segmentation model processing method described above, and will not be repeated here.

[0201] In some embodiments, such as Figure 13 As shown, an image segmentation apparatus 1300 is provided, comprising:

[0202] The image acquisition module 1302 is used to acquire a target image and a corresponding target interaction information map, extract initial feature maps based on the target image and the target interaction information map respectively, and fuse the extracted initial feature maps; the target interaction information map records the position of the pixels affected by the interaction performed on the target image;

[0203] The encoding module 1304 is used to perform at least one level of encoding processing based on the feature map obtained by fusion, to obtain an encoded feature map output by at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module 1304 is further used to, for a pixel in the feature map input at the current level, determine at least one neighboring pixel from the feature map input at the current level, extract attention features for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, update the target pixel according to the extracted attention features, and determine the feature map output at the current level based on the feature map of the updated pixel.

[0204] The decoding module 1306 is used to decode the encoded feature map to obtain the image segmentation result corresponding to the target image.

[0205] The aforementioned image segmentation device, during the encoding process of obtaining the coded feature map, extracts attention features based on positional deviation and updates pixels according to the extracted attention features. It can utilize the relationship between pixels to determine the path of interactive information propagation and perceive long-distance dependencies between pixels, thus giving the obtained coded feature map a better feature representation capability. Therefore, when decoding the coded feature map, it can obtain image segmentation results with higher accuracy.

[0206] In some embodiments, the encoding module is further configured to determine the size of the neighborhood window corresponding to the current level; generate at least one neighborhood window matching the size of the neighborhood window in the feature map input at the current level; when any neighborhood window covers the pixels directly adjacent to the target pixel in the feature map input at the current level and reaches the maximum number, at least one of the pixels covered by the neighborhood window that reaches the maximum number is determined as the pixel adjacent to the target pixel.

[0207] In some embodiments, the encoding module is further configured to obtain a preset number of adjacent pixels, determine an initial window size based on the number of adjacent pixels; obtain a hole coefficient set for the current level, determine a window expansion size based on the hole coefficient; and determine the neighborhood window size corresponding to the current level based on the initial window size and the window expansion size.

[0208] In some embodiments, the hole coefficient is greater than 1; the encoding module is further configured to perform modulo operation on the position parameters of the target pixel in the feature map of the input layer according to the hole coefficient, to obtain the modulo operation result corresponding to the target pixel; perform modulo operation on the position parameters of each pixel covered by the maximum number of neighborhood windows according to the hole coefficient, to obtain the modulo operation result corresponding to each covered pixel; select at least one pixel from the pixels covered by the neighborhood windows whose modulo operation result matches the modulo operation result corresponding to the target pixel, and determine the selected pixel as the pixel adjacent to the target pixel.

[0209] In some embodiments, at least one pixel adjacent to the targeted pixel includes each pixel in the neighborhood corresponding to the targeted pixel; the encoding module is further configured to perform pixel value optimization processing on the feature map of the updated pixel to obtain an intermediate feature map; for each intermediate pixel in the intermediate feature map, select an intermediate pixel from the neighborhood of the targeted intermediate pixel at a preset pixel distance as the intermediate pixel adjacent to the targeted intermediate pixel; the preset pixel distance is matched with the hole coefficient set for the level; attention features are extracted for each adjacent intermediate pixel based on the positional deviation between the targeted intermediate pixel and each of the adjacent intermediate pixels; the targeted intermediate pixel is updated according to the extracted attention features, and the feature map output for the level is determined according to the intermediate feature map of the updated pixel.

[0210] In some embodiments, the encoding module is further configured to update the targeted intermediate pixel based on the extracted attention features to obtain an intermediate feature map of the updated pixel; perform cross-connection fusion of the intermediate feature map of the updated pixel and the intermediate feature map obtained by pixel value optimization processing to obtain a fused feature map; and determine the feature map of the output layer based on the fused feature map.

[0211] In some embodiments, the encoding module is further configured to: determine the correlation between the target pixel and its neighboring pixels under various attention feature extraction methods; determine the deviation parameters of each neighboring pixel based on the positional deviation between the target pixel and its neighboring pixels; sum the correlation and deviation parameters of each neighboring pixel under each attention feature extraction method to obtain the attention weight of each neighboring pixel under each attention feature extraction method; perform feature weighting processing on each neighboring pixel based on its attention weight under each attention feature extraction method; fuse the pixel features obtained from the feature weighting processing under each attention feature extraction method; and update the target pixel based on the fused pixel features.

[0212] In some embodiments, the image acquisition module is further configured to acquire a target image and a corresponding target interaction information map, and acquire a historical segmentation image obtained by performing historical segmentation processing on the target image; overlay the historical segmentation image and the target interaction information map by channels to obtain a target interaction information map after channel overlay; and extract initial feature maps from the target interaction information map after channel overlay and the target image, respectively.

[0213] In some embodiments, at least one level includes multiple levels; the encoding module is further configured to, in the encoding process of the first level, for a pixel in the fused feature map, determine at least one pixel adjacent to the pixel in the fused feature map, extract attention features for each adjacent pixel based on the positional deviation between the pixel in the target and each adjacent pixel, update the pixel in the target according to the extracted attention features, and determine the feature map output of the first level based on the feature map of the updated pixel; for each level starting from the second level, perform encoding processing of the current level based on the feature map output of the previous level to obtain the feature map output of the current level; when the encoding process of the last level is completed, multiple levels of encoded feature maps are obtained.

[0214] In some embodiments, the encoding module is further configured to, at each level starting from the second level, divide the feature map output by the previous level into multiple feature blocks of the same size when the level satisfies the downsampling condition; extract pixels at the same position between the multiple feature blocks of the same size, and form pixel matrices corresponding to each position by combining the pixels extracted at each position; perform channel stacking on the pixel matrices, and optimize the pixel values ​​of the feature map obtained by channel stacking to obtain a sampled feature map; and perform encoding processing on the sampled feature map for the level to obtain the feature map output by the level.

[0215] In some embodiments, such as Figure 14 As shown, an image segmentation model processing apparatus 1400 is provided, comprising:

[0216] The image acquisition module 1402 is used to extract initial feature maps of samples based on the sample image and the corresponding sample interaction information map through an image segmentation model, and to fuse the extracted initial feature maps of samples; the sample interaction information map records the position of the pixels affected by the interaction performed on the sample image.

[0217] The encoding module 1404 is used to perform at least one level of encoding processing based on the fused sample feature map through an image segmentation model to obtain a sample encoded feature map output by at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module 1404 is further used to, for a pixel in the sample feature map input at the current level, determine at least one neighboring pixel from the sample feature map input at the current level, extract attention features for each neighboring pixel based on the positional deviation between each neighboring pixel, update the target pixel according to the extracted attention features, and determine the sample feature map output at the current level according to the feature map of the updated pixel.

[0218] The decoding module 1406 is used to decode the sample encoded feature map through the image segmentation model to obtain the sample image segmentation result corresponding to the sample image;

[0219] The model update module 1408 is used to update the image segmentation model based on the sample image segmentation results and continue training until the training ends.

[0220] The aforementioned image segmentation model processing device, because the image segmentation model extracts attention features based on positional deviation during the encoding process of obtaining the sample encoding feature map, and updates pixels according to the extracted attention features, can use the relationship between pixels to determine the propagation path of interactive information, thereby achieving a large receptive field. This makes the obtained sample encoding feature map have better feature representation capabilities, so when decoding the sample encoding feature map, a more accurate image segmentation result can be obtained. By updating the image segmentation model with this image segmentation result, a higher-performance image segmentation model can be obtained.

[0221] In some embodiments, the image acquisition module 1402 is further configured to acquire a sample image and a corresponding sample interaction information map, and acquire a historical sample segmentation image obtained by performing historical segmentation processing on the sample image; overlay the historical sample segmentation image and the sample interaction information map by channels to obtain a channel-overlayed sample interaction information map; and extract the initial feature map of the sample from the channel-overlayed sample interaction information map and the sample image, respectively.

[0222] In some embodiments, the image acquisition module 1402 is further configured to acquire a first historical interaction information map and an initial sample segmentation image corresponding to the sample image; the first historical interaction information map records the position of the pixel affected by the first interaction performed on the sample image, and the initial sample segmentation image is obtained by initializing the pixel values ​​of the sample image; the image segmentation model performs a first image segmentation process based on the sample image, the first historical interaction information map, and the initial sample segmentation image to obtain an intermediate sample segmentation image; in each image segmentation process after the first image segmentation process, the intermediate sample segmentation image obtained from the previous image segmentation process, the second historical interaction information map corresponding to the sample image, and the sample image are input into the image segmentation model to obtain an intermediate sample segmentation image for each image segmentation process; the second historical interaction information map records the position of the pixel affected by the subsequent interaction performed on the sample image; when the image segmentation process reaches a preset number of times, the intermediate sample segmentation image obtained from the last image segmentation process is used as the historical sample segmentation image.

[0223] In some embodiments, the image segmentation model processing apparatus further includes a first simulated interaction module, used to acquire a mask image of a sample image; the mask image contains the classification labels of each pixel in the sample image; based on the classification labels of each pixel in the mask image, the region of the target to be segmented in the sample image is determined from the sample image; at least one pixel is selected from the region of the target to be segmented, and a first historical interaction information map is determined based on the position of the selected pixel and the corresponding classification label.

[0224] In some embodiments, the image segmentation model processing apparatus further includes a second simulated interaction module for acquiring a mask image of the sample image; the mask image contains the classification labels of each pixel in the sample image; the classification results of each pixel in the intermediate sample segmentation image obtained from the previous image segmentation process are compared with the classification labels of each pixel in the mask image, and an error region in the intermediate sample segmentation image obtained from the previous image segmentation process is determined based on the comparison results; at least one pixel is selected from the center position of the error region, and a second historical interaction information map is determined based on the position of the selected pixel and the corresponding classification label.

[0225] Each module in the aforementioned image segmentation device and image segmentation model processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0226] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores sample image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image segmentation method or an image segmentation model processing method.

[0227] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image segmentation method or an image segmentation model processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0228] Those skilled in the art will understand that Figure 15 , Figure 16The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0229] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described image segmentation method or image segmentation model processing method.

[0230] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described image segmentation method or image segmentation model processing method.

[0231] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described image segmentation method or image segmentation model processing method.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0233] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0234] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0235] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image segmentation method, characterized in that, The method includes: Acquire a target image and a corresponding target interaction information map, and acquire a historical segmentation image obtained by performing historical segmentation processing on the target image; the target interaction information map records the position of the pixels affected by the interaction performed on the target image; the target interaction information map includes a foreground channel map and a background channel map; the pixels in the foreground region of the foreground channel map are different from the pixels in other regions, and the pixels in the background region of the background channel map are different from the pixels in other regions; The historical segmented image and the target interaction information map are overlaid by channels to obtain the target interaction information map after channel overlay. Initial feature maps are extracted from the target interaction information map after the channels are superimposed and the target image, respectively, and the extracted initial feature maps are fused together; At least one level of encoding processing is performed on the feature map obtained by fusion to obtain an encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, for a pixel in the feature map input at the current level, at least one neighboring pixel is determined from the feature map input at the current level, attention features are extracted for each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, the target pixel is updated according to the extracted attention features, and the feature map output at the current level is determined according to the feature map of the updated pixel. The encoded feature map is decoded to obtain the image segmentation result corresponding to the target image.

2. The method according to claim 1, characterized in that, Determining at least one neighboring pixel of the target pixel from the feature map input at the current level includes: Determine the size of the neighboring window corresponding to the current level; In the feature map of the input at the current level, at least one neighborhood window matching the size of the neighborhood window is generated; When any neighborhood window covers the directly adjacent pixels of the target pixel in the feature map of the input at the current level and reaches the maximum number, at least one of the pixels covered by the neighborhood window that reaches the maximum number is determined as the pixel adjacent to the target pixel.

3. The method according to claim 2, characterized in that, Determining the neighborhood window size corresponding to the current level includes: Obtain the preset number of adjacent pixels, and determine the initial window size based on the number of adjacent pixels; Obtain the hole coefficient set for the current level, and determine the window expansion size based on the hole coefficient; Based on the initial window size and the window expansion size, determine the neighboring window size corresponding to the current level.

4. The method according to claim 3, characterized in that, The void coefficient is greater than 1; Determining at least one of the pixels covered by the maximum number of the neighborhood windows as the pixels adjacent to the target pixel includes: The position parameters of the target pixel in the feature map of the input layer are moduloed according to the hole coefficient to obtain the modulo operation result corresponding to the target pixel. The position parameters of each pixel covered by the maximum number of neighborhood windows are moduloed according to the hole coefficient to obtain the modulo operation result corresponding to each covered pixel. From the pixels covered by the neighborhood window, select at least one pixel whose modulo operation result matches the modulo operation result corresponding to the target pixel, and determine the selected pixel as the pixel adjacent to the target pixel.

5. The method according to claim 1, characterized in that, The at least one pixel adjacent to the pixel being targeted includes each pixel in the neighborhood corresponding to the pixel being targeted; The step of determining the output feature map of the current layer based on the feature map of the updated pixels includes: Pixel value optimization is performed on the feature map of the updated pixels to obtain an intermediate feature map; For each intermediate pixel in the intermediate feature map, an intermediate pixel is selected from the neighborhood of the targeted intermediate pixel at a preset pixel distance as the intermediate pixel adjacent to the targeted intermediate pixel; the preset pixel distance matches the hole coefficient set for the level. Attention features are extracted from each of the adjacent intermediate pixels based on the positional deviation between the targeted intermediate pixel and each of the adjacent intermediate pixels. The intermediate pixel is updated based on the extracted attention features, and the feature map of the layer at which it is output is determined based on the intermediate feature map of the updated pixel.

6. The method according to claim 5, characterized in that, The step of updating the targeted intermediate pixel based on the extracted attention features, and determining the feature map of the output layer based on the intermediate feature map of the updated pixel, includes: The intermediate pixel is updated based on the extracted attention features to obtain the intermediate feature map of the updated pixel. The intermediate feature map of the updated pixel and the intermediate feature map obtained by pixel value optimization are fused across connections to obtain a fused feature map; The feature map output at the current level is determined based on the fused feature map.

7. The method according to claim 1, characterized in that, The step of extracting attention features from each neighboring pixel based on the positional deviation between the targeted pixel and each of its neighboring pixels, and updating the targeted pixel based on the extracted attention features, includes: Under various attention feature extraction methods, the correlation between the target pixel and its neighboring pixels is determined, and the deviation parameters of each neighboring pixel are determined based on the positional deviation between the target pixel and its neighboring pixels. For each attention feature extraction method, the correlation and deviation parameters of each adjacent pixel are added together to obtain the attention weight of each adjacent pixel under each attention feature extraction method. Under each attention feature extraction method, the features of each neighboring pixel are weighted based on the attention weight of each neighboring pixel. The pixel features obtained by weighted processing under various attention feature extraction methods are fused, and the target pixel is updated based on the fused pixel features.

8. The method according to any one of claims 1 to 7, characterized in that, The at least one level includes multiple levels; the encoding processing based on the fused feature map at least one level to obtain the encoded feature map output by the encoding processing at least one level includes: In the first-level encoding process, for a pixel in the fused feature map, at least one neighboring pixel is determined from the fused feature map. Attention features are extracted from each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel. The target pixel is updated according to the extracted attention features. The first-level output feature map is determined based on the feature map of the updated pixel. Starting from the second level, for each level, the encoding process of the current level is performed based on the feature map output by the previous level to obtain the feature map output by the current level. Once the final level of encoding is completed, the encoded feature maps of the multiple levels are obtained.

9. The method according to claim 8, characterized in that, For each level starting from the second level, the encoding feature map of the current level is obtained based on the feature map output by the previous level, including: Starting from the second level, at each level where the level meets the downsampling condition, the feature map output by the previous level is divided into multiple feature blocks of the same size. Extract the pixels that are at the same position among the multiple feature blocks of the same size, and form a pixel matrix corresponding to each position by combining the extracted pixels at each position. The pixel matrices are stacked by channels, and the pixel values ​​of the feature map obtained by channel stacking are optimized to obtain the sampled feature map. Based on the sampled feature map, the encoding process of the layer is performed to obtain the feature map output by the layer.

10. An image segmentation model processing method, characterized in that, The method includes: Using an image segmentation model, sample images and corresponding sample interaction information maps are obtained, and historical sample segmentation images obtained by performing historical segmentation processing on the sample images are also obtained. The historical sample segmentation images and the sample interaction information maps are overlaid by channels to obtain a channel-overlaid sample interaction information map. Initial sample feature maps are extracted from the channel-overlaid sample interaction information map and the sample images, and the extracted initial sample feature maps are fused. The sample interaction information map records the position of the pixels affected by the interaction performed on the sample image. The sample interaction information map includes a foreground channel map and a background channel map. In the foreground channel map, the pixels in the foreground region are different from the pixels in other regions, and in the background channel map, the pixels in the background region are different from the pixels in other regions. The image segmentation model performs at least one level of encoding processing based on the fused sample feature map to obtain a sample encoded feature map output by the at least one level of encoding processing. In each level of encoding processing, for a pixel in the sample feature map input to the current level, at least one neighboring pixel is determined from the sample feature map input to the current level. Attention features are extracted from each neighboring pixel based on the positional deviation between the neighboring pixels. The target pixel is updated according to the extracted attention features. The sample feature map output by the current level is determined based on the feature map of the updated pixel. The image segmentation model is used to decode the sample encoded feature map to obtain the sample image segmentation result corresponding to the sample image. The image segmentation model is updated based on the sample image segmentation results, and training continues until the training ends.

11. The method according to claim 10, characterized in that, The historical sample segmentation image is obtained through a sample segmentation step, which includes: Obtain the first historical interaction information map and the initial sample segmentation image corresponding to the sample image; the first historical interaction information map records the position of the pixel affected by the first interaction with the sample image, and the initial sample segmentation image is obtained by initializing the pixel values ​​of the sample image. Using the image segmentation model, an initial image segmentation process is performed based on the sample image, the first historical interaction information map, and the initial sample segmentation image to obtain an intermediate sample segmentation image. In each subsequent image segmentation process after the initial image segmentation process, the intermediate sample segmentation image obtained from the previous image segmentation process, the second historical interaction information map corresponding to the sample image, and the sample image are input into the image segmentation model to obtain the intermediate sample segmentation image for each image segmentation process; the second historical interaction information map records the position of the pixel affected by the subsequent interaction with the sample image. When the image segmentation process reaches a preset number of times, the intermediate sample segmentation image obtained from the last image segmentation process is used as the historical sample segmentation image.

12. The method according to claim 11, characterized in that, The first historical interaction information graph is generated through a first simulated interaction step, which includes: Obtain a mask image of the sample image; the mask image contains the classification label of each pixel in the sample image. Based on the classification labels of each pixel recorded in the mask image, the region of the target to be segmented in the sample image is determined from the sample image; At least one pixel is selected from the region where the target to be segmented is located, and a first historical interaction information map is determined based on the position of the selected pixel and the corresponding classification label.

13. The method according to claim 11, characterized in that, The second historical interaction information graph is generated through a second simulated interaction step, which includes: Obtain a mask image of the sample image; the mask image contains the classification label of each pixel in the sample image. The classification results of each pixel in the intermediate sample segmentation image obtained from the previous image segmentation process are compared with the classification labels of each pixel in the mask image. Based on the comparison results, the error region in the intermediate sample segmentation image obtained from the previous image segmentation process is determined. At least one pixel is selected from the center of the error region, and a second historical interaction information map is determined based on the position of the selected pixel and the corresponding classification label.

14. An image segmentation apparatus, characterized in that, The device includes: An image acquisition module is used to acquire a target image and a corresponding target interaction information map, and to acquire a historical segmentation image obtained by historical segmentation processing of the target image; to overlay the historical segmentation image and the target interaction information map by channels to obtain a channel-overlayed target interaction information map; to extract initial feature maps from the channel-overlayed target interaction information map and the target image respectively; the target interaction information map records the positions of pixels affected by the interaction performed on the target image; the target interaction information map includes a foreground channel map and a background channel map; the pixels in the foreground region of the foreground channel map are different from the pixels in other regions, and the pixels in the background region of the background channel map are different from the pixels in other regions; An encoding module is configured to perform at least one level of encoding processing based on the fused feature map to obtain an encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module is further configured to, for a pixel in the feature map input at the current level, determine at least one neighboring pixel from the feature map input at the current level, extract attention features from each neighboring pixel based on the positional deviation between the target pixel and each neighboring pixel, update the target pixel according to the extracted attention features, and determine the feature map output at the current level based on the feature map of the updated pixel; The decoding module is used to decode the encoded feature map to obtain the image segmentation result corresponding to the target image.

15. The apparatus according to claim 14, characterized in that, The encoding module is further configured to: determine the size of the neighborhood window corresponding to the current level; generate at least one neighborhood window that matches the size of the neighborhood window in the feature map input at the current level; when any neighborhood window covers the pixels directly adjacent to the target pixel in the feature map input at the current level and reaches the maximum number, determine at least one of the pixels covered by the neighborhood window that reaches the maximum number as the pixel adjacent to the target pixel.

16. The apparatus according to claim 15, characterized in that, The encoding module is further configured to: obtain a preset number of adjacent pixels, and determine an initial window size based on the number of adjacent pixels; obtain a hole coefficient set for the current level, and determine a window expansion size based on the hole coefficient; and determine the neighborhood window size corresponding to the current level based on the initial window size and the window expansion size.

17. The apparatus according to claim 16, characterized in that, The hole coefficient is greater than 1; the encoding module is further configured to: perform modulo operation on the position parameters of the target pixel in the feature map of the input layer according to the hole coefficient, and obtain the modulo operation result corresponding to the target pixel; perform modulo operation on the position parameters of each pixel covered by the maximum number of neighborhood windows according to the hole coefficient, and obtain the modulo operation result corresponding to each covered pixel; select at least one pixel from the pixels covered by the neighborhood windows whose modulo operation result matches the modulo operation result corresponding to the target pixel, and determine the selected pixel as the pixel adjacent to the target pixel.

18. The apparatus according to claim 14, characterized in that, The at least one pixel adjacent to the targeted pixel includes each pixel in the corresponding neighborhood of the targeted pixel; the encoding module is further configured to: perform pixel value optimization processing on the feature map of the updated pixel to obtain an intermediate feature map; for each intermediate pixel in the intermediate feature map, select an intermediate pixel at a preset pixel distance from the neighborhood of the targeted intermediate pixel as the intermediate pixel adjacent to the targeted intermediate pixel; the preset pixel distance is matched with the hole coefficient set for the level; extract attention features for each adjacent intermediate pixel based on the positional deviation between the targeted intermediate pixel and each of the adjacent intermediate pixels; update the targeted intermediate pixel according to the extracted attention features, and determine the feature map output for the level according to the intermediate feature map of the updated pixel.

19. The apparatus according to claim 18, characterized in that, The encoding module is further configured to: update the targeted intermediate pixel based on the extracted attention features to obtain an intermediate feature map of the updated pixel; perform cross-connection fusion of the intermediate feature map of the updated pixel and the intermediate feature map obtained by pixel value optimization to obtain a fused feature map; and determine the feature map output at the current layer based on the fused feature map.

20. The apparatus according to claim 14, characterized in that, The encoding module is further configured to: determine the correlation between the target pixel and its neighboring pixels under various attention feature extraction methods, and determine the deviation parameters of each neighboring pixel based on the positional deviation between the target pixel and its neighboring pixels; and under each attention feature extraction method, add the correlation and deviation parameters of each neighboring pixel to obtain the attention weight of each neighboring pixel under each attention feature extraction method. Under each attention feature extraction method, the features of each neighboring pixel are weighted based on the attention weight of each neighboring pixel. The pixel features obtained by weighted processing under various attention feature extraction methods are fused, and the target pixel is updated based on the fused pixel features.

21. The apparatus according to any one of claims 14 to 20, characterized in that, The at least one level includes multiple levels; the encoding module is further configured to, in the encoding process of the first level, for a pixel in the fused feature map, determine at least one pixel adjacent to the pixel in the fused feature map, extract attention features for each adjacent pixel based on the positional deviation between the pixel in the target pixel and each adjacent pixel, update the pixel in the target pixel according to the extracted attention features, and determine the feature map output of the first level according to the feature map of the updated pixel; Starting from the second level, for each level, the encoding process of the current level is performed based on the feature map output by the previous level to obtain the feature map output by the current level. Once the final level of encoding is completed, the encoded feature maps of the multiple levels are obtained.

22. The apparatus according to claim 21, characterized in that, The encoding module is further configured to: for each level starting from the second level, when the level satisfies the downsampling condition, divide the feature map output by the previous level into multiple feature blocks of the same size; extract pixels at the same position among the multiple feature blocks of the same size, and form pixel matrices corresponding to each position by combining the pixels extracted at each position; perform channel superposition on the pixel matrices, and optimize the pixel values ​​of the feature map obtained by channel superposition to obtain a sampled feature map; and perform encoding processing on the sampled feature map for the level to obtain the feature map output by the level.

23. An image segmentation model processing device, characterized in that, The device includes: The image acquisition module is used to acquire sample images and corresponding sample interaction information maps through an image segmentation model, and to acquire historical sample segmentation images obtained by historical segmentation processing of the sample images; to overlay the historical sample segmentation images and the sample interaction information maps by channels to obtain a channel-overlayed sample interaction information map; to extract initial sample feature maps from the channel-overlayed sample interaction information map and the sample images respectively, and to fuse the extracted initial sample feature maps; the sample interaction information map records the position of the pixels affected by the interaction performed on the sample image; the sample interaction information map includes a foreground channel map and a background channel map; the pixels in the foreground region of the foreground channel map are different from the pixels in other regions, and the pixels in the background region of the background channel map are different from the pixels in other regions. An encoding module is configured to perform at least one level of encoding processing based on the fused sample feature map using the image segmentation model to obtain a sample encoded feature map output by the at least one level of encoding processing; wherein, in each level of encoding processing, the encoding module is further configured to, for a pixel in the sample feature map input at the current level, determine at least one neighboring pixel from the sample feature map input at the current level, extract attention features from each neighboring pixel based on the positional deviation between each neighboring pixel, update the target pixel according to the extracted attention features, and determine the sample feature map output at the current level based on the feature map of the updated pixel; The decoding module is used to decode the sample encoded feature map through the image segmentation model to obtain the sample image segmentation result corresponding to the sample image; The model update module is used to update the image segmentation model based on the sample image segmentation results and continue training until the training ends.

24. The apparatus according to claim 23, characterized in that, The image acquisition module is further configured to: acquire a first historical interaction information map and an initial sample segmentation image corresponding to the sample image; the first historical interaction information map records the position of the pixel affected by the first interaction performed on the sample image, and the initial sample segmentation image is obtained by initializing the pixel values ​​of the sample image; perform a first image segmentation process based on the sample image, the first historical interaction information map, and the initial sample segmentation image through the image segmentation model to obtain an intermediate sample segmentation image; in each image segmentation process after the first image segmentation process, input the intermediate sample segmentation image obtained from the previous image segmentation process, the second historical interaction information map corresponding to the sample image, and the sample image into the image segmentation model to obtain the intermediate sample segmentation image for each image segmentation process; the second historical interaction information map records the position of the pixel affected by the subsequent interaction performed on the sample image; when the image segmentation process reaches a preset number of times, use the intermediate sample segmentation image obtained from the last image segmentation process as the historical sample segmentation image.

25. The apparatus according to claim 24, characterized in that, The device further includes a first analog interaction module, configured to: acquire a mask image of the sample image; the mask image contains classification labels for each pixel in the sample image; determine the region of the target to be segmented in the sample image based on the classification labels for each pixel in the mask image; select at least one pixel in the region of the target to be segmented; and determine a first historical interaction information map based on the position of the selected pixel and its corresponding classification label.

26. The apparatus according to claim 24, characterized in that, The device further includes a second analog interaction module, configured to: acquire a mask image of the sample image; the mask image contains the classification labels of each pixel in the sample image; compare the classification results of each pixel in the intermediate sample segmentation image obtained from the previous image segmentation process with the classification labels of each pixel in the mask image, and determine the error region in the intermediate sample segmentation image obtained from the previous image segmentation process based on the comparison results; select at least one pixel from the center position of the error region, and determine a second historical interaction information map based on the position of the selected pixel and the corresponding classification label.

27. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9 or 10 to 13.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9 or 10 to 13.

29. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9 or 10 to 13.

Images