Image segmentation processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide an image segmentation processing method, device and equipment and a medium. The method can be applied in the field of computer vision, and is used for improving the segmentation accuracy of an image. The method comprises the following steps: extracting a first spatial local feature and a first spatial attention feature in a source image; enhancing in a frequency domain by multiplying each associated interaction weight to obtain an attention-enhanced frequency domain feature and a local-enhanced frequency domain feature; then converting to a spatial domain to obtain a second spatial local feature and a second spatial attention feature; combining the first spatial local feature and the second spatial local feature to obtain a local joint feature, and combining the first spatial attention feature, the second spatial attention feature and the local joint feature to obtain an attention joint feature; outputting a local segmentation prediction result from the local joint feature, outputting a global segmentation prediction result from the attention joint feature, and averaging the local segmentation prediction result and the global segmentation prediction result to obtain a final segmentation image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image segmentation processing method, apparatus, device, and medium. Background Technology

[0002] Image segmentation is a crucial research area in computer vision and a vital component of image semantic understanding. Image segmentation refers to the process of dividing an image into non-overlapping regions. For example, dividing an image into regions representing the same object makes analysis of each region easier. In recent years, with the advancement of deep learning technology, image segmentation has made rapid progress and is now widely used in applications such as image classification, object detection, video analysis, robotics, and image processing for autonomous vehicles.

[0003] Currently, deep learning-based image segmentation techniques typically employ a deep encoder and decoder architecture. The encoder performs low-dimensional encoding (i.e., feature extraction) on the input image to obtain its feature representation. This feature representation is then fed into the decoder, which generates the corresponding segmented image. However, current encoder and decoder architectures are usually based on convolutional neural network structures, i.e., stacked convolutional layers. Due to the limited receptive field of convolutional neural networks, the image features extracted by these networks only capture local details within the input image and cannot represent the rich information contained within it. Segmentation based on these extracted features easily leads to unclear object boundaries or segmentation errors, resulting in low segmentation accuracy. Summary of the Invention

[0004] This application provides an image segmentation processing method, apparatus, device, and medium that can improve the accuracy of image segmentation results.

[0005] This application provides an image segmentation processing method, including:

[0006] The source image is acquired, and features are extracted from the source image to obtain the first spatial local features. Attention encoding is then performed on the source image to obtain the first spatial attention features. The first spatial attention features are used to characterize the information transfer between global information in the source image.

[0007] The first spatial local features are converted into local frequency domain features, the first spatial attention features are converted into attention frequency domain features, the first interaction weight associated with the local frequency domain features is obtained, and the second interaction weight associated with the attention frequency domain features is obtained.

[0008] The attention frequency domain features are enhanced according to the first interaction weight to obtain attention-enhanced frequency domain features. The local frequency domain features are enhanced according to the second interaction weight to obtain local-enhanced frequency domain features. The local frequency domain features and local-enhanced frequency domain features are converted into second-space local features. The attention frequency domain features and attention-enhanced frequency domain features are converted into second-space attention features.

[0009] The first spatial local features and the second spatial local features are combined into local joint features, and the first spatial attention features, the second spatial attention features, and the local joint features are combined into attention joint features.

[0010] Obtain the local segmentation prediction results corresponding to the local joint features and the global segmentation prediction results corresponding to the attention joint features. Generate the segmented image corresponding to the source image based on the average of the local segmentation prediction results and the global segmentation prediction results.

[0011] This application provides an image segmentation processing method, including:

[0012] Obtain labeled sample images and unlabeled sample images; the labeled sample images are used as input to the first network in the initial segmentation model, and the unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure.

[0013] The first sample local features corresponding to the unlabeled sample image are obtained by the initial local feature extractor in the first network, and the first sample attention features corresponding to the unlabeled sample image are obtained by the initial global feature extractor in the first network; the first sample attention features are used to characterize the information transfer between global information of the unlabeled sample images.

[0014] The first sample local features are converted into sample local frequency domain features by the interaction components in the first network, the first sample attention features are converted into sample attention frequency domain features, the third interaction weight associated with the sample local frequency domain features is obtained, and the fourth interaction weight associated with the sample attention frequency domain features is obtained.

[0015] The sample attention frequency domain features are enhanced according to the third interaction weight to obtain sample attention enhanced frequency domain features. The sample local frequency domain features are enhanced according to the fourth interaction weight to obtain sample local enhanced frequency domain features. The sample local frequency domain features and sample local enhanced frequency domain features are converted into second sample local features. The sample attention frequency domain features and sample attention enhanced frequency domain features are converted into second sample attention features.

[0016] The local features of the first sample and the local features of the second sample are combined into the joint local features of the first sample. The attention features of the first sample, the attention features of the second sample, and the joint local features of the first sample are combined into the joint attention features of the first sample. The local prediction result of the first sample corresponding to the joint local features of the first sample is output through the initial local segmentation component in the first network. The global prediction result of the first sample corresponding to the joint attention features of the first sample is output through the initial global segmentation component in the first network.

[0017] The labeled sample image and the unlabeled sample image are input into the second network. The second network outputs the local prediction result of the second sample and the global prediction result of the second sample. Based on the local prediction result of the first sample, the global prediction result of the first sample, the local prediction result of the second sample, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, the network parameters of the initial segmentation model are corrected. The second network containing the corrected network parameters is determined as the image segmentation model. The image segmentation model is used to output the segmented image corresponding to the source image.

[0018] One embodiment of this application provides an image segmentation processing apparatus, including:

[0019] The first feature extraction module is used to acquire the source image, extract features from the source image to obtain the first spatial local features, and perform attention encoding processing on the source image to obtain the first spatial attention features; the first spatial attention features are used to characterize the information transfer between global information in the source image;

[0020] The first frequency domain transformation module is used to convert the first spatial local features into local frequency domain features, convert the first spatial attention features into attention frequency domain features, obtain the first interaction weight associated with the local frequency domain features, and obtain the second interaction weight associated with the attention frequency domain features.

[0021] The first feature enhancement module is used to enhance the attention frequency domain features according to the first interaction weight to obtain attention-enhanced frequency domain features, enhance the local frequency domain features according to the second interaction weight to obtain local-enhanced frequency domain features, convert the local frequency domain features and the local-enhanced frequency domain features into second spatial local features, and convert the attention frequency domain features and the attention-enhanced frequency domain features into second spatial attention features.

[0022] The residual connection module is used to combine the first spatial local features and the second spatial local features into local joint features, and to combine the first spatial attention features, the second spatial attention features, and the local joint features into attention joint features.

[0023] The first segmentation prediction module is used to obtain the local segmentation prediction results corresponding to the local joint features and the global segmentation prediction results corresponding to the attention joint features, and generate the segmented image corresponding to the source image based on the average of the local segmentation prediction results and the global segmentation prediction results.

[0024] Specifically, the first feature extraction module is used for:

[0025] The source image is obtained and input into the image feature extraction component in the image segmentation model. The initial image features corresponding to the source image are obtained through the image feature extraction component.

[0026] The initial image features are input into the local feature extractor in the image segmentation model, and the first spatial local features corresponding to the initial image features are obtained through the local feature extractor; the local feature extractor includes N convolutional components, where N is a positive integer;

[0027] The initial image features are input into the global feature extractor in the image segmentation model, and the first spatial attention features corresponding to the initial image features are obtained through the global feature extractor; the global feature extractor includes N attention components.

[0028] The first feature extraction module obtains the initial image features corresponding to the source image through the image feature extraction component, including:

[0029] The source image is convolved by the convolutional layer in the image feature extraction component to obtain the image convolution result corresponding to the source image.

[0030] The image convolution result is normalized by the normalization layer in the image feature extraction component to obtain image normalized features. The image normalized features are then activated by the activation function in the image feature extraction component to obtain image activation features.

[0031] The pooling layer in the image feature extraction component performs pooling operations on the image activation features to obtain the initial image features corresponding to the source image.

[0032] The image segmentation model also includes an interaction component, which is used to exchange frequency domain information between the i-th convolutional component in the local feature extractor and the i-th attention component in the global feature extractor, where i is a positive integer less than or equal to N.

[0033] The first feature extraction module obtains the first spatial local features corresponding to the initial image features through a local feature extractor, including:

[0034] Obtain the input features of the i-th convolutional component in the local feature extractor; when i is 1, the input features of the i-th convolutional component are the initial image features; when i is not 1, the input features of the i-th convolutional component include the output features of the (i-1)-th convolutional component, as well as the spatial local features output by the interaction components acting on the (i-1)-th attention component and the (i-1)-th convolutional component.

[0035] Based on the multiple convolutional layers in the i-th convolutional component, perform convolution operations on the input features of the i-th convolutional component to obtain the local convolution result;

[0036] Residual connections are performed between the local convolution result and the input features of the i-th convolutional component to obtain the output features of the i-th convolutional component. The output features of the N-th convolutional component in the local feature extractor are then determined as the first spatial local features.

[0037] The first feature extraction module obtains the first spatial attention features corresponding to the initial image features through a global feature extractor, including:

[0038] Obtain the input features of the i-th attention component in the global feature extractor; when i is 1, the input features of the i-th attention component are the initial image features; when i is not 1, the input features of the i-th attention component include the output features of the (i-1)-th attention component, the output features of the (i-1)-th convolutional component, and the spatial attention features and spatial local features output by the interaction components acting on the (i-1)-th attention component and the (i-1)-th convolutional component.

[0039] Based on the transformation weight matrix corresponding to the i-th attention component, the input features of the i-th attention component are transformed into a query component, a key matrix, and a value matrix. The dot product operation is performed on the transpose of the query matrix and the key matrix to obtain the candidate weight matrix.

[0040] Obtain the number of columns in the query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and determine the dot product between the attention weight matrix and the value matrix as the attention encoding feature.

[0041] The attention encoding features and the input features of the i-th attention component are combined to form the attention residual features. The attention residual features are then transformed according to the multilayer perceptron in the i-th attention component to obtain the attention transformed features.

[0042] The attention residual features and attention transformation features are combined into the output features of the i-th attention component, and the output features of the N-th attention component in the global feature extractor are determined as the first spatial attention features.

[0043] The first frequency domain conversion module converts the first spatial local features into local frequency domain features, including:

[0044] The first spatial local features are input into the interactive component in the image segmentation model. The interactive component performs a discrete Fourier transform on the first spatial local features to obtain the spectral features corresponding to the first spatial local features.

[0045] Half of the spectral feature corresponding to the first spatial local feature is determined as the local frequency domain feature corresponding to the first spatial local feature.

[0046] The first frequency domain transformation module obtains the first interaction weights associated with local frequency domain features, including:

[0047] The first spectral convolution result is obtained by performing a convolution operation on the real part information of the local frequency domain features based on the first convolutional layer in the interactive component.

[0048] The first spectral convolution result is normalized according to the normalization layer in the interactive component to obtain spectral normalized features. The spectral normalized features are then activated using the first activation function in the interactive component to obtain spectral activated features.

[0049] The second convolutional layer in the interactive component performs a convolution operation on the spectral activation features to obtain the second spectral convolution result. The second activation function in the interactive component is then used to activate the second spectral convolution result to obtain the first interactive weight associated with the local frequency domain features.

[0050] The first feature enhancement module converts local frequency domain features and locally enhanced frequency domain features into second spatial local features, including:

[0051] The local frequency domain features and the local enhanced frequency domain features are combined into frequency domain residual features. The frequency domain residual features are then subjected to inverse Fourier transform through interactive components to obtain the spatial domain features corresponding to the frequency domain residual features.

[0052] Obtain the hidden weight matrix and bias vector corresponding to the multilayer perceptron in the interactive component. Based on the bias vector and the dot product between the hidden weight matrix and the spatial domain features, determine the second spatial local features corresponding to the source image.

[0053] The first segmentation prediction module obtains the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features, including:

[0054] The local joint features are input into the local segmentation component of the image segmentation model. The local segmentation component upsamples the local joint features to obtain the local segmentation prediction result corresponding to the source image.

[0055] The attention joint features are input into the global segmentation component of the image segmentation model. The global segmentation component upsamples the attention joint features to obtain the global segmentation prediction result corresponding to the source image.

[0056] One embodiment of this application provides an image segmentation processing apparatus, including:

[0057] The sample image acquisition module is used to acquire labeled sample images and unlabeled sample images; the labeled sample images are used as input to the first network in the initial segmentation model, and the unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure.

[0058] The second feature extraction module is used to obtain the first sample local features corresponding to the unlabeled sample image through the initial local feature extractor in the first network, and to obtain the first sample attention features corresponding to the unlabeled sample image through the initial global feature extractor in the first network; the first sample attention features are used to characterize the information transfer between global information of the unlabeled sample images;

[0059] The second frequency domain conversion module is used to convert the local features of the first sample into local frequency domain features of the sample through the interactive components in the first network, convert the attention features of the first sample into frequency domain features of the sample attention, obtain the third interactive weight associated with the local frequency domain features of the sample, and obtain the fourth interactive weight associated with the frequency domain features of the sample attention.

[0060] The second feature enhancement module is used to enhance the sample attention frequency domain features according to the third interaction weight to obtain sample attention enhanced frequency domain features, enhance the sample local frequency domain features according to the fourth interaction weight to obtain sample local enhanced frequency domain features, convert the sample local frequency domain features and sample local enhanced frequency domain features into second sample local features, and convert the sample attention frequency domain features and sample attention enhanced frequency domain features into second sample attention features.

[0061] The second segmentation prediction module is used to combine the local features of the first sample and the local features of the second sample into a first sample local joint feature, combine the attention features of the first sample, the attention features of the second sample and the local joint features of the first sample into a first sample attention joint feature, output the local prediction result of the first sample corresponding to the local joint feature of the first sample through the initial local segmentation component in the first network, and output the global prediction result of the first sample corresponding to the attention joint feature of the first sample through the initial global segmentation component in the first network.

[0062] The network parameter correction module is used to input labeled sample images and unlabeled sample images into the second network, and output the local prediction results of the second sample and the global prediction results of the second sample through the second network. Based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample images, the network parameters of the initial segmentation model are corrected, and the second network containing the corrected network parameters is determined as the image segmentation model; the image segmentation model is used to output the segmented image corresponding to the source image.

[0063] The network parameter correction module corrects the network parameters of the initial segmentation model based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample images. The second network, containing the corrected network parameters, is then determined as the image segmentation model, including:

[0064] Based on the local prediction results and global prediction results of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image;

[0065] Based on the local prediction results of the second sample corresponding to the labeled sample image, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image;

[0066] The unsupervised loss and the fully supervised loss are weighted and summed to obtain the total loss of the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total loss until the total loss meets the training termination condition. Training is then stopped, and the second network at the end of training is determined as the image segmentation model.

[0067] Specifically, the network parameter correction module is used for:

[0068] The second network obtains the second sample local joint features and the second sample attention joint features corresponding to the unlabeled sample images. The initial local segmentation component in the second network outputs the second sample local prediction results corresponding to the second sample local joint features. The initial global segmentation component in the second network outputs the second sample global prediction results corresponding to the second sample attention joint features.

[0069] A class distribution map structure is created for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features corresponding to the unlabeled sample images. The class consistency loss between the first network and the second network is determined based on the class distribution map structure.

[0070] Based on the local prediction results and global prediction results of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image;

[0071] Based on the local prediction results of the second sample corresponding to the labeled sample image, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image;

[0072] The class consistency loss, unsupervised loss, and fully supervised loss are weighted and summed to obtain the total model loss corresponding to the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Training is then stopped, and the second network at the end of training is determined as the image segmentation model.

[0073] In the initial segmentation model, both the first and second networks are used to segment C semantic categories, where C is a positive integer;

[0074] The network parameter correction module creates a class distribution map structure for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features. Based on the class distribution map structure, it determines the class consistency loss between the first network and the second network, including:

[0075] Obtain the average feature value of the pixel features belonging to the same semantic category in the local joint features of the first sample, and determine the average feature value of the C semantic categories as the C initial category nodes corresponding to the local joint features of the first sample;

[0076] The local joint features of the first sample are subjected to average pooling to obtain the set of patch nodes corresponding to the local joint features of the first sample.

[0077] For the patch node set and C initial category nodes, construct C independent interaction graph structures, combine the C independent interaction graph structures into a first joint interaction graph structure, obtain C enhanced category nodes from the C independent interaction graph structures, and construct a first category interaction graph structure for the C enhanced category nodes; the first joint interaction graph structure and the first category interaction graph structure belong to the category distribution graph structure corresponding to the local joint features of the first sample;

[0078] Obtain the second joint interaction graph structure and the second category interaction graph structure corresponding to the joint attention features of the first sample; obtain the third joint interaction graph structure and the third category interaction graph structure corresponding to the local joint features of the second sample; obtain the fourth joint interaction graph structure and the fourth category interaction graph structure corresponding to the joint attention features of the second sample.

[0079] Obtain the first mean square error between the second joint interaction graph structure and the third joint interaction graph structure, obtain the second mean square error between the second category interaction graph structure and the third category interaction graph structure, and determine the sum of the first mean square error and the second mean square error as the first model complementary loss.

[0080] The third mean square error between the fourth joint interaction graph structure and the first joint interaction graph structure is obtained, and the fourth mean square error between the fourth category interaction graph structure and the first category interaction graph structure is obtained. The third mean square error and the fourth mean square error are determined as the second model complementarity loss, and the sum of the first model complementarity loss and the second model complementarity loss is determined as the category consistency loss between the first network and the second network.

[0081] The network parameter correction module constructs C independent interaction graph structures for the patch node set and C initial category nodes, including:

[0082] Combine the set of patch nodes and the j-th initial category node from the C initial category nodes to form the j-th initial interaction graph structure; j is a positive integer less than or equal to C;

[0083] Multiply the node representation features in the j-th initial interaction graph structure with the transpose of the node representation features to obtain the matrix product result. Normalize the matrix product result to obtain the adjacency matrix corresponding to the j-th initial interaction graph structure.

[0084] Based on the adjacency matrix and the weight matrix corresponding to the j-th initial interaction graph structure, graph convolution is performed on the j-th initial interaction graph structure to obtain the j-th independent interaction graph structure.

[0085] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0086] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0087] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above aspect.

[0088] In this embodiment, after obtaining the source image, a first spatial local feature and a first spatial attention feature can be extracted from the source image. The first spatial attention feature is used to characterize the information transfer between global information in the source image. That is, both local and global information in the source image can be extracted, which can improve the representation ability of image features. Furthermore, the interaction weights associated with the first spatial local feature and the first spatial attention feature can be obtained in the frequency domain. By exchanging the interaction weights, enhancement processing is performed to obtain local frequency domain enhancement features and attention frequency domain enhancement features, which can fully integrate local and global information in the source image in the frequency domain. Then, it can be transformed to the spatial domain, and residual connections can be performed on the second spatial local feature and the second spatial attention feature after local-global interaction. This can combine the complementarity between local spatial features and spatial attention features, thereby generating a more accurate segmented image and improving the accuracy of image segmentation results. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0091] Figure 2 This is a schematic diagram of an image segmentation scene provided in an embodiment of this application;

[0092] Figure 3 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 1 ;

[0093] Figure 4 This is a schematic diagram of the structure of an image segmentation model provided in an embodiment of this application;

[0094] Figure 5 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 2 ;

[0095] Figure 6 This is a training illustration of an initial segmentation model provided in an embodiment of this application. Figure 1 ;

[0096] Figure 7 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 3 ;

[0097] Figure 8 This is a training illustration of an initial segmentation model provided in an embodiment of this application. Figure 2 ;

[0098] Figure 9 This is a schematic diagram illustrating the category consistency loss between a first network and a second network provided in an embodiment of this application;

[0099] Figure 10 This is a visual schematic diagram of an image segmentation result provided in an embodiment of this application;

[0100] Figure 11 This is a schematic diagram of the structure of an image segmentation processing device provided in an embodiment of this application. Figure 1 ;

[0101] Figure 12 This is a schematic diagram of the structure of an image segmentation processing device provided in an embodiment of this application. Figure 2 ;

[0102] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0103] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0104] For ease of understanding, the basic technical concepts involved in the embodiments of this application will be described below:

[0105] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, locating, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0106] This application specifically relates to image segmentation technology under computer vision, and proposes an image segmentation model based on Convolutional Neural Networks (CNN) and Transformer network (which introduces an attention mechanism). By introducing a local-global interaction strategy into the image segmentation model (such as interactively enhancing the local features extracted by CNN and the attention features extracted by Transformer in the frequency domain), the advantages of CNN in obtaining local image information and Transformer in capturing global dependencies can be fully combined, thereby improving the accuracy of image segmentation results.

[0107] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. The network architecture may include a server 10d and a terminal cluster. The terminal cluster may include one or more terminal devices; the number of terminal devices included in the terminal cluster is not limited here. Figure 1 As shown, the terminal cluster may specifically include terminal device 10a, terminal device 10b, and terminal device 10c, etc.; all terminal devices in the terminal cluster (for example, may include terminal device 10a, terminal device 10b, and terminal device 10c, etc.) can be connected to server 10d via the network so that each terminal device can interact with server 10d through the network connection.

[0108] The terminal devices in a terminal cluster can include smartphones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches and smart bracelets), smart voice interaction devices, smart home appliances (such as smart TVs), in-vehicle devices, aircraft, and other electronic devices. This application does not limit the type of terminal device. It is understood that, as Figure 1 Each terminal device in the terminal cluster shown can install an application client. When the application client runs on each terminal device, it can interact with the aforementioned... Figure 1 Data interaction is performed between the servers 10d shown. The application clients running on each terminal device can be independent clients or embedded sub-clients integrated into a client; this application does not impose any limitations on this.

[0109] The application clients can specifically include, but are not limited to: browsers, in-vehicle clients, smart home clients, entertainment clients (e.g., game clients), multimedia clients (e.g., video clients, short video clients), conferencing clients, and social clients with image processing capabilities. If the terminal devices included in the terminal cluster are in-vehicle devices, then these in-vehicle devices can be intelligent terminals in smart transportation scenarios, and the application clients running on these in-vehicle devices can be called in-vehicle clients.

[0110] Among them, server 10d can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit the type of server.

[0111] Understandable Figure 1The application clients installed on each of the terminal devices shown can call pre-trained image segmentation models. For example, the trained image segmentation model can be deployed online within the application client. In the image segmentation scenario, after acquiring the source image, the application client installed on terminal device 10a can use the trained image segmentation model to perform semantic segmentation on the source image, generating a segmented image corresponding to the source image. This segmented image can be displayed in the application client. Optionally, the application client on terminal device 10a can also upload the source image to server 10d (which can be a backend service for the application client). Server 10d then uses the trained image segmentation model to perform semantic segmentation on the source image, generating a segmented image corresponding to the source image, and transmitting this segmented image to the application client for display. It should be understood that the semantic segmentation process of the source image can be performed by a computer device, which can be... Figure 1 Any terminal device in the terminal cluster shown can be either a server 10d or a computer program (including program code, such as an application client installed on the terminal device). This application embodiment does not limit this.

[0112] The source image can be any type of image, including but not limited to natural scene images, medical images, road photographs, etc. This means the image segmentation model trained in this embodiment can be applied to segmentation tasks of various image types, with a wide range of applications. For ease of understanding, the aforementioned image segmentation model can refer to a network model used for semantic segmentation of images. The input data of this image segmentation model is a source image of any type, and the output data is a segmented image generated from the source image. The segmentation model during the training phase (i.e., the segmentation model that has not completed training) can be called the initial segmentation model. The initial segmentation model and the image segmentation model can have the same network structure or different network structures (the image segmentation model is only a part of the network structure in the trained initial segmentation model). The specific structures of the image segmentation model and the initial segmentation model will be described in detail in subsequent embodiments.

[0113] Further, please see Figure 2 , Figure 2 This is a schematic diagram of an image segmentation scene provided in an embodiment of this application. For example... Figure 2As shown, after acquiring the source image 20a, the terminal device 10a can transmit the source image to the server 10d for image segmentation processing. That is, the server 10d can receive the source image 20a transmitted by the terminal device and obtain the trained image segmentation model; it then performs image segmentation processing on the source image 20a using this model, outputting the segmentation result of the source image 20a. This segmentation result may include the category labels (e.g., ...) corresponding to all pixels in the source image 20a. Figure 2 As shown, the category label can include the characters "0" and "1", where "0" represents the image background in source image 20a, and "1" represents an object in source image 20a (such as a dog). The category label corresponding to all pixels in source image 20a can be as follows: Figure 2 As shown in region 20b, a segmented image 20c corresponding to the source image 20a can be generated based on the category labels corresponding to each pixel in region 20b.

[0114] Understandable Figure 2 The pixel category labels shown are merely an example presented in this embodiment; in practical applications, other types of labels may also be used. The source image to be segmented may include one or more objects besides the image background, such as vehicles, pedestrians, animals, trees, roads, streetlights, buildings, etc. Each object in the source image can be segmented using an image segmentation model. In the segmented image, different colors can be used to fill the areas where each object in the source image is located, or different styles (e.g., solid lines, dashed lines, etc.) or different colors of lines can be used to mark the outline boundaries of each object in the source image. This embodiment does not limit this approach.

[0115] Among them, the following combination Figure 3 and Figure 4 The corresponding embodiments describe in detail the application process of the image segmentation images provided in the embodiments of this application in the image segmentation scenario.

[0116] Please see Figure 3 , Figure 3 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 1 Understandably, this image segmentation processing method can be performed by a computer device, which can be a server (such as...). Figure 1 The server shown (10d) can be a terminal device (such as...) Figure 1 This application does not limit the scope to any single terminal device in the terminal cluster shown. Figure 3 As shown, the image segmentation processing method may include the following steps S101 to S105:

[0117] Step S101: Obtain the source image, extract features from the source image to obtain the first spatial local features, and perform attention encoding on the source image to obtain the first spatial attention features; the first spatial attention features are used to characterize the information transfer between global information in the source image.

[0118] In this embodiment, the computer device can acquire the source image to be segmented from different channels. Here, "channel" refers to the method of acquiring the source image, which may include, but is not limited to: directly capturing images with a camera, directly downloading them from application clients such as video platforms and short video platforms, or directly accessing them from systems such as medical management systems and traffic management systems with data reading permissions. Examples are not provided here, and this application does not limit the type of source image. For instance, the source image can be a medical image, such as a pathological image taken under a microscope, a computed tomography (CT) scan, an MRI, or a computed tomography scan; or a natural image, such as a natural scene image captured with a camera; or an image extracted from videos such as TV series, movies, variety shows, animations, news, music videos, or live streams.

[0119] After obtaining the source image, a trained image segmentation model can be acquired. This model can perform semantic segmentation on the source image, identifying pixels belonging to the same semantic category as the same object and generating a segmented image corresponding to the source image (e.g., Figure 2 The segmented image 20c in the corresponding embodiment allows for a clear and intuitive understanding of the regions containing each object in the source image. This image segmentation model may include image feature extraction components, local feature extractors, global feature extractors, interactive components, local segmentation components, and global segmentation components, etc.

[0120] The image feature extraction component can be used to extract initial image features from the source image. These initial image features can be simultaneously input into two parallel branches: a local feature extractor and a global feature extractor. This image feature extraction component can consist of one or more network layers such as convolutional layers, normalization layers, activation functions, and pooling layers. This image feature extraction component can be referred to as the Stem component. This application embodiment does not limit the network structure of the Stem component. The local feature extractor can be used to extract local information from the source image. The network structure of the local feature extractor can be a convolutional neural network (CNN), including but not limited to ResNet (a residual network model), DenseNet (a dense connection network model), VGGNet (a convolutional neural network model), and any variation of the above convolutional neural networks, or any combination of two or more of the above networks. This application embodiment does not limit the network structure of the local feature extraction. A global feature extractor can be used to extract global information from a source image. The network structure of this global feature extractor can be a transformer (a network model utilizing an attention mechanism), a Conformer (a network model combining a convolutional neural network and a transformer), or a variation of any of the aforementioned network models. This application does not limit the network structure of the global feature extractor. Interaction components are used to exchange information between local and global feature extractors in the frequency domain. This fully combines the advantages of local feature extractors in obtaining local details and global feature extractors in capturing global dependencies. Interaction components in image segmentation models can also be called local-global interactions (LGI).

[0121] The local segmentation component and the global segmentation component can have the same network structure. The local segmentation component and the local feature extractor belong to the same branch. The local segmentation component is used to restore the local features output by the local feature extractor to obtain the local segmentation prediction result. The global segmentation component and the global feature extractor belong to the same branch. The global segmentation component can restore the global features (attention features) output by the global feature extractor to obtain the global segmentation prediction result.

[0122] For ease of understanding, this application uses a network structure where the local feature extractor is based on ResNet and the global feature extractor is based on Transformer as an example to describe the image segmentation process of the source image. Based on the ResNet architecture, the local feature extractor can include N convolutional components (convolutional blocks), where N can be a positive integer, such as 1, 2, 3, etc. When ResNet is specifically a ResNet-101 structure, the N convolutional components in the local feature extractor can be divided into four stages. The deeper the model network layer, the more the channel dimension increases, while the resolution of the features output by the four stages of convolutional components gradually decreases. For example, the channel dimensions of the four stages of convolutional components can be 256, 512, 1024, and 2048, respectively, and the resolution of the features output by each stage decreases by 1 / 4, 1 / 8, 1 / 16, and 1 / 16, respectively. The global feature extractor and the local feature extractor are parallel. The global feature extractor can include N attention components (also called encoding blocks). The size of the features output by the attention components in each stage is consistent with the size of the features output by the convolutional components in the corresponding stages of local feature extraction. Specifically, a PVT (Pyramid Vision Transformer, a network model that introduces a pyramid structure into the Transformer) can be used as the global feature extractor. The PVT can be composed of lightweight Multi-head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) attention components for feature extraction. To extract richer image information, the enhanced local features can be added to the Transformer and used as the input for the next stage.

[0123] Specifically, the acquired source image can be input into the image feature extraction component in the image segmentation model. This component can obtain the initial image features corresponding to the source image. The initial image features can then be input into a parallel local feature extractor and a global feature extractor. The local feature extractor can obtain the first spatial local features corresponding to the initial image features, and the global feature extractor can obtain the first spatial attention features corresponding to the initial image features. The first spatial local features and the first spatial attention features have the same size.

[0124] In one or more embodiments, the image feature extraction component in the image segmentation model may include three consecutive 3×3 convolutional layers (the kernel size of the convolutional layer is 3×3), a normalization layer, an activation function, and a pooling layer with a stride of 2. The activation function may be a ReLU function, a Sigmoid function, a LeakyReLU function, a PReLU function, an ELU (Exponential Linear Unit) function, etc. The type of activation function used in the image feature extraction component is not limited in this application embodiment. The pooling layer may be a max pooling layer or an average pooling layer. The following embodiments are not limited in this regard. It is understandable that the process of obtaining initial image features through the image feature extraction component can include: performing a convolution operation on the source image using the convolutional layers (specifically, three consecutive 3×3 convolutional layers) in the image feature extraction component to obtain the image convolution result corresponding to the source image; then, normalizing the image convolution result using the normalization layer in the image feature extraction component to obtain image normalized features; activating the image normalized features using the activation function (e.g., the ReLU function) in the image feature extraction component to obtain image activation features; and pooling the image activation features using the pooling layer in the image feature extraction component (e.g., a max pooling layer with a stride of 2, which can be understood as representing every four adjacent pixel values ​​in the image activation features using the largest pixel value) to obtain the initial image features corresponding to the source image. By extracting features from the source image using the image feature extraction component, the dimensionality of the input data can be reduced, and some non-critical information in the source image can be filtered out, thereby reducing the computational complexity of subsequent steps.

[0125] Understandably, the interactive component in the image segmentation model can be used to exchange frequency domain information between the i-th convolutional component in the local feature extractor and the i-th attention component in the global feature extractor, thereby enhancing their respective output features to obtain enhanced spatial local features and enhanced spatial attention features; i is a positive integer less than or equal to N. For example, the interactive component can act on the output features of the first convolutional component in the local feature extractor and the first attention component in the global feature extractor; the enhanced features through the interactive component can be input into the second convolutional component and the second attention component, respectively, and the interactive component can continue to act on the output features of the second convolutional component and the second attention component, and so on, taking the output feature of the last convolutional component (i.e., the N-th convolutional component) in the local feature extractor as the first spatial local feature, and the output feature of the last attention component (i.e., the N-th attention component) in the global feature extractor as the first spatial attention feature. It should be understood that the implementation process of the interactive component is the same when it acts on each group of convolutional components and attention components. The embodiments of this application take the last group of convolutional components and attention components as an example for description. For details, please refer to the description in steps S102 to S104 below.

[0126] The process of obtaining the first spatial local features through the local feature extractor may include, but is not limited to: obtaining the input features of the i-th convolutional component in the local feature extractor; when i is 1, the input features of the i-th convolutional component are the initial image features, that is, the input features of the first convolutional component in the local feature extractor are the initial image features; when i is not 1, the input features of the i-th convolutional component include the output features of the (i-1)-th convolutional component, and the spatial local features output by the interaction component acting on the (i-1)-th attention component and the (i-1)-th convolutional component; that is, the input features of the next (e.g., the i-th) convolutional component can be the output features of the previous (e.g., the (i-1)-th) convolutional component, and the spatial local features after the output features of the previous convolutional component are enhanced by the interaction component. Based on the multiple convolutional layers in the i-th convolutional component, a convolution operation is performed on the input features of the i-th convolutional component to obtain the local convolution result. The local convolution result and the input features of the i-th convolutional component are then joined by a residual connection (i.e., the local convolution result and the input features of the i-th convolutional component are added together) to obtain the output features of the i-th convolutional component. This process is repeated to obtain the output features of each convolutional component in the local feature extractor. Finally, the output features of the N-th convolutional component in the local feature extractor can be determined as the first spatial local features.

[0127] The process of obtaining the first spatial attention feature through the global feature extractor may include, but is not limited to: obtaining the input feature of the i-th attention component in the global feature extractor; when i is 1, the input feature of the i-th attention component is the initial image feature, that is, the input feature of the first attention component in the global feature extractor is the initial image feature; when i is not 1, the input feature of the i-th attention component includes the output feature of the (i-1)-th attention component, the output feature of the (i-1)-th convolutional component, and the spatial attention feature and spatial local feature output by the interaction component acting on the (i-1)-th attention component and the (i-1)-th convolutional component; that is, the input feature of the next (e.g., the i-th) attention component can be the output feature of the previous (e.g., the (i-1)-th) attention component, the spatial attention feature after the output feature of the previous (e.g., the (i-1)-th) attention component is enhanced by the interaction component, and the input feature of the current i-th convolutional component.

[0128] It should be understood that each attention component in the global feature extractor can include one or more self-attention structures, and the number of self-attention structures included in each attention component can be the same. For ease of understanding, the following description assumes that the i-th attention component contains only one self-attention structure. Then, the transformation weight matrix corresponding to the i-th attention component can be obtained. Based on the transformation weight matrix corresponding to the i-th attention component, the input features of the i-th attention component can be transformed into a query component Q, a key matrix K, and a value matrix V. The transformation weight matrix corresponding to the i-th attention component can include three parameter matrices, such as parameter matrix W. q Parameter matrix W k and parameter matrix W v The transformation weight matrix corresponding to the i-th attention component is a parameter learned during the training of the image segmentation model. The input features of the i-th attention component are then compared with the parameter matrix W in the transformation weight matrix. q Performing a dot product yields the query matrix Q. The input features of the i-th attention component are then combined with the parameter matrix W in the transformation weight matrix. k Dot product yields the key matrix K, which is then used to transform the input features of the i-th attention component with the parameter matrix W in the weight matrix. v Performing a dot product yields the value matrix V. Each query vector in the query matrix above can be used to encode the similarity relationship between each feature and other features, which can determine the dependency information between that feature and its preceding features.

[0129] Furthermore, a dot product is performed on the transpose of the query matrix and the key matrix to obtain the candidate weight matrix (which can be represented as QK). TThe candidate weight matrix can be considered as the inner product (also called dot product) of each row vector in the query matrix Q and the key matrix K. To prevent the inner product from being too large, the number of columns in the query matrix Q can be obtained (the query matrix Q and the key matrix K have the same number of columns, also called the vector dimension). Then, the candidate weight matrix is ​​divided by the square root of the number of columns (which can be denoted as...). The ratio between the two values ​​is normalized to obtain the attention weight matrix. The dot product between the attention weight matrix and the value matrix V is determined as the attention encoding feature of the i-th attention component for the initial image features.

[0130] The attention weight matrix can be represented as follows: The softmax function is a function used for normalization. It can be used to calculate the self-attention coefficient of a single feature with respect to other features. Each row in the matrix is ​​subjected to softmax. The dot product between the attention weight matrix and the value matrix V is used to determine the encoded features of the i-th attention component for the initial image features (which can be represented as...). When the i-th attention component contains only one self-attention structure, the encoded feature O1 can be used as the attention encoding feature of the i-th attention component for the initial image features.

[0131] Optionally, if each attention component in the global feature extractor includes multiple self-attention structures, then each self-attention structure in the i-th attention component can correspond to an encoded feature, such as encoded feature O1, encoded feature O2, encoded feature O3, ...; furthermore, the encoded features corresponding to multiple self-attention structures can be concatenated into an attention encoded feature of the i-th attention component on the initial image features. This concatenation can be a concat operation. Further, the attention encoded feature of the i-th attention component on the initial image features and the input features of the i-th attention component can be combined into attention residual features.

[0132] After obtaining the attention residual features corresponding to the i-th attention component, these features can be input into the multilayer perceptron within the i-th attention component. The multilayer perceptron in the i-th attention component performs feature transformation on the attention residual features to obtain the attention transformation features. For example, a linear transformation can be performed on the attention residual features based on the hidden weights and bias vectors corresponding to the multilayer perceptron in the i-th attention component, ultimately outputting the attention transformation features. The attention residual features and attention transformation features can then be combined to form the output features of the i-th attention component, i.e., the attention residual features and attention transformation features are added together to obtain the output features of the i-th attention component. Similarly, the output features of each attention component in the global feature extractor can be obtained sequentially, and finally, the output feature of the N-th attention component in the global feature extractor can be determined as the first spatial attention feature.

[0133] Optionally, in one or more embodiments, before inputting the initial image features into the global feature extractor, the initial image features can be positionally encoded to obtain positional encoding information corresponding to the initial image features. The combined feature obtained by adding the initial image features and the positional encoding information is then input into the first attention component of the global feature extractor. Subsequently, the combined feature obtained by adding the initial image features and the positional encoding information can be calculated based on the N attention components in the global feature extractor to finally obtain the first spatial attention feature. The positional encoding methods involved in the embodiments may include, but are not limited to, 2D sine position embedding, learnable position embedding, etc.

[0134] Step S102: Convert the first spatial local features into local frequency domain features, convert the first spatial attention features into attention frequency domain features, obtain the first interaction weight associated with the local frequency domain features, and obtain the second interaction weight associated with the attention frequency domain features.

[0135] Specifically, according to the spectral convolution theorem in Fourier theory, pointwise updates in the Fourier domain can globally affect all input features. Therefore, learning in the frequency domain can simultaneously adjust all frequencies to capture long-term or short-term interactions. Inspired by this, embodiments of this application introduce an intra-model local-global interaction component (which can be simply referred to as the interaction component) in the image segmentation model. This interaction component explores interactions in the frequency domain. The interaction component in the image segmentation model can act on each set of convolutional components and attention components located at the same position in the local feature extractor and the global feature extractor. The implementation of the interaction component is described in detail below, taking the interaction component acting on the first spatial local features and the first spatial attention features (i.e., the last convolutional component in the local feature extractor and the last attention component in the global feature extractor) as an example.

[0136] Both the first spatial local features and the first spatial attention features can be transformed from the spatial domain to the frequency domain. For example, the first spatial local features can be transformed into local frequency domain features, and the first spatial attention features can be transformed into attention frequency domain features. The frequency domain transformation methods used in this application embodiment may include, but are not limited to, Discrete Fourier Transform (DFT), Fast Fourier Transform (FFT), Laplace Transform, Z-Transform, wavelet transform, etc. For ease of understanding, this application embodiment uses FFT as an example to perform frequency domain transformation on the first spatial local features and the first spatial attention features. FFT is a fast algorithm for Discrete Fourier Transform, obtained by improving the algorithm based on the odd, even, imaginary, and real characteristics of Discrete Fourier Transform; using FFT for frequency domain transformation can improve the efficiency of subsequent data processing.

[0137] In this process, after inputting the first spatial local features into the interactive component of the image segmentation model, the interactive component can perform a Discrete Fourier Transform (FFT) on the first spatial local features to obtain the corresponding spectral features, which can be used as local frequency domain features. Performing an FFT on the first spatial attention features yields the corresponding spectral features, which can be used as attention frequency domain features. The FFT, as the first step of the interactive component, allows the first spatial local features and the first spatial attention features to be converted from the spatial domain to the Fourier domain in the following way: Where, x CNN Represents local features of the first space. X represents the Fast Fourier Transform. CNN The spectral features representing local features in the first space (which can be used as local frequency domain features), x TransX represents the first-space attention feature. Trans The spectral features representing the attention features of the first space (can be used as attention frequency domain features).

[0138] Optionally, in one or more embodiments, after obtaining the spectral features corresponding to the first spatial local features and the spectral features corresponding to the first spatial attention features, half of the spectral features corresponding to the first spatial local features can be determined as the local frequency domain features corresponding to the first spatial local features. In this case, the local frequency domain features can be denoted as 1 / 2X. CNN Half of the spectral feature corresponding to the first spatial attention feature is defined as the attention frequency domain feature corresponding to the first spatial attention feature. This attention frequency domain feature can then be denoted as 1 / 2X. Trans In other words, the conjugate symmetry property of the discrete Fourier transform can be used to retain only the spectral features (X). CNN and X Trans Half of the median, that is, the spectral feature X CNN Half of the median is used as a local frequency domain feature, and the spectral feature X is... Trans Half of the median is used as the attention frequency domain feature. Essentially, the conjugate symmetry of the DFT can be used to calculate the discarded half of the value using the retained half. The dimensions of the local frequency domain feature and the attention frequency domain feature can then be expressed as H×(W / 2)×D, where H represents the height of the first spatial local feature (first spatial attention feature), W represents the width of the first spatial local feature (first spatial attention feature), and D represents the number of channels of the first spatial local feature (first spatial attention feature). This embodiment utilizes the conjugate symmetry of the Discrete Fourier Transform to retain only half of the spectral feature value, which can improve computational efficiency without losing important information.

[0139] Furthermore, interaction weights can be learned from both the local feature extractor and the global feature extractor branches. For example, the first interaction weights associated with local frequency domain features can be learned from the local feature extractor. Learn the second interaction weights for attention-based frequency domain feature associations from the global feature extractor. The first and second interaction weights here can be collectively referred to as interaction weights. First interaction weight Second interaction weight All can be generated using simple and effective network structures. These network structures may include two convolutional layers (e.g., two 1×1 convolutional layers), a normalization layer, a GELU activation layer, a sigmoid activation layer, etc. For example, an activation layer can be connected after a convolutional layer. The number of these network layers can be increased or decreased according to the actual application scenario requirements, and they can be connected in any form. The embodiments of this application do not limit the network structure used to generate interaction weights.

[0140] Understandably, the first interaction weight Second interaction weight The generation methods are the same; for ease of understanding, this application embodiment only addresses the first interaction weight. The generation process is described in detail below. The generation process of the first interaction weight may include: performing a convolution operation on the real part information of the local frequency domain features using the first convolutional layer (e.g., a 1×1 convolutional layer) in the interaction component to obtain a first spectral convolution result; normalizing the first spectral convolution result using a normalization layer (here, the normalization layer can be a BN layer) in the interaction component to obtain spectral normalized features; activating the spectral normalized features using the first activation function (e.g., the GELU function) in the interaction component to obtain spectral activation features; performing a convolution operation on the spectral activation features using the second convolutional layer (again, a 1×1 convolutional layer) in the interaction component to obtain a second spectral convolution result; and activating the second spectral convolution result using the second activation function (e.g., the sigmoid function) in the interaction component to obtain the first interaction weight associated with the local frequency domain features. In this embodiment, only the real part information of the local frequency domain features (attention frequency domain features) is used to generate the interaction weights, which can improve computational efficiency.

[0141] Step S103: Enhance the attention frequency domain features according to the first interaction weight to obtain attention-enhanced frequency domain features; enhance the local frequency domain features according to the second interaction weight to obtain local-enhanced frequency domain features; convert the local frequency domain features and the local-enhanced frequency domain features into second spatial local features; and convert the attention frequency domain features and the attention-enhanced frequency domain features into second spatial attention features.

[0142] Specifically, local frequency domain features and attention frequency domain features can be enhanced by exchanging interaction weights, effectively exchanging information between the local and global feature extractor branches. Based on the first interaction weight... By enhancing the attention frequency domain features, we can obtain attention-enhanced frequency domain features, based on the second interaction weights. By enhancing local frequency domain features, locally enhanced frequency domain features can be obtained; among them, the first interaction weights Attention frequency domain features, attention enhancement frequency domain features, and second interaction weights The relationship between local frequency domain features and local enhanced frequency domain features can be expressed as shown in the following formula (1):

[0143]

[0144] in, X represents the local enhanced frequency domain characteristics. CNN Represents local frequency domain characteristics, ⊙ represents matrix dot product. X represents attention-enhanced frequency domain features. Trans This represents the frequency domain features of attention.

[0145] Furthermore, the enhanced local frequency domain features and the original local frequency domain features can be transformed back into the spatial domain together, and a multilayer perceptron can be used as the channel mixer to obtain the second spatial local features. Similarly, the enhanced attention-enhanced frequency domain features and the original attention-enhanced frequency domain features can be transformed back into the spatial domain together, and a multilayer perceptron can be used as the channel mixer to obtain the second spatial attention features. It should be understood that the process of transforming from the frequency domain to the spatial domain can be achieved using the inverse Fourier transform (IFFT).

[0146] Among them, the local frequency domain features and the local enhanced frequency domain features are combined into frequency domain residual features. The frequency domain residual features are then subjected to inverse Fourier transform through the interactive component to obtain the spatial domain features corresponding to the frequency domain residual features. The hidden weight matrix and bias vector corresponding to the multilayer perceptron in the interactive component are obtained. Based on the bias vector and the dot product between the hidden weight matrix and the spatial domain features, the second spatial local features corresponding to the source image are determined. Similarly, the attention frequency domain features and the attention enhanced frequency domain features can be combined, and the combined features are subjected to inverse Fourier transform to obtain the spatial domain features on the global feature extractor branch. Then, the second spatial attention features are output through the multilayer perceptron. The relationship between the local frequency domain features, the local enhanced frequency domain features, the second spatial local features, and the attention frequency domain features, the attention enhanced frequency domain features, and the second spatial attention features can be shown in the following formula (2):

[0147]

[0148] in, This represents the spatial domain features of the second spatial local features before passing through the multilayer perceptron. Indicates the inverse Fourier transform. H×W×D represents the spatial domain features of the second spatial attention feature before passing through the multilayer perceptron; H×W×D represents the size of the second spatial local feature and the second spatial attention feature.

[0149] Step S104: Combine the first spatial local features and the second spatial local features into a local joint feature, and combine the first spatial attention feature, the second spatial attention feature, and the local joint feature into an attention joint feature.

[0150] Specifically, the second spatial local features, after enhancement processing by the interactive components in the image segmentation model, can be combined with the first spatial local features originally input to the interactive components to form a residual path. In other words, the first and second spatial local features can be added together to form a local joint feature. Similarly, the first and second spatial attention features can be added together to form a joint feature; for the global feature extractor branch, the second spatial local features from the local feature extraction branch can also be added as attention joint features.

[0151] Step S105: Obtain the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features. Generate the segmented image corresponding to the source image based on the average value of the local segmentation prediction result and the global segmentation prediction result.

[0152] Specifically, local joint features can be input into the local segmentation component of the image segmentation model. The local segmentation component upsamples the local joint features to obtain the local segmentation prediction result corresponding to the source image. Attention joint features can be input into the global segmentation component of the image segmentation model. The global segmentation component upsamples the attention joint features to obtain the global segmentation prediction result corresponding to the source image. The local segmentation component and the global segmentation component are segmentation heads on two parallel branches. Both have the same network structure, such as the segmentation head in the existing DeepLabV3+ (an image segmentation technique), or they can be other decoding structures with image recovery capabilities. This application does not limit the network structure of the local segmentation component and the global segmentation component.

[0153] Understandably, the local segmentation prediction result is obtained through local joint feature prediction from the local feature extractor branch, while the global segmentation prediction result is obtained through attention joint feature prediction from the global feature extractor branch. By averaging the local and global segmentation prediction results, the final segmented image (e.g., ...) can be generated. Figure 2 The segmented image 20c is shown.

[0154] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an image segmentation model provided in an embodiment of this application. For example... Figure 4As shown, after acquiring the source image, it can be input into the image feature extractor in the image segmentation model. This feature extractor obtains the initial image features corresponding to the source image. These initial features can be input into two parallel branches: the local feature extractor branch (which can be a network structure based on a convolutional neural network, hence called the CNN branch) and the global feature extractor branch (which can be a network structure based on a Transformer, hence called the Transformer branch). In other words, the initial image features can be input into the first convolutional component (also called the first convolutional block, or Conv Block) in the CNN branch and the first attention component (also called the Transformer coding block, or TransBlock) in the Transformer branch.

[0155] The output features of the first convolutional component are obtained by calculating the initial image features, and can be denoted as follows: Similarly, by calculating the initial image features using the first attention component, we can obtain the output features of the first attention component, which can be denoted as... Furthermore, the local-global interaction component 30a in the image segmentation model can be used to... and Enhancement processing can be performed, such as... Perform a Fast Fourier Transform (FFT) to obtain The spectrum (which can be denoted as) Similarly, it can also be used for... Perform a Fast Fourier Transform (FFT) to obtain The spectrum (which can be denoted as) Furthermore, the first interaction weights on the CNN branch, i.e., the spectrum, can be generated through two 1×1 convolutional layers (the first convolutional layer and the second convolutional layer), two activation functions (the first activation function can be GELU, and the second activation function can be sigmoid), and a normalization layer. The first interaction weights are associated. The second interaction weights, i.e., the spectrum, can be generated on the Transformer branch in the same way. The associated second interaction weight.

[0156] Furthermore, this can be achieved by multiplying the first interaction weight by the spectrum. Enhancement processing is performed to obtain the enhanced spectrum. This can then enhance the spectrum. Compared with the spectrum before enhancement The sums are then subjected to an inverse Fourier transform (IFFT), and the enhanced local features are obtained by passing them through a multilayer perceptron. Similarly, attention features before and after augmentation can be obtained. This allows for the enhancement of local features. and the original local features The features are added together and used as the input features for the second convolutional component (i.e., For the Transformer branch, the enhanced attention features can be... and the original attention characteristics And the input features of the second convolutional component are used as the input features of the second attention component (i.e. It is understandable that each subsequent set of convolutional and attention components needs to be enhanced using a local-global interaction component, the process of which is consistent with the aforementioned description and will not be repeated here. The local joint features (which can be represented as...) can ultimately be obtained from the local-global interaction component and the N convolutional components in the CNN branch. in This represents the output feature of the Nth convolutional component, i.e., the aforementioned first spatial local feature. (Representing local features in the second space); the joint attention features (which can be represented as) can ultimately be obtained from the local-global interaction components and the N attention components in the Transformer branch. in The output feature of the Nth attention component, namely the aforementioned first spatial attention feature, (This represents the second spatial attention feature).

[0157] Furthermore, the local joint features can be input into the local segmentation component in the CNN branch to output the local segmentation prediction result, and the attention joint features can be input into the global segmentation component in the Transformer branch to output the global segmentation prediction result. The local segmentation prediction result and the global segmentation prediction result are averaged to obtain the segmented image corresponding to the source image.

[0158] In this embodiment, after obtaining the source image, a first spatial local feature and a first spatial attention feature can be extracted from the source image. The first spatial attention feature is used to characterize the information transfer between global information in the source image. That is, both local and global information in the source image can be extracted, which can improve the representation ability of image features. Furthermore, the interaction weights associated with the first spatial local feature and the first spatial attention feature can be obtained in the frequency domain. By exchanging the interaction weights, enhancement processing is performed to obtain local frequency domain enhancement features and attention frequency domain enhancement features, which can fully integrate local and global information in the source image in the frequency domain. Then, it can be transformed to the spatial domain, and residual connections can be performed on the second spatial local feature and the second spatial attention feature after local-global interaction. This can combine the complementarity between local spatial features and spatial attention features, thereby generating a more accurate segmented image and improving the accuracy of image segmentation results.

[0159] It is understandable that the image segmentation model is a trained segmentation model; that is, only a segmentation model that has completed training can be formally applied in image semantic segmentation scenarios. For ease of understanding, the segmentation model in the training phase can be called the initial segmentation model. The following will combine this with the appendix... Figures 5 to 10 The training process of the initial segmentation model is described.

[0160] Please see Figure 5 , Figure 5 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 2 It is understood that this image segmentation processing method can be executed by a computer device, which can be a server or a terminal device; this application does not limit this. Figure 5 As shown, the image segmentation processing method may include the following steps S201 to S209:

[0161] Step S201: Obtain labeled sample images and unlabeled sample images; labeled sample images are used as input to the first network in the initial segmentation model, and unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure.

[0162] In this embodiment, the training process of the initial segmentation model is a semi-supervised learning (SSL) process, which utilizes unlabeled sample images to reduce the dependence on a large number of labeled sample images, thereby improving the data utilization of the image segmentation model. During the model training phase, labeled and unlabeled sample images can be obtained for training the initial segmentation model; labeled sample images are those carrying labeled information, and unlabeled sample images are those without labeled information. The initial segmentation model is a semi-supervised model with a Mean-Teacher structure, meaning that two networks are introduced: a first network and a second network. The first network acts as the teacher network to guide the training of the second network (student network). The second network can simultaneously input labeled and unlabeled sample images, while the first network only inputs unlabeled sample images. The first and second networks have the same network structure, for example, the network structures of both networks are as described above. Figure 4 The network structure in the corresponding embodiment. In this application embodiment, the unlabeled sample data and labeled sample data used to train the initial segmentation model can be obtained from currently available datasets or obtained by any other means; this application embodiment does not limit this.

[0163] Step S202: Obtain the first sample local features corresponding to the unlabeled sample image through the initial local feature extractor in the first network, and obtain the first sample attention features corresponding to the unlabeled sample image through the initial global feature extractor in the first network.

[0164] Step S203: Convert the local features of the first sample into local frequency domain features of the sample through the interaction components in the first network, convert the attention features of the first sample into frequency domain features of the sample attention, obtain the third interaction weight associated with the local frequency domain features of the sample, and obtain the fourth interaction weight associated with the frequency domain features of the sample attention.

[0165] Step S204: Enhance the sample attention frequency domain features according to the third interaction weight to obtain sample attention enhanced frequency domain features; enhance the sample local frequency domain features according to the fourth interaction weight to obtain sample local enhanced frequency domain features.

[0166] Step S205: Convert the sample local frequency domain features and sample local enhanced frequency domain features into second sample local features, and convert the sample attention frequency domain features and sample attention enhanced frequency domain features into second sample attention features.

[0167] Step S206: Combine the local features of the first sample and the local features of the second sample into the joint local features of the first sample, and combine the attention features of the first sample, the attention features of the second sample, and the joint local features of the first sample into the joint attention features of the first sample.

[0168] Step S207: Output the local prediction result of the first sample corresponding to the local joint features of the first sample through the initial local segmentation component in the first network, and output the global prediction result of the first sample corresponding to the attention joint features of the first sample through the initial global segmentation component in the first network.

[0169] Please see below. Figure 6 , Figure 6 This is a training illustration of an initial segmentation model provided in an embodiment of this application. Figure 1 .like Figure 6 As shown, the initial segmentation model includes a first network and a second network. The two networks have the same network structure. For example, the first network can include two branches, namely a CNN branch and a Transformer branch. The second network can also include two branches, CNN and Transformer. The CNN branch can be used to extract local features in the input sample image, and the Transformer branch can be used to extract global features in the input sample image.

[0170] Unlabeled sample images can be input into the first network. Through the two parallel branches (CNN branch and Transformer branch) of the first network and the introduced local-global interaction component, the first sample local joint features and the first sample attention joint features corresponding to the unlabeled sample images can be obtained. The process of obtaining the first sample local joint features can be found in the preceding text. Figure 3 The description of the local joint feature acquisition process in the corresponding embodiment, and the first sample attention joint feature, can be found in the foregoing. Figure 3 The relevant descriptions of the attention joint feature acquisition process in the corresponding embodiments will not be repeated here. Furthermore, through the initial local segmentation component in the CNN branch of the first network, the local prediction results of the first sample corresponding to the unlabeled sample image can be obtained (e.g., Figure 6 The local prediction result shown in 1); through the initial global segmentation component in the Transformer branch of the first network, the global prediction result of the first sample corresponding to the unlabeled sample image can be obtained (e.g., Figure 6 The global prediction result shown is 1). The process for obtaining the local prediction result and the global prediction result of the first sample can also be found in [reference 1]. Figure 3 The descriptions of the local segmentation prediction results and the global segmentation prediction results in the corresponding embodiments will not be repeated here.

[0171] Step S208: Input the labeled sample image and the unlabeled sample image into the second network, and output the local prediction result of the second sample and the global prediction result of the second sample through the second network.

[0172] Specifically, such as Figure 6 As shown, both labeled and unlabeled sample images can be input into the second network. Through two parallel branches (CNN branch and Transformer branch) and the introduced local-global interaction component, the joint local features and joint attention features of the second samples corresponding to the labeled / unlabeled sample images can be obtained. Furthermore, through the initial local segmentation component in the CNN branch of the second network, the local prediction results of the second samples corresponding to the labeled / unlabeled sample images (such as...) can be obtained. Figure 6 The local prediction result shown in 2) can be obtained by using the initial global segmentation component in the Transformer branch of the second network to obtain the global prediction result of the second sample corresponding to the labeled sample image / unlabeled sample image (e.g., local prediction result 2); through the initial global segmentation component in the Transformer branch of the second network, the global prediction result of the second sample corresponding to the labeled sample image / unlabeled sample image can be obtained (e.g., local prediction result 2); Figure 6 The global prediction result shown is 2). It should be noted that the trained second network can be applied as an image segmentation model to image semantic segmentation scenarios. Therefore, the calculation process for labeled / unlabeled sample images in the second network can be found in [reference needed]. Figure 3 The calculation process of the source image in the image segmentation model in the corresponding embodiments will not be described in detail in this application.

[0173] Step S209: Based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, the network parameters of the initial segmentation model are corrected, and the second network containing the corrected network parameters is determined as the image segmentation model.

[0174] Specifically, such as Figure 6 As shown, based on the local prediction results and global prediction results of the first sample corresponding to the unlabeled sample image, an unsupervised loss associated with the unlabeled sample image is determined; that is, the unsupervised loss is a pixel-level loss applied to the unlabeled sample image, allowing the second network to learn relevant information about the unlabeled sample image from the first network. Based on the local prediction results and global prediction results of the second sample corresponding to the labeled sample image, as well as the annotation information carried by the labeled sample image, a fully supervised loss associated with the labeled sample image is determined. The fully supervised loss is applied to the labeled sample image, and can specifically use cross-entropy loss or other commonly used loss function types; this embodiment does not limit the specific loss function used.

[0175] Furthermore, by weighted summing of the unsupervised and fully supervised losses, the total model loss corresponding to the initial segmentation model is obtained. The network parameters of the initial segmentation model are then iteratively trained based on this total model loss until the total model loss meets the training termination condition. Training is then stopped, and the second network at the end of training can be determined as the image segmentation model. For example, suppose the unsupervised loss is... Full supervision loss is Therefore, the total loss of the model can be expressed as: in, λ represents the total loss of the initial segmentation model. s Losses due to full supervision The corresponding weights can be used to constrain the fully supervised loss. λ u Unsupervised loss The corresponding weights can be used to constrain unsupervised loss. λ s , λ u The settings can be configured according to the actual needs of the application scenario, and this application embodiment does not limit this. The training termination condition can be a pre-set termination condition, such as a pre-set maximum number of iterations and a pre-set error range, etc. The training can be stopped as long as the total loss of the initial segmentation model meets any one of the training termination conditions, and the second network at the end of training is determined as the image segmentation model.

[0176] In this embodiment, the initial segmentation model adopts a student network (second network) - teacher network (first network) structure and is trained using a semi-supervised learning method. An intra-model local-global interaction component (hereinafter referred to as the interaction component) is introduced in both the first and second networks, and this intra-model local-global interaction component is implemented in the frequency domain. By applying this local-global interaction component between the two parallel CNN and Transformer branches in the first and second networks, information from the CNN and Transformer branches can be effectively exchanged. This fuses the complementarity between the features extracted by the CNN and Transformer branches, which is beneficial for improving the representation ability of image features. The resulting segmented image is more accurate, thus improving the accuracy of the image segmentation results.

[0177] Please see Figure 7 , Figure 7 This is a flowchart illustrating an image segmentation processing method provided in an embodiment of this application. Figure 3 It is understood that this image segmentation processing method can be executed by a computer device, which can be a server or a terminal device; this application does not limit this. Figure 7As shown, the image segmentation processing method may include the following steps S301 to S311:

[0178] Step S301: Obtain labeled sample images and unlabeled sample images; labeled sample images are used as input to the first network in the initial segmentation model, and unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure.

[0179] Step S302: Obtain the first sample local features corresponding to the unlabeled sample image through the initial local feature extractor in the first network, and obtain the first sample attention features corresponding to the unlabeled sample image through the initial global feature extractor in the first network.

[0180] Step S303: Convert the local features of the first sample into local frequency domain features of the sample through the interaction components in the first network, convert the attention features of the first sample into frequency domain features of the sample attention, obtain the third interaction weight associated with the local frequency domain features of the sample, and obtain the fourth interaction weight associated with the frequency domain features of the sample attention.

[0181] Step S304: Enhance the sample attention frequency domain features according to the third interaction weight to obtain sample attention enhanced frequency domain features; enhance the sample local frequency domain features according to the fourth interaction weight to obtain sample local enhanced frequency domain features.

[0182] Step S305: Convert the sample local frequency domain features and sample local enhanced frequency domain features into second sample local features, and convert the sample attention frequency domain features and sample attention enhanced frequency domain features into second sample attention features.

[0183] Step S306: Combine the local features of the first sample and the local features of the second sample into the joint local features of the first sample, and combine the attention features of the first sample, the attention features of the second sample, and the joint local features of the first sample into the joint attention features of the first sample.

[0184] Step S307: Output the local prediction result of the first sample corresponding to the local joint features of the first sample through the initial local segmentation component in the first network, and output the global prediction result of the first sample corresponding to the attention joint features of the first sample through the initial global segmentation component in the first network.

[0185] Step S308: Obtain the second sample local joint features and second sample attention joint features corresponding to the unlabeled sample image through the second network; output the second sample local prediction result corresponding to the second sample local joint features through the initial local segmentation component in the second network; and output the second sample global prediction result corresponding to the second sample attention joint features through the initial global segmentation component in the second network.

[0186] Step S309: Create a category distribution map structure for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features corresponding to the unlabeled sample image, and determine the category consistency loss between the first network and the second network based on the category distribution map structure.

[0187] In this embodiment, the forward computation process of unlabeled sample images and labeled sample images in the initial segmentation model is the same as described above. Figure 5 The forward computation process is the same in the corresponding embodiments, and will not be described again here. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a training illustration of an initial segmentation model provided in an embodiment of this application. Figure 2 ;like Figure 8 As shown, similar to the aforementioned Figure 6 Unlike the corresponding embodiments, this application's embodiment can introduce class consistency loss between models during the initial segmentation model training process. This is achieved by supplementing the class-level distribution statistics of CNN and Transformer through cross-teaching. This class consistency loss applies to unlabeled sample images. Specifically, the class consistency loss is determined by creating a class distribution map structure for the first sample's local joint features, the first sample's attention joint features, the second sample's local joint features, and the second sample's attention joint features. The determination process of the class consistency loss is described in detail below.

[0188] Understandably, in image segmentation tasks, class relevance is key to contextual modeling, aiming to explore high-level semantic relationships. Since the first and second networks in the initial segmentation model contain different feature extractors (local feature extractors in the CNN branch and global feature extractors in the Transformer branch), they can generate statistical information at different class levels. Therefore, this embodiment can generate more stable and accurate pseudo-labels for unlabeled sample images by constructing a class distribution graph structure (here, the class distribution graph structure is a two-level graph structure) and employing an implicit consistency regularization cross-teaching strategy. To construct the class distribution graph structure, it is first necessary to obtain the graph nodes required for construction. This embodiment involves two types of graph nodes: class nodes and high-level semantic patch nodes. Class nodes can be denoted as... Patch nodes can be denoted as It should be understood that both the first and second networks in the initial segmentation model can be used to segment C semantic categories, where C is a positive integer. The semantic categories involved in the embodiments of this application can be understood as the identifiable object categories in the unlabeled sample images input to the initial segmentation model.

[0189] It is understood that the embodiments of this application require the creation of category distribution map structures for the aforementioned first sample local joint features, first sample attention joint features, second sample local joint features, and second sample attention joint features, respectively. The creation process is the same for all of them. The following will take the first sample local joint features as an example to describe the creation process of the category distribution map structure.

[0190] In one or more embodiments, the average feature value corresponding to pixel features belonging to the same semantic category in the local joint features of the first sample can be obtained, and the average feature value corresponding to C semantic categories can be determined as C initial category nodes corresponding to the local joint features of the first sample; that is, the average value of all pixel features of the same semantic category can be used as an initial category node. Taking the first network as an example, for any unlabeled sample image I u The initial local feature extractor and interactive components in the first network can generate the first sample's local joint features. The corresponding prediction result is the local prediction result of the first sample. The initial global feature extractor and interaction components in the second network can generate the first sample attention joint features. The corresponding prediction result is the global prediction result for the first sample. Furthermore, based on the probability that a pixel feature belongs to the j-th semantic category (where j is a positive integer less than or equal to C), all pixel features belonging to the same semantic category can be grouped into an initial category node, such as: Where z i Represents the local joint features of the first sample (or joint features of first sample attention) The i-th pixel feature in the sample image (where i is a positive integer less than or equal to the total number of pixels in the sample image). It is the probability that the i-th pixel feature belongs to the j-th semantic category.

[0191] It should be understood that This is the total number of initial category nodes, which is the same as the number of semantic categories. Local joint features of the first sample The initial set of category nodes in can be represented as Using the above method, local joint features of the first sample in the first network can be obtained. The initial set of category nodes is obtained from the process. Joint attention features from the first sample of the first network The initial set of category nodes is obtained from the process. Local joint features of the second sample from the second network The initial set of category nodes is obtained from the process. Joint attention features from the second sample of the second network The initial set of category nodes is obtained from the process. It should be noted that the number of category nodes in the above initial category node sets is the same, and all of these initial category node sets can be called initial category node sets.

[0192] Furthermore, the local joint features of the first sample can be subjected to average pooling to obtain the set of patch nodes corresponding to the local joint features of the first sample. More specifically, an average pooling layer with a stride of r can be used to perform average pooling on the local joint features of the first sample to obtain the set of patch nodes corresponding to the local joint features of the first sample. The number of patch nodes contained in this set is H / r×W / r, and the patch nodes in the set are high-level representations with rich semantics. In this way, the local joint features of the first sample of the first network can be used to obtain the local joint features of the first sample. The set of patch nodes is obtained from Joint attention features from the first sample of the first network The set of patch nodes is obtained from Local joint features of the second sample from the second network The set of patch nodes is obtained from Joint attention features from the second sample of the second network The set of patch nodes is obtained from It should be noted that the number of patch nodes in the above patch node sets is the same, and all of these patch node sets can be called patch node sets.

[0193] After obtaining the two types of graph nodes mentioned above, C independent interaction graph structures can be constructed for the patch node set and C initial category nodes. These C independent interaction graph structures are then combined into a first joint interaction graph structure. This independent interaction graph structure can be called a Class-Patch graph structure, which explicitly captures the correlation between category nodes and patch nodes. It can be understood that for each category node (e.g., the j-th category node, i.e., the category node corresponding to the j-th semantic category), this category node can be combined with the patch node set to form a Class-Patch graph structure, which can be used to learn specific information.

[0194] In this embodiment, when constructing the category distribution map structure, a branch of the initial segmentation model is used as the unit of consideration. Both the first and second networks contain two branches: CNN and Transformer. That is, the initial segmentation model contains four branches. For the j-th category node in any branch, the set of patch nodes can be... And the j-th initial category node among C initial category nodes (which can be denoted as...) The combination forms the j-th initial interaction graph structure (which can be denoted as...). The j-th initial interaction graph structure is a Class-Patch graph structure, which is a lightweight fully connected graph with adjacency relationships.

[0195] The j-th initial interaction graph structure The node representation features in the graph are multiplied with their transposes to obtain a matrix product. This matrix product is then normalized to obtain the adjacency matrix corresponding to the j-th initial interaction graph structure. The adjacency matrix and the j-th initial interaction graph structure are then considered together. The relationships between the node representation features can be shown by the following formula (3):

[0196]

[0197] Among them, in formula (3) Represents the structure of the j-th initial interaction graph. The corresponding adjacency matrix, This represents matrix multiplication. The softmax(·) operation can be used to obtain a normalized adjacency matrix. The softmax(·) operation can be considered as a normalization function. Represents the structure of the j-th initial interaction graph. The nodes in the text represent features. This represents the transpose of the node representation features. The adjacency matrix is ​​also included. In other words, the j-th initial interaction graph structure It can include The set of nodes (the j-th initial category node and patch node) (all patch nodes in) Represents a set of patch nodes The total number of patch nodes in the adjacency matrix, i.e., H / r × W / r. Used to represent the similarity between nodes.

[0198] Based on the adjacency matrix and the weight matrix corresponding to the j-th initial interaction graph structure, graph convolution can be performed on the j-th initial interaction graph structure to obtain the j-th independent interaction graph structure; information aggregation is achieved through graph convolution, as shown in the following formula (4):

[0199]

[0200] Among them, in formula (4) This represents the enhanced feature map after graph convolution, i.e., the j-th independent interaction graph structure; This represents the weight matrix corresponding to the j-th initial interaction graph structure. That is, the weight matrix The size is D×D (number of channels). Using the above method, C initial category nodes can be constructed, each corresponding to a set of patch nodes. The independent interaction graph structure between them, such as the first initial category node and the set N of patch nodes. p The independent interaction graph structure between them can be represented as The second initial category node and patch node set The independent interaction graph structure between them can be represented as Following this logic, we can obtain C independent interaction graph structures; that is, each branch in the initial segmented image can correspond to C independent interaction graph structures. These C independent interaction graph structures can be combined into a comprehensive graph structure, which can be called a joint interaction graph structure.

[0201] Furthermore, in image segmentation tasks, the dependencies between semantic categories are also crucial for context modeling. Therefore, a category interaction graph structure can be constructed to explore the correlations between category nodes. Specifically, C augmented category nodes can be obtained from C independent interaction graph structures, with one augmented category node corresponding to one independent interaction graph structure. A category interaction graph structure can then be constructed for these C augmented category nodes, representing that one branch of the initial segmentation model corresponds to one category interaction graph structure. This category interaction graph structure can be denoted as... Among them, the category interaction graph structure Creation and Independent Interaction Graph Structure The creation process is similar; for example, an initial category graph structure can be created for C augmented category nodes. Obtain the initial category graph structure adjacency matrix (Similar to the aforementioned formula (3)), an enhanced category interaction graph structure can be obtained through the graph convolution operation shown in the above formula (4). The category interaction graph structure and the aforementioned joint interaction graph structure involved in the embodiments of this application can be collectively referred to as the category distribution graph structure.

[0202] In this embodiment of the application, the joint interaction graph structure can be obtained from the CNN and Transformer branches of the first network through the above steps. Interaction graph structure with categories The joint interaction graph structure is obtained from the CNN and Transformer branches of the second network, respectively. Interaction graph structure with categories The joint interaction graph structure constructed from the local joint features of the first sample obtained from the CNN branch in the first network can be called the first joint interaction graph structure (denoted as ). The category interaction graph structure constructed for it can be called the first category interaction graph structure (denoted as ). The joint interaction graph structure constructed from the joint attention features of the first samples obtained from the Transformer branch in the first network can be called the second joint interaction graph structure (denoted as ). The category interaction graph structure constructed for it can be called the second category interaction graph structure (denoted as ). The joint interaction graph structure constructed from the local joint features of the second samples obtained from the CNN branch in the second network can be called the third joint interaction graph structure (denoted as ). The category interaction graph structure constructed for it can be called a third category interaction graph structure (denoted as ). The joint interaction graph structure constructed from the joint attention features of the second samples obtained from the Transformer branch in the second network can be called the fourth joint interaction graph structure (denoted as...). The category interaction graph structure constructed for it can be called the fourth category interaction graph structure (denoted as ). ).

[0203] Please see below. Figure 9 , Figure 9 This is a schematic diagram illustrating the category consistency loss between a first network and a second network provided in an embodiment of this application. Figure 9 As shown, graph structure 40c represents the third-class interaction graph structure corresponding to the local joint features of the second sample in the second network. Graph structure 40a and graph structure 40b are both independent interactive graph structures corresponding to the local joint features of the second sample. Graph structure 40a is an independent interactive graph structure composed of the set of the second enhanced category node and the set of the patch node in the local joint features of the second sample. Graph structure 40b is an independent interactive graph structure composed of the set of the j-th enhanced category node and the set of the patch node in the local joint features of the second sample.

[0204] Similarly, graph structure 40i represents the fourth category interaction graph structure corresponding to the joint attention features of the second sample in the second network. Graph structures 40h and 40g are both independent interaction graph structures corresponding to the joint attention features of the second sample. Graph structure 40d represents the second-class interaction graph structure corresponding to the joint attention features of the first sample in the first network. Graph structures 40f and 40e are both independent interaction graph structures corresponding to the joint attention features of the first sample. Graph structure 40j represents the first category interaction graph structure corresponding to the local joint features of the first sample in the first network. Both graph structure 40m and graph structure 40k are independent interactive graph structures corresponding to the local joint features of the first sample.

[0205] Furthermore, the structure of the second joint interaction graph can be obtained. With the third joint interaction graph structure The first mean squared error (MSE) between the two values; where the first mean squared error can be denoted as... The specific calculation process can be shown in the following formula (5):

[0206]

[0207] The second category of interaction graph structure can be obtained. and the third category of interaction graph structure The second mean square error between; where the second mean square error can be denoted as... The specific calculation process can be shown in the following formula (6):

[0208]

[0209] The sum of the first mean square error and the second mean square error can be defined as the first model complementarity loss, which can then be denoted as: The specific calculation method can be as follows:

[0210] Similarly, the structure of the fourth joint interaction graph can be obtained. With the first joint interaction graph structure The third mean square error between; where the third mean square error can be denoted as... The specific calculation process can be shown in the following formula (7):

[0211]

[0212] Obtain the fourth category of interaction graph structure Interaction graph structure with the first category The fourth mean square error between; where the fourth mean square error can be denoted as... The specific calculation process can be shown in the following formula (8):

[0213]

[0214] The third and fourth mean square errors can be defined as the complementary loss of the second model, which can then be denoted as... The specific calculation method can be as follows: Finally, the sum of the first model complementarity loss and the second model complementarity loss can be determined as the class consistency loss between the first network and the second network. This class consistency loss can be denoted as... The specific calculation method can be as follows:

[0215] Step S310: Determine the unsupervised loss associated with the unlabeled sample image based on the local prediction result and global prediction result of the first sample corresponding to the unlabeled sample image; determine the fully supervised loss associated with the labeled sample image based on the local prediction result and global prediction result of the second sample corresponding to the labeled sample image and the annotation information carried by the labeled sample image.

[0216] Step S311: The class consistency loss, unsupervised loss, and fully supervised loss are weighted and summed to obtain the total model loss corresponding to the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Then, training is stopped, and the second network at the end of training is determined as the image segmentation model.

[0217] During the training of the initial segmentation model, in addition to introducing a class consistency loss applied to unlabeled sample images... In addition, it can also include fully supervised loss applied to labeled sample images. and unsupervised loss applied to unlabeled sample images This can then be used to analyze the class consistency loss. Unsupervised loss and full supervision loss By performing a weighted summation, we obtain the total model loss corresponding to the initial segmentation model. This total model loss can then be expressed as: Where L represents the total loss of the initial segmentation model, λ s Losses due to full supervision The corresponding weights can be used to constrain the fully supervised loss. λ u Unsupervised loss The corresponding weights can be used to constrain unsupervised loss. λ c For class consistency loss The corresponding weights can be used to constrain the class consistency loss. u , λ s , λ c The settings can be customized according to the actual needs of the application scenario; this embodiment does not impose any limitations on this. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Training is then stopped, and the second network at the end of training is determined as the image segmentation model. The training termination condition can be found in the foregoing. Figure 5 The relevant descriptions in step S209 of the corresponding embodiment will not be repeated here.

[0218] Understandably, the image segmentation model trained through the above steps can be built on an artificial intelligence cloud service, also known as AIaaS (AI as a Service). This is currently a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace, where all developers can access and use one or more AI services provided by the platform through Application Programming Interfaces (APIs). Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services. If the trained image segmentation model is built on an artificial intelligence cloud service, other objects (e.g., application clients) can access the image segmentation cloud service through API interfaces and perform image segmentation tasks based on the image segmentation model within that service.

[0219] This application proposes a semi-supervised image segmentation method based on CNN and Transformer. By utilizing unlabeled samples, it reduces the reliance on a large number of labeled samples, thus improving the data utilization of the image segmentation model. By introducing intra-model (within the first or second network) local-global interaction components into the segmentation model, and fully fusing local and global information in the frequency domain, the representational power of the finally extracted image features can be improved. Furthermore, during model training, by constructing a class distribution graph structure and introducing class consistency loss between models (i.e., between the first and second networks) in the graph domain, the accuracy and stability of the pseudo-labels generated by the first network for unlabeled sample images can be improved. Applying the image segmentation model trained in this way to image segmentation tasks can improve the segmentation accuracy.

[0220] Optionally, to verify the performance of the semi-supervised image segmentation method based on CNN and Transformer proposed in this application, experiments can be conducted on several publicly available multi-class datasets. These multi-class datasets may include, but are not limited to, the PASCAL VOC dataset and the Cityscapes dataset, which contain natural images with varying numbers of categories. Specifically: ① The PASCAL VOC dataset is a natural scene dataset containing 21 semantic categories; it initially consists of 1464 training images and 1449 validation images. The training set can also be augmented by using 9118 coarsely annotated images from the SBD (Semantic Boundary Dataset) dataset, resulting in 10582 training images. In this application embodiment, the aforementioned semi-supervised image segmentation method based on CNN and Transformer can be evaluated on the original (classic) dataset (1464 annotated images and 9118 unannotated images) and the blender dataset (10582 annotated images), respectively. ② The Cityscapes dataset finely annotates 19 object categories in real-world urban scenes, containing 2975 training images and 500 validation images.

[0221] Before the experiment, the following parameters can be set for the three datasets (the original PASCAL VOC dataset, the hybrid PASCAL VOC dataset, and the Cityscapes dataset): crop size (cs), batch size (bs), learning rate (lr), maximum training epochs (ep), and solver (optimization algorithm, opt). Specifically: ① PASCAL VOC dataset: cs = 513 × 513; bs = 16; lr = 0.001; ep = 80; opt = SGD (Stochastic Gradient Descent). ② Cityscapes dataset: cs = 769 × 769; bs = 16; lr = 0.01; ep = 200; opt = SGD. Mean Intersection-over-Union (mIoU) can be used as the model evaluation metric for all these datasets.

[0222] In experimental verification, the semi-supervised image segmentation method based on CNN and Transformer provided in this application embodiment can be divided into two methods: one is a semi-supervised image segmentation method that does not use class consistency loss between models for model training (hereinafter referred to as SemiCVT-), and the other is the complete technical method proposed in this application embodiment, namely, a semi-supervised image segmentation method that uses class consistency loss between models for model training (hereinafter referred to as SemiCVT). To evaluate the segmentation performance of Method 1 and Scheme 2 proposed in this application embodiment, they can be compared with fully supervised (sup) segmentation methods and existing semi-supervised segmentation methods (SSL methods), and the segmentation performance of each model can be quantitatively compared using mIoU.

[0223] Optionally, on the original PASCAL VOC dataset, the SemiCVT proposed in this application embodiment can be compared with the baseline (benchmark model, based on CNN network structure) of fully supervised learning, CVT (the parallel CNN-Transformer network in the image segmentation model proposed in this application embodiment), and existing algorithms such as MT (Mean-Teacher algorithm), CutMix (data augmentation), PseudoSeg (a method for generating labeled structured pseudo-labels for training on unlabeled or weakly labeled data), and PC. 2Seg (Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation, a semi-supervised semantic segmentation method based on pixel contrast consistency), ST++ (a self-training scheme for semi-supervised semantic segmentation), U 2 Performance comparisons were performed on semi-supervised segmentation methods such as PL (a semi-supervised semantic segmentation scheme using unreliable pseudo-labels). The mIoU values ​​of these methods are shown in Table 1 below:

[0224] Table 1

[0225]

[0226] Optionally, on the hybrid PASCAL VOC dataset, the SemiCVT- and SemiCVT schemes proposed in this application can be compared with the baseline of fully supervised learning (using only labeled sample images), CVT, as well as MT, CutMix, CCT (Compact Convolutional Transformer), GCT (Gated Channel Attention Mechanism), ST++, and U... 2 Performance comparisons were performed using semi-supervised segmentation methods such as PL, and the mIoU values ​​of these methods are shown in Table 2 below:

[0227] Table 2

[0228]

[0229] Optionally, on the Cityscapes dataset, the SemiCVT- and SemiCVT schemes proposed in this application can be compared with the fully supervised learning baseline, CVT, and existing MT, CutMix, CCT, GCT, ST++, and U... 2 Performance comparisons were performed using semi-supervised segmentation methods such as PL, and the mIoU values ​​of these methods are shown in Table 3 below:

[0230] Table 3

[0231]

[0232]

[0233] Optionally, please see Figure 10 , Figure 10 This is a visual schematic diagram of an image segmentation result provided in an embodiment of this application. For example... Figure 10As shown, for input image 50a in the PASCAL VOC dataset, the corresponding ground truth label (annotation information) is shown in image 50b; in other words, image 50b represents the true segmentation of input image 50a, and can also be considered as the expected prediction result of various models. Image 50c is the result of using existing methods (e.g., U...). 2 Image 50d is the segmentation prediction result obtained after performing image semantic segmentation processing on input image 50a using the SemiCVT scheme proposed in this application embodiment; Image 50e is the segmentation prediction result obtained after performing image semantic segmentation processing on input image 50a using the SemiCVT scheme proposed in this application embodiment.

[0234] In this embodiment, the segmentation performance is primarily evaluated on two multi-class datasets, including quantitative results (such as mIoU values) comparing with other existing schemes, and visual segmentation results (such as...). Figure 10 (As shown). Specifically, Tables 1 and 2 present the quantitative performance of SemiCVT-, SemiCVT, and other existing methods on different datasets. In fully supervised experiments (model training using only labeled sample images), the CVT network structure outperformed the baseline network by +1.19%, +4.82%, +2.5%, +1.99%, and +2.05% respectively when using 1 / 16, 1 / 8, 1 / 4, 1 / 2, and all labeled data in the dataset for fully supervised learning. The CVT network structure even surpasses the semi-supervised method based on MT (trained using additional unlabeled data). These experimental results demonstrate that the CVT network structure proposed in this application can extract complementary information from CNN and Transformer, achieving satisfactory performance even with limited labeled data. In the semi-supervised experiments, this application first... c The experiment was conducted under the following conditions (i.e., the SemiCVT scheme), as shown in Tables 1 to 3, and Figure 10 The visualized segmentation prediction results show that SemiCVT- outperforms existing semi-supervised segmentation methods. Furthermore, it utilizes a class consistency loss L between models. cThe SemiCVT approach further improves image segmentation performance, achieving the highest segmentation accuracy in our experiments. It's worth noting that compared to semi-supervised segmentation methods based on machine learning (MT), SemiCVT achieves significant improvements in image segmentation: +16.84%, +12.33%, +11.13%, +9.03%, and +9.36%, respectively. Similarly, it surpasses existing semi-supervised segmentation methods on the BlenderPASCAL VOC and Cityscapes datasets.

[0235] Figure 10 This application demonstrates the method (SemiCVT-, SemiCVT) and the existing method (U). 2 The visualization of the segmentation prediction results of the PL method on the PASCALVOC dataset is shown. Clearly, the SemiCVT method proposed in this embodiment can accurately segment objects of different sizes, shapes, and locations; the boundaries of the segmentation results are clearer than those of other existing semi-supervised segmentation methods, reducing the probability of incorrect segmentation and improving the accuracy of image segmentation.

[0236] It is understood that the specific implementation of this application may involve image segmentation processing in traffic management systems and medical management systems. When the above embodiments of this application are applied to specific products or technologies, permission or consent from relevant institutions or departments, or the user themselves, is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0237] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an image segmentation processing device provided in an embodiment of this application. Figure 1 .like Figure 11 As shown, the image segmentation processing device 1 includes: a first feature extraction module 11, a first frequency domain conversion module 12, a first feature enhancement module 13, a residual connection module 14, and a first segmentation prediction module 15;

[0238] The first feature extraction module 11 is used to acquire a source image, extract features from the source image to obtain a first spatial local feature, and perform attention encoding processing on the source image to obtain a first spatial attention feature; the first spatial attention feature is used to characterize the information transfer between global information in the source image;

[0239] The first frequency domain conversion module 12 is used to convert the first spatial local features into local frequency domain features, convert the first spatial attention features into attention frequency domain features, obtain the first interaction weight associated with the local frequency domain features, and obtain the second interaction weight associated with the attention frequency domain features.

[0240] The first feature enhancement module 13 is used to enhance the attention frequency domain features according to the first interaction weight to obtain attention-enhanced frequency domain features, enhance the local frequency domain features according to the second interaction weight to obtain local enhanced frequency domain features, convert the local frequency domain features and the local enhanced frequency domain features into second spatial local features, and convert the attention frequency domain features and the attention-enhanced frequency domain features into second spatial attention features.

[0241] The residual connection module 14 is used to combine the first spatial local features and the second spatial local features into local joint features, and to combine the first spatial attention features, the second spatial attention features, and the local joint features into attention joint features;

[0242] The first segmentation prediction module 15 is used to obtain the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features, and generate the segmented image corresponding to the source image based on the average value of the local segmentation prediction result and the global segmentation prediction result.

[0243] In one or more embodiments, the first feature extraction module 11 is specifically used for:

[0244] The source image is obtained and input into the image feature extraction component in the image segmentation model. The initial image features corresponding to the source image are obtained through the image feature extraction component.

[0245] The initial image features are input into the local feature extractor in the image segmentation model, and the first spatial local features corresponding to the initial image features are obtained through the local feature extractor; the local feature extractor includes N convolutional components, where N is a positive integer;

[0246] The initial image features are input into the global feature extractor in the image segmentation model, and the first spatial attention features corresponding to the initial image features are obtained through the global feature extractor; the global feature extractor includes N attention components.

[0247] In one or more embodiments, the first feature extraction module 11 obtains initial image features corresponding to the source image through the image feature extraction component, including:

[0248] The source image is convolved by the convolutional layer in the image feature extraction component to obtain the image convolution result corresponding to the source image.

[0249] The image convolution result is normalized by the normalization layer in the image feature extraction component to obtain image normalized features. The image normalized features are then activated by the activation function in the image feature extraction component to obtain image activation features.

[0250] The pooling layer in the image feature extraction component performs pooling operations on the image activation features to obtain the initial image features corresponding to the source image.

[0251] In one or more embodiments, the image segmentation model further includes an interaction component for exchanging frequency domain information between the i-th convolutional component in the local feature extractor and the i-th attention component in the global feature extractor, where i is a positive integer less than or equal to N.

[0252] The first feature extraction module 11 obtains the first spatial local features corresponding to the initial image features through a local feature extractor, including:

[0253] Obtain the input features of the i-th convolutional component in the local feature extractor; when i is 1, the input features of the i-th convolutional component are the initial image features; when i is not 1, the input features of the i-th convolutional component include the output features of the (i-1)-th convolutional component, as well as the spatial local features output by the interaction components acting on the (i-1)-th attention component and the (i-1)-th convolutional component.

[0254] Based on the multiple convolutional layers in the i-th convolutional component, perform convolution operations on the input features of the i-th convolutional component to obtain the local convolution result;

[0255] Residual connections are performed between the local convolution result and the input features of the i-th convolutional component to obtain the output features of the i-th convolutional component. The output features of the N-th convolutional component in the local feature extractor are then determined as the first spatial local features.

[0256] In one or more embodiments, the first feature extraction module 11 obtains the first spatial attention feature corresponding to the initial image features through a global feature extractor, including:

[0257] Obtain the input features of the i-th attention component in the global feature extractor; when i is 1, the input features of the i-th attention component are the initial image features; when i is not 1, the input features of the i-th attention component include the output features of the (i-1)-th attention component, the output features of the (i-1)-th convolutional component, and the spatial attention features and spatial local features output by the interaction components acting on the (i-1)-th attention component and the (i-1)-th convolutional component.

[0258] Based on the transformation weight matrix corresponding to the i-th attention component, the input features of the i-th attention component are transformed into a query component, a key matrix, and a value matrix. The dot product operation is performed on the transpose of the query matrix and the key matrix to obtain the candidate weight matrix.

[0259] Obtain the number of columns in the query matrix, normalize the ratio between the candidate weight matrix and the square root of the number of columns to obtain the attention weight matrix, and determine the dot product between the attention weight matrix and the value matrix as the attention encoding feature.

[0260] The attention encoding features and the input features of the i-th attention component are combined to form the attention residual features. The attention residual features are then transformed according to the multilayer perceptron in the i-th attention component to obtain the attention transformed features.

[0261] The attention residual features and attention transformation features are combined into the output features of the i-th attention component, and the output features of the N-th attention component in the global feature extractor are determined as the first spatial attention features.

[0262] In one or more embodiments, the first frequency domain conversion module 12 converts the first spatial local features into local frequency domain features, including:

[0263] The first spatial local features are input into the interactive component in the image segmentation model. The interactive component performs a discrete Fourier transform on the first spatial local features to obtain the spectral features corresponding to the first spatial local features.

[0264] Half of the spectral feature corresponding to the first spatial local feature is determined as the local frequency domain feature corresponding to the first spatial local feature.

[0265] In one or more embodiments, the first frequency domain conversion module 12 obtains the first interaction weight associated with local frequency domain features, including:

[0266] The first spectral convolution result is obtained by performing a convolution operation on the real part information of the local frequency domain features based on the first convolutional layer in the interactive component.

[0267] The first spectral convolution result is normalized according to the normalization layer in the interactive component to obtain spectral normalized features. The spectral normalized features are then activated using the first activation function in the interactive component to obtain spectral activated features.

[0268] The second convolutional layer in the interactive component performs a convolution operation on the spectral activation features to obtain the second spectral convolution result. The second activation function in the interactive component is then used to activate the second spectral convolution result to obtain the first interactive weight associated with the local frequency domain features.

[0269] In one or more embodiments, the first feature enhancement module 13 converts local frequency domain features and locally enhanced frequency domain features into second spatial local features, including:

[0270] The local frequency domain features and the local enhanced frequency domain features are combined into frequency domain residual features. The frequency domain residual features are then subjected to inverse Fourier transform through interactive components to obtain the spatial domain features corresponding to the frequency domain residual features.

[0271] Obtain the hidden weight matrix and bias vector corresponding to the multilayer perceptron in the interactive component. Based on the bias vector and the dot product between the hidden weight matrix and the spatial domain features, determine the second spatial local features corresponding to the source image.

[0272] In one or more embodiments, the first segmentation prediction module 15 obtains local segmentation prediction results corresponding to local joint features and global segmentation prediction results corresponding to attention joint features, including:

[0273] The local joint features are input into the local segmentation component of the image segmentation model. The local segmentation component upsamples the local joint features to obtain the local segmentation prediction result corresponding to the source image.

[0274] The attention joint features are input into the global segmentation component of the image segmentation model. The global segmentation component upsamples the attention joint features to obtain the global segmentation prediction result corresponding to the source image.

[0275] According to one embodiment of this application, the foregoing Figure 3 and Figure 5 The steps involved in the image segmentation processing method shown can be derived from... Figure 11 The image segmentation processing apparatus 1 shown is executed by each module. For example, Figure 3 The step S101 shown can be performed by Figure 11 The first feature extraction module 11 shown is used to perform this operation. Figure 3 The step S102 shown can be performed by Figure 11 The first frequency domain conversion module 12 shown is used to perform this. Figure 3 The step S103 shown can be performed by Figure 11 The first feature enhancement module 13 shown is used to perform this. Figure 3 The step S104 shown can be performed by Figure 11 The residual connection module 14 shown is used to perform this. Figure 3 The step S105 shown can be performed by Figure 11 The first segmentation prediction module 15 shown is used to perform the above.

[0276] According to one embodiment of this application, Figure 11The modules in the image segmentation processing apparatus 1 shown can be individually or entirely combined into one or more units, or some of the units can be further divided into at least two functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one unit. In other embodiments of this application, the image segmentation processing apparatus 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by at least two units.

[0277] In this embodiment, after obtaining the source image, a first spatial local feature and a first spatial attention feature can be extracted from the source image. The first spatial attention feature is used to characterize the information transfer between global information in the source image. That is, both local and global information in the source image can be extracted, which can improve the representation ability of image features. Furthermore, the interaction weights associated with the first spatial local feature and the first spatial attention feature can be obtained in the frequency domain. By exchanging the interaction weights, enhancement processing is performed to obtain local frequency domain enhancement features and attention frequency domain enhancement features, which can fully integrate local and global information in the source image in the frequency domain. Then, it can be transformed to the spatial domain, and residual connections can be performed on the second spatial local feature and the second spatial attention feature after local-global interaction. This can combine the complementarity between local spatial features and spatial attention features, thereby generating a more accurate segmented image and improving the accuracy of image segmentation results.

[0278] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an image segmentation processing device provided in an embodiment of this application. Figure 2 .like Figure 12 As shown, the image segmentation processing device 2 includes: a sample image acquisition module 21, a second feature extraction module 22, a second frequency domain conversion module 23, a second feature enhancement module 24, a second segmentation prediction module 25, and a network parameter correction module 26.

[0279] The sample image acquisition module 21 is used to acquire labeled sample images and unlabeled sample images; the labeled sample images are used to input the first network in the initial segmentation model, and the unlabeled sample images are used to input the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure.

[0280] The second feature extraction module 22 is used to obtain the first sample local features corresponding to the unlabeled sample image through the initial local feature extractor in the first network, and to obtain the first sample attention features corresponding to the unlabeled sample image through the initial global feature extractor in the first network; the first sample attention features are used to characterize the information transfer between global information of the unlabeled sample images;

[0281] The second frequency domain conversion module 23 is used to convert the local features of the first sample into local frequency domain features of the sample through the interactive components in the first network, convert the attention features of the first sample into frequency domain features of the sample attention, obtain the third interactive weight associated with the local frequency domain features of the sample, and obtain the fourth interactive weight associated with the frequency domain features of the sample attention.

[0282] The second feature enhancement module 24 is used to enhance the sample attention frequency domain features according to the third interaction weight to obtain sample attention enhanced frequency domain features, enhance the sample local frequency domain features according to the fourth interaction weight to obtain sample local enhanced frequency domain features, convert the sample local frequency domain features and sample local enhanced frequency domain features into second sample local features, and convert the sample attention frequency domain features and sample attention enhanced frequency domain features into second sample attention features.

[0283] The second segmentation prediction module 25 is used to combine the local features of the first sample and the local features of the second sample into a first sample local joint feature, combine the attention features of the first sample, the attention features of the second sample and the local joint features of the first sample into a first sample attention joint feature, output the local prediction result of the first sample corresponding to the local joint features of the first sample through the initial local segmentation component in the first network, and output the global prediction result of the first sample corresponding to the attention joint features of the first sample through the initial global segmentation component in the first network.

[0284] The network parameter correction module 26 is used to input labeled sample images and unlabeled sample images into the second network, output the local prediction results of the second sample and the global prediction results of the second sample through the second network, and correct the network parameters of the initial segmentation model based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample images. The second network containing the corrected network parameters is determined as the image segmentation model. The image segmentation model is used to output the segmented image corresponding to the source image.

[0285] In one or more embodiments, the network parameter correction module 26 corrects the network parameters of the initial segmentation model based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, and determines the second network containing the corrected network parameters as the image segmentation model, including:

[0286] Based on the local prediction results and global prediction results of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image;

[0287] Based on the local prediction results of the second sample corresponding to the labeled sample image, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image;

[0288] The unsupervised loss and the fully supervised loss are weighted and summed to obtain the total loss of the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total loss until the total loss meets the training termination condition. Training is then stopped, and the second network at the end of training is determined as the image segmentation model.

[0289] In one or more embodiments, the network parameter correction module 26 is specifically used for:

[0290] The second network obtains the second sample local joint features and the second sample attention joint features corresponding to the unlabeled sample images. The initial local segmentation component in the second network outputs the second sample local prediction results corresponding to the second sample local joint features. The initial global segmentation component in the second network outputs the second sample global prediction results corresponding to the second sample attention joint features.

[0291] A class distribution map structure is created for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features corresponding to the unlabeled sample images. The class consistency loss between the first network and the second network is determined based on the class distribution map structure.

[0292] Based on the local prediction results and global prediction results of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image;

[0293] Based on the local prediction results of the second sample corresponding to the labeled sample image, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image;

[0294] The class consistency loss, unsupervised loss, and fully supervised loss are weighted and summed to obtain the total model loss corresponding to the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Training is then stopped, and the second network at the end of training is determined as the image segmentation model.

[0295] In one or more embodiments, both the first and second networks in the initial segmentation model are used to segment C semantic categories, where C is a positive integer;

[0296] The network parameter correction module 26 creates a class distribution map structure for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features. Based on the class distribution map structure, it determines the class consistency loss between the first network and the second network, including:

[0297] Obtain the average feature value of the pixel features belonging to the same semantic category in the local joint features of the first sample, and determine the average feature value of the C semantic categories as the C initial category nodes corresponding to the local joint features of the first sample;

[0298] The local joint features of the first sample are subjected to average pooling to obtain the set of patch nodes corresponding to the local joint features of the first sample.

[0299] For the patch node set and C initial category nodes, construct C independent interaction graph structures, combine the C independent interaction graph structures into a first joint interaction graph structure, obtain C enhanced category nodes from the C independent interaction graph structures, and construct a first category interaction graph structure for the C enhanced category nodes; the first joint interaction graph structure and the first category interaction graph structure belong to the category distribution graph structure corresponding to the local joint features of the first sample;

[0300] Obtain the second joint interaction graph structure and the second category interaction graph structure corresponding to the joint attention features of the first sample; obtain the third joint interaction graph structure and the third category interaction graph structure corresponding to the local joint features of the second sample; obtain the fourth joint interaction graph structure and the fourth category interaction graph structure corresponding to the joint attention features of the second sample.

[0301] Obtain the first mean square error between the second joint interaction graph structure and the third joint interaction graph structure, obtain the second mean square error between the second category interaction graph structure and the third category interaction graph structure, and determine the sum of the first mean square error and the second mean square error as the first model complementary loss.

[0302] The third mean square error between the fourth joint interaction graph structure and the first joint interaction graph structure is obtained, and the fourth mean square error between the fourth category interaction graph structure and the first category interaction graph structure is obtained. The third mean square error and the fourth mean square error are determined as the second model complementarity loss, and the sum of the first model complementarity loss and the second model complementarity loss is determined as the category consistency loss between the first network and the second network.

[0303] In one or more embodiments, the network parameter correction module 26 constructs C independent interaction graph structures for the patch node set and C initial category nodes, including:

[0304] Combine the set of patch nodes and the j-th initial category node from the C initial category nodes to form the j-th initial interaction graph structure; j is a positive integer less than or equal to C;

[0305] Multiply the node representation features in the j-th initial interaction graph structure with the transpose of the node representation features to obtain the matrix product result. Normalize the matrix product result to obtain the adjacency matrix corresponding to the j-th initial interaction graph structure.

[0306] Based on the adjacency matrix and the weight matrix corresponding to the j-th initial interaction graph structure, graph convolution is performed on the j-th initial interaction graph structure to obtain the j-th independent interaction graph structure.

[0307] According to one embodiment of this application, the foregoing Figure 5 The steps involved in the image segmentation processing method shown can be derived from... Figure 12 The image segmentation processing device 2 shown is executed by each module. For example, Figure 5 The step S201 shown can be performed by Figure 12 The sample image acquisition module 21 shown is used to perform this task. Figure 5 The step S202 shown can be performed by Figure 12 The second feature extraction module 22 shown is used to perform this operation. Figure 5 The step S203 shown can be performed by Figure 12 The second frequency domain conversion module 23 shown is used to perform this. Figure 5 Steps S204 and S205 shown can be derived from... Figure 12 The second feature enhancement module 24 shown is used to perform this. Figure 5 Steps S206 and S207 shown can be derived from... Figure 12 The second segmentation prediction module 25 shown is used to perform this. Figure 5 Steps S208 and S209 shown are by Figure 12 The network parameter correction module 26 shown is used to perform the above.

[0308] According to one embodiment of this application, Figure 12The modules in the image segmentation processing apparatus 2 shown can be individually or entirely combined into one or more units, or some of the units can be further divided into at least two functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by at least two units, or the function of at least two modules can be implemented by one unit. In other embodiments of this application, the image segmentation processing apparatus 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by at least two units.

[0309] This application proposes a semi-supervised technique based on CNN and Transformer, which reduces the reliance on a large number of labeled samples by utilizing unlabeled samples, thereby improving the data utilization of the image segmentation model. By introducing intra-model (within the first or second network) local-global interaction components into the segmentation model, and by fully fusing local and global information in the frequency domain, the representational power of the finally extracted image features can be improved. Furthermore, during model training, by constructing a class distribution graph structure and introducing class consistency loss between models (i.e., between the first and second networks) in the graph domain, the accuracy and stability of the pseudo-labels generated by the first network for unlabeled sample images can be improved. Applying the image segmentation model trained in this way to image segmentation tasks can improve the segmentation accuracy of images.

[0310] Further, please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1000 can be a terminal device, for example, the one described above. Figure 1 The terminal device 10a in the corresponding embodiment can also be a server, for example, as described above. Figure 1The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes a computer device as an example as the terminal device. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located remotely from the aforementioned processor 1001. Figure 13 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0311] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display screen and a keyboard. Figure 13 In the computer device 1000 shown, the network interface 1004 provides network communication functions; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application stored in the memory 1005.

[0312] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 5 as well as Figure 7 The description of the image segmentation processing method in any of the embodiments can also be performed using the methods described above. Figure 11 The description of the image segmentation processing device 1 in the corresponding embodiment, or the execution of the preceding text Figure 12 The description of the image segmentation processing apparatus 2 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0313] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the image segmentation processing apparatus 1 or the image segmentation processing apparatus 2 mentioned above. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 5 as well as Figure 7 The description of the image segmentation processing method in any of the embodiments is already provided, and therefore will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here either. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed and executed on a computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed in multiple locations and interconnected through a communication network. Multiple computing devices distributed in multiple locations and interconnected through a communication network can constitute a blockchain system.

[0314] Furthermore, it should be noted that this application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, causing the computer device to perform the aforementioned actions. Figure 3 , Figure 5 as well as Figure 7 The description of the image segmentation processing method in any of the embodiments is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0315] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different media content, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0316] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0317] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0318] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image segmentation processing method, characterized in that, include: A source image is acquired, and feature extraction is performed on the source image to obtain a first spatial local feature. Attention encoding is then performed on the source image to obtain a first spatial attention feature. The first spatial attention feature is used to characterize the information transfer between global information in the source images; The first spatial local feature is converted into a local frequency domain feature, the first spatial attention feature is converted into an attention frequency domain feature, the first interaction weight associated with the local frequency domain feature is obtained, and the second interaction weight associated with the attention frequency domain feature is obtained. The attention frequency domain features are enhanced according to the first interaction weight to obtain attention-enhanced frequency domain features. The local frequency domain features are enhanced according to the second interaction weight to obtain local enhanced frequency domain features. The local frequency domain features and the local enhanced frequency domain features are converted into second spatial local features. The attention frequency domain features and the attention-enhanced frequency domain features are converted into second spatial attention features. The first spatial local feature and the second spatial local feature are combined into a local joint feature, and the first spatial attention feature, the second spatial attention feature, and the local joint feature are combined into an attention joint feature; Obtain the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features. Generate the segmented image corresponding to the source image based on the average value of the local segmentation prediction result and the global segmentation prediction result.

2. The method according to claim 1, characterized in that, The process of acquiring a source image, extracting features from the source image to obtain first spatial local features, and performing attention encoding on the source image to obtain first spatial attention features includes: A source image is acquired, and the source image is input into the image feature extraction component in the image segmentation model. The initial image features corresponding to the source image are obtained through the image feature extraction component. The initial image features are input into the local feature extractor in the image segmentation model, and the first spatial local features corresponding to the initial image features are obtained through the local feature extractor; the local feature extractor includes N convolutional components, where N is a positive integer; The initial image features are input into the global feature extractor in the image segmentation model, and the first spatial attention features corresponding to the initial image features are obtained through the global feature extractor; the global feature extractor includes N attention components.

3. The method according to claim 2, characterized in that, The step of obtaining the initial image features corresponding to the source image through the image feature extraction component includes: The source image is convolved using the convolutional layer in the image feature extraction component to obtain the image convolution result corresponding to the source image. The image convolution result is normalized by the normalization layer in the image feature extraction component to obtain image normalized features. The image normalized features are then activated by the activation function in the image feature extraction component to obtain image activation features. The image activation features are pooled using the pooling layer in the image feature extraction component to obtain the initial image features corresponding to the source image.

4. The method according to claim 2, characterized in that, The image segmentation model further includes an interaction component, which is used to exchange frequency domain information between the i-th convolutional component in the local feature extractor and the i-th attention component in the global feature extractor, where i is a positive integer less than or equal to N; The step of obtaining the first spatial local features corresponding to the initial image features through the local feature extractor includes: Obtain the input features of the i-th convolutional component in the local feature extractor; when i is 1, the input features of the i-th convolutional component are the initial image features; when i is not 1, the input features of the i-th convolutional component include the output features of the (i-1)-th convolutional component, and the spatial local features output by the interaction component acting on the (i-1)-th attention component and the (i-1)-th convolutional component. Based on the multiple convolutional layers in the i-th convolutional component, perform convolution operations on the input features of the i-th convolutional component to obtain the local convolution result; The local convolution result and the input features of the i-th convolutional component are residually connected to obtain the output features of the i-th convolutional component. The output features of the N-th convolutional component in the local feature extractor are then determined as the first spatial local features.

5. The method according to claim 4, characterized in that, The step of obtaining the first spatial attention feature corresponding to the initial image features through the global feature extractor includes: Obtain the input features of the i-th attention component in the global feature extractor; when i is 1, the input features of the i-th attention component are the initial image features; when i is not 1, the input features of the i-th attention component include the output features of the (i-1)-th attention component, the output features of the (i-1)-th convolutional component, and the spatial attention features and spatial local features output by the interaction component acting on the (i-1)-th attention component and the (i-1)-th convolutional component. Based on the transformation weight matrix corresponding to the i-th attention component, the input features of the i-th attention component are converted into a query component, a key matrix, and a value matrix. The query matrix and the transpose of the key matrix are multiplied by a dot product to obtain the candidate weight matrix. The number of columns in the query matrix is ​​obtained, and the ratio between the candidate weight matrix and the square root of the number of columns is normalized to obtain the attention weight matrix. The dot product between the attention weight matrix and the value matrix is ​​determined as the attention encoding feature. The attention encoding features and the input features of the i-th attention component are combined to form attention residual features. The attention residual features are then transformed according to the multilayer perceptron in the i-th attention component to obtain attention transformed features. The attention residual feature and the attention transformation feature are combined into the output feature of the i-th attention component, and the output feature of the N-th attention component in the global feature extractor is determined as the first spatial attention feature.

6. The method according to claim 1, characterized in that, The step of converting the first spatial local features into local frequency domain features includes: The first spatial local feature is input into the interactive component in the image segmentation model. The interactive component performs a discrete Fourier transform on the first spatial local feature to obtain the spectral feature corresponding to the first spatial local feature. Half of the spectral feature corresponding to the first spatial local feature is determined as the local frequency domain feature corresponding to the first spatial local feature.

7. The method according to claim 6, characterized in that, The step of obtaining the first interaction weight associated with the local frequency domain features includes: The real part information of the local frequency domain features is convolved by the first convolutional layer in the interactive component to obtain the first spectral convolution result. The first spectral convolution result is normalized according to the normalization layer in the interactive component to obtain spectral normalization features. The spectral normalization features are then activated using the first activation function in the interactive component to obtain spectral activation features. The spectral activation features are convolved by the second convolutional layer in the interaction component to obtain a second spectral convolution result. The second activation function in the interaction component is then used to activate the second spectral convolution result to obtain the first interaction weight associated with the local frequency domain features.

8. The method according to claim 6, characterized in that, The step of converting the local frequency domain features and the local enhanced frequency domain features into second spatial local features includes: The local frequency domain features and the local enhanced frequency domain features are combined into frequency domain residual features. The frequency domain residual features are then subjected to inverse Fourier transform through the interactive component to obtain the spatial domain features corresponding to the frequency domain residual features. Obtain the hidden weight matrix and bias vector corresponding to the multilayer perceptron in the interactive component, and determine the second spatial local features corresponding to the source image based on the bias vector and the dot product between the hidden weight matrix and the spatial domain features.

9. The method according to claim 1, characterized in that, The step of obtaining the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features includes: The local joint features are input into the local segmentation component of the image segmentation model. The local segmentation component performs upsampling processing on the local joint features to obtain the local segmentation prediction result corresponding to the source image. The attention joint features are input into the global segmentation component of the image segmentation model. The global segmentation component then upsamples the attention joint features to obtain the global segmentation prediction result corresponding to the source image.

10. An image segmentation processing method, characterized in that, include: Obtain labeled and unlabeled sample images; The labeled sample images are used as input to the first network in the initial segmentation model, and the unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure. The first sample local features corresponding to the unlabeled sample image are obtained through the initial local feature extractor in the first network, and the first sample attention features corresponding to the unlabeled sample image are obtained through the initial global feature extractor in the first network. The first sample attention feature is used to characterize the information transfer between global information of the unlabeled sample images; The first sample local features are converted into sample local frequency domain features by the interaction components in the first network, the first sample attention features are converted into sample attention frequency domain features, the third interaction weight associated with the sample local frequency domain features is obtained, and the fourth interaction weight associated with the sample attention frequency domain features is obtained. The sample attention frequency domain features are enhanced according to the third interaction weight to obtain sample attention enhanced frequency domain features. The sample local frequency domain features are enhanced according to the fourth interaction weight to obtain sample local enhanced frequency domain features. The sample local frequency domain features and the sample local enhanced frequency domain features are converted into second sample local features. The sample attention frequency domain features and the sample attention enhanced frequency domain features are converted into second sample attention features. The first sample local features and the second sample local features are combined into a first sample local joint feature. The first sample attention feature, the second sample attention feature, and the first sample local joint feature are combined into a first sample attention joint feature. The first sample local prediction result corresponding to the first sample local joint feature is output through the initial local segmentation component in the first network. The first sample global prediction result corresponding to the first sample attention joint feature is output through the initial global segmentation component in the first network. The labeled sample image and the unlabeled sample image are input into the second network. The second network outputs the local prediction result of the second sample and the global prediction result of the second sample. Based on the local prediction result of the first sample, the global prediction result of the first sample, the local prediction result of the second sample, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, the network parameters of the initial segmentation model are corrected. The second network containing the corrected network parameters is determined as the image segmentation model. The image segmentation model is used to output the segmented image corresponding to the source image.

11. The method according to claim 10, characterized in that, The step of correcting the network parameters of the initial segmentation model based on the local prediction results of the first sample, the global prediction results of the first sample, the local prediction results of the second sample, the global prediction results of the second sample, and the annotation information carried by the labeled sample image, and determining the second network containing the corrected network parameters as the image segmentation model, includes: Based on the local prediction result of the first sample and the global prediction result of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image; Based on the local prediction result of the second sample corresponding to the labeled sample image, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image; The unsupervised loss and the fully supervised loss are weighted and summed to obtain the total model loss corresponding to the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Then, training is stopped, and the second network at the end of training is determined as the image segmentation model.

12. The method according to claim 10, characterized in that, The second network outputs the local prediction result of the second sample and the global prediction result of the second sample. Based on the local prediction result of the first sample, the global prediction result of the first sample, the local prediction result of the second sample, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, the network parameters of the initial segmentation model are corrected, and the second network containing the corrected network parameters is determined as the image segmentation model. The image segmentation model is used to output a segmented image corresponding to the source image, including: The second network obtains the second sample local joint features and the second sample attention joint features corresponding to the unlabeled sample image. The initial local segmentation component in the second network outputs the second sample local prediction result corresponding to the second sample local joint features. The initial global segmentation component in the second network outputs the second sample global prediction result corresponding to the second sample attention joint features. A category distribution map structure is created for the first sample local joint feature, the first sample attention joint feature, the second sample local joint feature, and the second sample attention joint feature corresponding to the unlabeled sample image. The category consistency loss between the first network and the second network is determined based on the category distribution map structure. Based on the local prediction result of the first sample and the global prediction result of the first sample corresponding to the unlabeled sample image, determine the unsupervised loss associated with the unlabeled sample image; Based on the local prediction result of the second sample corresponding to the labeled sample image, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, determine the fully supervised loss associated with the labeled sample image; The class consistency loss, the unsupervised loss, and the fully supervised loss are weighted and summed to obtain the total model loss corresponding to the initial segmentation model. The network parameters of the initial segmentation model are iteratively trained based on the total model loss until the total model loss meets the training termination condition. Then, training is stopped, and the second network at the end of training is determined as the image segmentation model.

13. The method according to claim 12, characterized in that, In the initial segmentation model, both the first network and the second network are used to segment C semantic categories, where C is a positive integer. The step of creating a category distribution map structure for the first sample local joint features, the first sample attention joint features, the second sample local joint features, and the second sample attention joint features, and determining the category consistency loss between the first network and the second network based on the category distribution map structure, includes: Obtain the average feature value of pixel features belonging to the same semantic category in the local joint features of the first sample, and determine the average feature value of C semantic categories as the C initial category nodes corresponding to the local joint features of the first sample; The local joint features of the first sample are subjected to average pooling to obtain the set of patch nodes corresponding to the local joint features of the first sample. C independent interaction graph structures are constructed for the patch node set and the C initial category nodes. The C independent interaction graph structures are combined into a first joint interaction graph structure. C enhanced category nodes are obtained from the C independent interaction graph structures. A first category interaction graph structure is constructed for the C enhanced category nodes. The first joint interaction graph structure and the first category interaction graph structure belong to the category distribution graph structure corresponding to the local joint features of the first sample. Obtain the second joint interaction graph structure and the second category interaction graph structure corresponding to the joint attention features of the first sample; obtain the third joint interaction graph structure and the third category interaction graph structure corresponding to the local joint features of the second sample; obtain the fourth joint interaction graph structure and the fourth category interaction graph structure corresponding to the joint attention features of the second sample. Obtain the first mean square error between the second joint interaction graph structure and the third joint interaction graph structure, obtain the second mean square error between the second category interaction graph structure and the third category interaction graph structure, and determine the sum of the first mean square error and the second mean square error as the first model complementary loss; The third mean square error between the fourth joint interaction graph structure and the first joint interaction graph structure is obtained, and the fourth mean square error between the fourth category interaction graph structure and the first category interaction graph structure is obtained. The third mean square error and the fourth mean square error are determined as the second model complementarity loss, and the sum of the first model complementarity loss and the second model complementarity loss is determined as the category consistency loss between the first network and the second network.

14. The method according to claim 13, characterized in that, The construction of C independent interaction graph structures for the patch node set and the C initial category nodes includes: The patch node set and the j-th initial category node from the C initial category nodes are combined to form the j-th initial interaction graph structure; j is a positive integer less than or equal to C; Multiply the node representation features in the j-th initial interaction graph structure with the transpose of the node representation features to obtain a matrix product result. Normalize the matrix product result to obtain the adjacency matrix corresponding to the j-th initial interaction graph structure. Based on the adjacency matrix and the weight matrix corresponding to the j-th initial interaction graph structure, graph convolution is performed on the j-th initial interaction graph structure to obtain the j-th independent interaction graph structure.

15. An image segmentation processing apparatus, characterized in that, include: The first feature extraction module is used to acquire a source image, extract features from the source image to obtain a first spatial local feature, and perform attention encoding processing on the source image to obtain a first spatial attention feature; The first spatial attention feature is used to characterize the information transfer between global information in the source images; The first frequency domain conversion module is used to convert the first spatial local features into local frequency domain features, convert the first spatial attention features into attention frequency domain features, obtain the first interaction weight associated with the local frequency domain features, and obtain the second interaction weight associated with the attention frequency domain features. The first feature enhancement module is used to enhance the attention frequency domain features according to the first interaction weight to obtain attention-enhanced frequency domain features, enhance the local frequency domain features according to the second interaction weight to obtain local enhanced frequency domain features, convert the local frequency domain features and the local enhanced frequency domain features into second spatial local features, and convert the attention frequency domain features and the attention-enhanced frequency domain features into second spatial attention features. The residual connection module is used to combine the first spatial local feature and the second spatial local feature into a local joint feature, and to combine the first spatial attention feature, the second spatial attention feature, and the local joint feature into an attention joint feature; The first segmentation prediction module is used to obtain the local segmentation prediction result corresponding to the local joint features and the global segmentation prediction result corresponding to the attention joint features, and generate a segmented image corresponding to the source image based on the average value of the local segmentation prediction result and the global segmentation prediction result.

16. An image segmentation processing apparatus, characterized in that, include: The sample image acquisition module is used to acquire labeled sample images and unlabeled sample images; The labeled sample images are used as input to the first network in the initial segmentation model, and the unlabeled sample images are used as input to the first network and the second network in the initial segmentation model. The first network and the second network have the same network structure. The second feature extraction module is used to obtain the first sample local features corresponding to the unlabeled sample image through the initial local feature extractor in the first network, and to obtain the first sample attention features corresponding to the unlabeled sample image through the initial global feature extractor in the first network. The first sample attention feature is used to characterize the information transfer between global information of the unlabeled sample images; The second frequency domain conversion module is used to convert the first sample local features into sample local frequency domain features, convert the first sample attention features into sample attention frequency domain features, obtain the third interaction weight associated with the sample local frequency domain features, and obtain the fourth interaction weight associated with the sample attention frequency domain features through the interaction components in the first network. The second feature enhancement module is used to enhance the sample attention frequency domain features according to the third interaction weight to obtain sample attention enhanced frequency domain features, enhance the sample local frequency domain features according to the fourth interaction weight to obtain sample local enhanced frequency domain features, convert the sample local frequency domain features and the sample local enhanced frequency domain features into second sample local features, and convert the sample attention frequency domain features and the sample attention enhanced frequency domain features into second sample attention features. The second segmentation prediction module is used to combine the first sample local features and the second sample local features into a first sample local joint feature, combine the first sample attention feature, the second sample attention feature and the first sample local joint feature into a first sample attention joint feature, output the first sample local prediction result corresponding to the first sample local joint feature through the initial local segmentation component in the first network, and output the first sample global prediction result corresponding to the first sample attention joint feature through the initial global segmentation component in the first network. The network parameter correction module is used to input the labeled sample image and the unlabeled sample image into the second network, output the local prediction result of the second sample and the global prediction result of the second sample through the second network, and correct the network parameters of the initial segmentation model based on the local prediction result of the first sample, the global prediction result of the first sample, the local prediction result of the second sample, the global prediction result of the second sample, and the annotation information carried by the labeled sample image, and determine the second network containing the corrected network parameters as the image segmentation model; the image segmentation model is used to output the segmented image corresponding to the source image.

17. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to invoke the computer program so that the computer device performs the method according to any one of claims 1 to 9, or performs the method according to any one of claims 10 to 14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1 to 9, or to perform the method of any one of claims 10 to 14.

19. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1 to 9, or implements the method of any one of claims 10 to 14.

Images