Image segmentation method, apparatus, device, medium, and product

Abstract

The application discloses an image segmentation method, device, equipment, medium and product. The method comprises the following steps: acquiring a first image; performing convolution on the first image through at least one size of a convolution kernel to generate a local feature representation of the first image; generating a global feature representation of the first image through a recursive linear attention mechanism; fusing the global feature representation and the local feature representation to obtain a fused feature representation; and predicting the content of the first image based on the fused feature representation to obtain a segmentation prediction result of segmenting the content of the first image in the first image. The local features embodied by each image region are considered, and the correlation between relatively distant pixels in the first image is introduced, that is, the long-range features of the first image are introduced, so that the analysis accuracy of the first image is improved, and the segmentation prediction accuracy of the content of the first image segmented from the first image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image segmentation method, apparatus, device, medium and product. Background Technology

[0002] Image segmentation refers to dividing pixels in a digital image into different regions or objects based on some similarity criterion (such as color, texture, brightness, spatial location, etc.), so that the pixels in different regions have significant differences. The segmented regions can be objects, background, or other meaningful components of the image.

[0003] In related technologies, when performing image segmentation based on deep learning methods, image features are extracted by sliding the convolution kernel of a convolutional neural network across the image.

[0004] However, the receptive field of the convolution kernel is local, which causes the correlation between distant features to be ignored in the image segmentation results, resulting in low accuracy of the image segmentation results. Summary of the Invention

[0005] This application provides an image segmentation method, apparatus, device, medium, and product. The technical solutions provided by this application include the following aspects.

[0006] According to one aspect of the embodiments of this application, an image segmentation method is provided, the method comprising:

[0007] Acquire a first image, which is an image whose content is to be obtained through image segmentation;

[0008] The first image is convolved with a convolution kernel of at least one size to generate a local feature representation of the first image;

[0009] A global feature representation of the first image is generated through a recursive linear attention mechanism, wherein the global feature representation is used to express the global features of the first image in the spatial dimension;

[0010] The global feature representation and the local feature representation are fused to obtain the fused feature representation;

[0011] Based on the fusion feature representation, the content of the first image is predicted, and the segmentation prediction result of the first image content in the first image is obtained.

[0012] According to one aspect of the embodiments of this application, an image segmentation apparatus is provided, the apparatus comprising:

[0013] The acquisition module is used to acquire a first image, which is an image to be segmented to obtain the content of the first image;

[0014] A convolution module is used to convolve the first image with a convolution kernel of at least one size to generate a local feature representation of the first image;

[0015] The generation module is used to generate a global feature representation of the first image through a recursive linear attention mechanism, wherein the global feature representation is used to express the global features of the first image in the spatial dimension.

[0016] The fusion module is used to fuse the global feature representation and the local feature representation to obtain a fused feature representation;

[0017] The prediction module is used to predict the content of the first image based on the fused feature representation, and to obtain the segmentation prediction result of the first image content in the first image.

[0018] In an optional embodiment, the acquisition module is further configured to acquire the image feature representation of the first image, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship;

[0019] The generation module is further configured to generate the global feature representation of the image feature representation through a recursive linear attention mechanism.

[0020] In an optional embodiment, the image feature representation includes at least two channel dimensions;

[0021] The generation module is further configured to split the at least two channel dimensions to obtain a first feature representation containing at least one channel dimension, wherein the first feature representation is a feature representation applied to the global branch; and generate the global feature representation corresponding to the first feature representation through a recursive linear attention mechanism.

[0022] In an optional embodiment, the generation module is further configured to divide the first feature representation into at least two feature blocks of a preset feature size; cross-mix adjacent feature blocks in the at least two feature blocks to obtain a mixed feature representation; and transform the mixed feature representation based on a bidirectional global attention mechanism to obtain the global feature representation.

[0023] In an optional embodiment, the generation module is further configured to obtain an image feature representation of the first image, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship; and to obtain the local feature representation of the first image by convolving the image feature representation with a convolution kernel of at least one size.

[0024] In an optional embodiment, the image feature representation includes at least two channel dimensions;

[0025] The generation module is further configured to split the at least two channel dimensions to obtain a second feature representation containing at least one channel dimension, wherein the second feature representation is a feature representation applied to a local branch;

[0026] The convolution module is further configured to convolve the second feature representation with a convolution kernel of at least one size to obtain the local feature representation of the first image.

[0027] In an optional embodiment, the convolution module is further configured to: split the second feature representation into n sub-feature representations, where n is an integer greater than 1; convolve the i-th sub-feature representation with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; obtain the convolutional feature representations corresponding to the n sub-feature representations respectively; and concatenate the convolutional feature representations corresponding to the n sub-feature representations respectively to obtain the local feature representation of the first image.

[0028] In an optional embodiment, the apparatus includes an image segmentation model, which includes an encoder and a decoder;

[0029] The fusion module is further configured to fuse the global feature representation and the local feature representation of the k-th encoding stage in the k-th encoding stage to obtain the fused feature representation of the k-th encoding stage, where k is a positive integer;

[0030] The prediction module is further configured to, in the q-th decoding stage of the decoder, fuse the fusion feature representation of the (m+1-q)-th encoding stage and the decoding feature representation of the (q-1)-th decoding stage to obtain the encoding-decoding fusion feature representation of the q-th decoding stage, and decode the encoding-decoding fusion feature representation of the q-th decoding stage to obtain the decoding feature representation of the q-th decoding stage, where m is the number of decoding stages and q is a positive integer; predict the first image content based on the decoding feature representation of the m-th decoding stages to obtain the segmentation prediction result of segmenting the first image content in the first image.

[0031] In an optional embodiment, the prediction module is further configured to fuse the fusion feature representations of the at least two encoding stages based on the channel mixing mode to obtain an encoding fusion feature representation; split the encoding fusion feature representation into at least two encoding feature representations, wherein the (m+1-q)th encoding stage corresponds to the (m+1-q)th encoding feature representation; and fuse the (m+1-q)th encoding feature representation and the decoding feature representation of the (q-1)th decoding stage to obtain the encoding-decoding fusion feature representation of the qth decoding stage.

[0032] In an optional embodiment, the acquisition module is further configured to acquire an image feature representation of the first image, the image feature representation including at least two channel dimensions;

[0033] The generation module is also used to split the at least two channel dimensions to obtain an identity feature representation containing at least one channel dimension;

[0034] The fusion module is further configured to fuse the global feature representation, the local feature representation, and the identity feature representation to obtain the fused feature representation.

[0035] In an optional embodiment, the apparatus includes an image segmentation model;

[0036] The acquisition module is further configured to acquire a sample image, wherein the sample image is marked with a reference content area corresponding to the content of the first image;

[0037] The prediction module is further configured to predict the first image content in the sample image using the image segmentation model, thereby obtaining the predicted content region;

[0038] The device further includes:

[0039] The training module is used to train the image segmentation model based on the difference between the reference content region and the predicted content region.

[0040] In an optional embodiment, the prediction module is further configured to predict the first image content in the sample image using the image segmentation model, to obtain the predicted probability that a pixel in the sample image belongs to the first image content; and to determine the pixel in the sample image that belongs to the first image content based on the predicted probability, thereby obtaining the predicted content region.

[0041] In an optional embodiment, the training module is further configured to obtain a first loss value based on the pixel difference between the reference content region and the predicted content region; and train the image segmentation model based on the first loss value.

[0042] In an optional embodiment, the training module is further configured to: obtain the number of intersection pixels between the reference content region and the predicted content region; obtain the number of union pixels between the reference content region and the predicted content region; obtain a second loss value based on the number of intersection pixels and the number of union pixels; and train the image segmentation model based on the second loss value.

[0043] In an optional embodiment, the training module is further configured to: acquire a first edge pixel of the reference content region; acquire a second edge pixel of the predicted content region; acquire a third loss value based on the distance between the first edge pixel and the second edge pixel; and train the image segmentation model based on the third loss value.

[0044] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described image segmentation method.

[0045] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described image segmentation method.

[0046] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described image segmentation method.

[0047] The technical solutions provided in this application can bring at least the following beneficial effects:

[0048] By generating a global feature representation of the first image and obtaining its local feature representation through a recursive linear attention mechanism, the system combines the global features of the first image with the local features of each image region to comprehensively analyze the content of the first image. This allows for the segmentation of the first image content and yields a segmentation prediction result. This approach avoids the problem of a one-sided analysis of the first image, which is often insufficient in analyzing long-range features, by considering both the local features of each image region and the correlation between relatively distant pixels in the first image, thus incorporating long-range features. This improves the accuracy of the analysis and the segmentation prediction accuracy of the first image content. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0050] Figure 2 This is a flowchart of an image segmentation method provided in an exemplary embodiment of this application;

[0051] Figure 3This is a schematic diagram illustrating the process of obtaining global feature representation, local feature representation, and fused feature representation provided in an exemplary embodiment of this application;

[0052] Figure 4 This is a flowchart of an image segmentation method provided in another exemplary embodiment of this application;

[0053] Figure 5 This is a schematic diagram of an encoding / decoding process provided in an exemplary embodiment of this application;

[0054] Figure 6 This is a flowchart of an image segmentation method provided in another exemplary embodiment of this application;

[0055] Figure 7 This is a schematic diagram of a lesion area identification process provided in an exemplary embodiment of this application;

[0056] Figure 8 This is a block diagram of an image segmentation apparatus provided in an exemplary embodiment of this application;

[0057] Figure 9 This is a block diagram of an image segmentation apparatus provided in another exemplary embodiment of this application;

[0058] Figure 10 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0060] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0061] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multi-dimensional data. Computer vision technologies typically include image processing, image segmentation, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0062] Image segmentation is an important task in the field of computer vision. Its goal is to divide an image into several regions with specific meanings or attributes to facilitate further analysis and processing. In applications, image segmentation can be applied to a wide range of image domains, such as: 1. In medical image processing, it can be used to identify lesion areas from images, helping doctors make more accurate diagnoses; 2. In satellite remote sensing image processing, it can more accurately identify various information about the Earth's surface, such as forest cover areas, from remote sensing images; 3. In autonomous driving, it can be used to identify road elements such as obstacles in road images, thereby improving the accuracy of target recognition and ensuring the safety of autonomous driving; 4. In agriculture, it can be used to identify pest and disease areas from agricultural images, thereby addressing pest and disease problems in a timely manner and improving agricultural cultivation efficiency; 5. In industrial quality inspection, it can be applied to the detection of defective parts, identifying defective areas in part images, improving the efficiency and accuracy of industrial quality inspection. It is worth noting that the above application areas are merely illustrative examples, and the embodiments of this application do not limit the scope of the application.

[0063] A channel refers to data in an image or feature map related to a specific color or feature. For images, channels are usually associated with color components. For example, a grayscale image has only one channel, representing the image's brightness information, while a color (RGB) image typically has three channels, corresponding to the red, green, and blue color components. Besides color information, channels can also represent other image features, such as depth information (in depth images, channels represent the depth values ​​of objects) and transparency (in some image formats (such as PNG), in addition to the three RGB channels, there is an alpha channel representing the image's transparency).

[0064] A feature map is the output of a convolutional layer after performing a convolution operation on an input image. It captures local features of the input image at different locations and orientations. For a feature map, channels represent information in different feature directions. Each channel can be viewed as a two-dimensional "feature plane," capturing features of the input image in a specific aspect. When extracting features from an input image, different convolutional kernels extract different features, thus generating different channels. Each channel corresponds to different features, such as edges, textures, and shapes.

[0065] Please refer to Figure 1 This illustrates a schematic diagram of a computer system provided in one embodiment of this application.

[0066] The computer system 100 includes: a terminal 120 and a server 140.

[0067] Terminal 120 has an application with image segmentation capabilities installed and running. The device type of terminal 120 includes at least one of the following: smartphone, laptop, desktop computer, tablet, smart robot, AR (Augmented Reality) device, VR (Virtual Reality) device, etc.

[0068] Terminal 120 is connected to server 140 via a wireless network or a wired network.

[0069] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0070] Server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 140 provides functional services for implementing model training. Optionally, server 140 undertakes the main computing work, and terminal 120 undertakes the secondary computing work; or, server 140 undertakes the secondary computing work, and terminal 120 undertakes the main computing work; or, server 140 and terminal 120 collaborate on computing using a distributed computing architecture.

[0071] It is worth noting that the server 140 described above can be implemented as a physical server or as a cloud server in the cloud. In some embodiments, the server 140 can also be implemented as a node in a blockchain system.

[0072] Optionally, the image segmentation method provided in this application embodiment can be executed independently by terminal 120 or server 140, or it can be implemented by terminal 120 and server 140 in cooperation, which is not limited here.

[0073] In one example, the image segmentation method is implemented independently by terminal 120. For instance, terminal 120 is equipped with an Artificial Intelligence (AI) chip, which deploys an image segmentation model. When a user requests image segmentation, the application in terminal 120 obtains the first image indicated by the user and inputs it into the image segmentation model deployed in the AI ​​chip. The image segmentation model, using the image segmentation method provided in this embodiment, obtains an image segmentation result that segments the content of the first image from the first image. Terminal 120 displays the image segmentation result to show the user the content of the first image in the first image. The segmentation of the first image content is merely an illustrative example; the image segmentation method provided in this embodiment can segment at least one image content from the first image, such as the first image content itself, background content, etc.

[0074] In another example, the image segmentation method is implemented collaboratively by terminal 120 and server 140. Illustratively, server 140 deploys an image segmentation model to provide backend services for the application running on terminal 120. The user uploads a first image to server 140 through the application on terminal 120. After obtaining the first image, server 140 inputs it into the image segmentation model. The image segmentation model uses the image segmentation method provided in this embodiment to obtain an image segmentation result that segments the content of the first image from the first image. Server 140 sends the image segmentation result to terminal 120, and terminal 120 displays the content of the first image in the first image to the user.

[0075] Taking image segmentation performed by server 140 as an example, after acquiring the first image, server 140 inputs the first image into the image segmentation model. During the encoding stage, the image segmentation model acquires the global and local feature representations of the first image, and fuses them to obtain a fused feature representation. This fused feature representation is then used to predict the image segmentation result corresponding to the content of the first image. Specifically, the global feature representation is obtained by analyzing the first image through a recursive linear attention mechanism, while the local feature representation is obtained by convolving the first image with a convolutional kernel of at least one size.

[0076] Please refer to Figure 2 The diagram illustrates a flowchart of an image segmentation method provided in one embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method, executed by the terminal 120 or server 140 shown, may include steps 210 to 250.

[0077] Step 210: Obtain the first image, which is the image whose content is to be obtained through image segmentation.

[0078] The first image content may include one or more elements. For example, when the first image content includes one element, that is, the first image is the image to be segmented to obtain the first image content. For instance, if the first image content includes a person, then the first image is to be segmented to obtain the image of the region containing the person. When the first image content includes multiple elements, that is, the first image is to be segmented to obtain images of multiple first image contents. For instance, if the first image content includes a person and a pet, then the first image is to be segmented to obtain a first region containing the person and a second region containing the pet.

[0079] The first image is the image for which image segmentation is required. Optionally, the first image can be a camera image, a drawn image, a computer-generated image, a satellite image, a microscope image, an image acquired by a sensor (e.g., thermal imaging, X-ray image, ultrasound image, etc.), a radar image, etc., which are not limited here.

[0080] The segmentation requirements for the first image vary depending on the specific scenario.

[0081] To illustrate, consider the following examples: In a medical diagnosis scenario, the first image is a medically acquired image, such as an X-ray image. The segmentation requirement for this first image includes segmenting the lesion area. In a geological analysis scenario, the first image is a satellite remote sensing image. The segmentation requirement includes segmenting information about the Earth's surface, such as forest cover areas. In an intelligent driving scenario, the first image is a road image acquired by a vehicle-mounted camera. The segmentation requirement includes segmenting road elements, such as obstacles. In an agricultural analysis scenario, the first image is an agricultural image acquired by a drone or other image acquisition device. The segmentation requirement includes segmenting areas affected by pests and diseases. In an industrial quality inspection scenario, the first image is a part image captured by photographing a component. The segmentation requirement includes segmenting defective areas from the component.

[0082] It is worth noting that the type of the first image and the image segmentation requirements mentioned above are merely illustrative examples. The image segmentation method provided in this application embodiment can be applied to any scenario that uses image segmentation technology, and no specific application scenario is limited.

[0083] Step 220: Generate a global feature representation of the first image through a recursive linear attention mechanism.

[0084] The global feature representation is used to express the global features of the first image in the spatial dimension. In some embodiments, when the first image is implemented as an image frame in a video, the global feature representation can also be used to express the global features of the first image in the spatiotemporal dimension.

[0085] In some embodiments, a global feature representation of the first image is generated using a Visual RWKV (Vision Receptance Weighted Key Value) network. RWKV is a deep learning architecture that combines a Recurrent Neural Network (RNN) and a Transformer model. VRWKV is a network built upon RWKV applied to visual tasks, capable of capturing long-range dependent features in visual data.

[0086] Linear attention is an improved attention computation method that reduces the computational complexity of attention, enabling efficient processing of long sequences of data. It replaces quadratic calculations (such as dot products) in traditional attention mechanisms with linear operations, thus reducing time complexity. VRWKV achieves efficient modeling of long sequences of information through linear attention, while further enhancing model performance by introducing spatial mixing and channel mixing modules.

[0087] In some embodiments, firstly, an image feature representation of the first image is obtained. This image feature representation is obtained by transforming the color channels of the first image according to a preset channel transformation relationship. Then, a global feature representation corresponding to the image feature representation is generated using a recursive linear attention mechanism.

[0088] In some embodiments, image feature representations are used to indicate content features of the first image, that is, image feature representations are used to indicate content features extracted from the first image. Here, content features refer to features in the image that can be perceived and recognized by a human visual system or a computer vision system. Optionally, content features include features directly related to the pixel values ​​of the first image, such as color, texture, and edge features; or, content features include semantic features describing shapes, local structures, or patterns in the first image; or, content features include semantic features describing object categories, scene categories, or events in the first image.

[0089] Optionally, the image feature representation is extracted through a pre-trained first processing network. Optionally, the pre-trained first processing network can be implemented as a convolutional neural network (CNN), a feedforward neural network (FNN), a residual network (ResNet), a transformer, etc.

[0090] Optionally, when generating a global feature representation based on the image feature representation, at least one of the following methods is included:

[0091] First, the image feature representation is divided into at least two feature blocks of a preset feature size. Adjacent feature blocks within these two feature blocks are cross-blended to obtain a hybrid feature representation. This hybrid feature representation is then transformed using a bidirectional global attention mechanism to obtain a global feature representation. Specifically, spatial blending operations in VRWKV are used to cross-blend adjacent feature blocks within the at least two feature blocks. Illustratively, the image feature representation is divided into at least two 1×1 feature blocks, concatenated to obtain a one-dimensional feature sequence. Then, spatial blending operations in VRWKV are applied to obtain the hybrid feature representation. Finally, the hybrid feature representation expressed by the feature sequence is converted into a two-dimensional feature map to obtain the global feature representation.

[0092] By directly performing spatial mixing operations on the feature sequence based on the image feature representation and then converting it back to a two-dimensional feature map, the efficiency of obtaining global feature representation is improved.

[0093] Second, the image feature representation includes at least two channel dimensions. The at least two channel dimensions are split to obtain a first feature representation containing at least one channel dimension. This first feature representation is applied to the global branch. A global feature representation corresponding to the first feature representation is generated through a recursive linear attention mechanism. Specifically, the first feature representation is divided into at least two feature blocks of a preset feature size. Adjacent feature blocks within these at least two feature blocks are cross-mixed to obtain a hybrid feature representation. This hybrid feature representation is then transformed based on a bidirectional global attention mechanism to obtain the global feature representation.

[0094] Using the channel dimension as the dividing criterion, a first feature representation applied to the global branch is extracted from the image feature representation. Based on the first feature representation, spatial blending of features is performed through spatial blending operations in VRWKV, which distinguishes the global branch, local branch, and identity branch from the channel dimension, thereby improving the expression accuracy of the global branch.

[0095] Specifically, the spatial blending operation in VRWKV is used to cross-blend adjacent feature blocks in at least two feature blocks. Illustratively, the image feature representation is divided into at least two 1×1 feature blocks, concatenated to obtain a one-dimensional feature sequence. Then, the spatial blending operation in VRWKV is applied to obtain a blended feature representation. Finally, the blended feature representation expressed by the feature sequence is converted into a two-dimensional feature map to obtain the global feature representation.

[0096] By dividing the feature into at least two feature blocks and concatenating them to obtain a one-dimensional feature sequence, and performing a spatial mixing operation on the basis of the one-dimensional feature sequence, information exchange and perception can be achieved between pixels that are far apart, thereby improving the global information representation capability of the global feature representation.

[0097] In the VRWKV model, spatial mixing refers to the process of feature aggregation and information integration of local regions of the input image through specific network design. Its core objective is to enhance the model's ability to model the spatial structure of the image by expanding the local receptive field, while maintaining computational efficiency.

[0098] In each coding layer of VRWKV, the spatial blending operation includes the following steps: 1. Multi-directional shifting of the tokens in the first feature representation of the input, such as up, down, left, and right, and linear interpolation and combination of features of locally adjacent tokens. 2. Inputting the shifted tokens into a bidirectional attention mechanism to calculate the global attention result. After feature fusion, normalization is performed through residual connections and layer normalization.

[0099] Step 230: Convolve the first image with a convolution kernel of at least one size to generate a local feature representation of the first image.

[0100] In some embodiments, the first image is convolved with convolution kernels of at least two sizes to obtain convolution feature representations corresponding to the convolution kernels of at least two sizes respectively, and the convolution feature representations corresponding to the convolution kernels of at least two sizes are concatenated to obtain local feature representations.

[0101] In a schematic way, the first image is convolved with four convolution kernels of different sizes to obtain convolution feature representations corresponding to the four kernels. The four convolution feature representations are then concatenated to obtain local feature representations.

[0102] For example, by convolving the first image with 3×3, 5×5, 7×7, and 9×9 convolution kernels respectively, we can obtain the convolution feature representations corresponding to the four kernels respectively. By concatenating the four convolution feature representations, we can obtain the local feature representation.

[0103] It is worth noting that the above-mentioned kernel size is only an illustrative example, and the embodiments of this application do not limit the choice of kernel size.

[0104] Optionally, an image feature representation of the first image is obtained, which is a feature representation obtained by transforming the color channels of the first image according to a preset channel transformation relationship. The image feature representation is convolved with a convolution kernel of at least one size to obtain a local feature representation of the first image. In some embodiments, the image feature representation includes at least two channel dimensions. The at least two channel dimensions are split to obtain a second feature representation containing at least one channel dimension. The second feature representation is a feature representation applied to local branches. The second feature representation is convolved with a convolution kernel of at least one size to obtain a local feature representation of the first image.

[0105] Using the channel dimension as the dividing criterion, a second feature representation applied to local branches is extracted from the image feature representation. Based on the second feature representation, a local feature representation is obtained by convolving the second feature representation with a convolution kernel of at least one size. This distinguishes the global branch, local branch, and identity branch from the channel dimension, improving the accuracy of local branch representation.

[0106] In some embodiments, the second feature representation is split into n sub-feature representations, where n is an integer greater than 1. The i-th sub-feature representation is convolved with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n. The convolutional feature representations corresponding to the n sub-feature representations are obtained respectively, and the convolutional feature representations corresponding to the n sub-feature representations are concatenated to obtain the local feature representation of the first image.

[0107] By splitting the feature representation into n sub-representations, and then using convolution kernels of different sizes for each sub-representation, the convolution process of different sizes is distinguished, thereby improving the accuracy of local information representation of local features.

[0108] Step 240: Fuse the global feature representation and the local feature representation to obtain the fused feature representation.

[0109] In some embodiments, the method of fusing global feature representation and local feature representation includes at least one of the following:

[0110] 1. Directly concatenate the global feature representation and the local feature representation to obtain the fused feature representation.

[0111] If the global feature representation and the local feature representation have the same dimension, then the global feature representation and the local feature representation are concatenated according to the matrix rows or matrix columns to obtain the fused feature representation.

[0112] 2. Add the matrix elements of the global feature representation and the local feature representation to obtain the fused feature representation.

[0113] In this model, the global feature representation and the local feature representation have the same dimension. The elements in the global feature representation and the local feature representation are added according to their matrix positions or a weighted sum is calculated to obtain the fused feature representation.

[0114] 3. Multiply the matrix elements of the global feature representation and the local feature representation to obtain the fused feature representation.

[0115] The global feature representation and the local feature representation have the same dimension. The elements in the global feature representation and the local feature representation are multiplied according to their matrix positions to obtain the fused feature representation.

[0116] In some embodiments, an image feature representation of a first image is obtained, the image feature representation including at least two channel dimensions, the at least two channel dimensions are split to obtain an identity feature representation including at least one channel dimension, and the global feature representation, local feature representation and identity feature representation are fused to obtain a fused feature representation.

[0117] When splitting the image feature representation, a first feature representation, a second feature representation, and an identity feature representation are split according to the channel dimension. The first feature representation is used for global branch analysis, the second feature representation is used for local feature analysis, and the identity feature representation is used to express the initial feature representation. Global and local analysis are performed while preserving the original information of the image, which improves the accuracy of predicting the content of the first image by fusing feature representation.

[0118] Indicative, Figure 3 This is a schematic diagram illustrating the process of obtaining global feature representation, local feature representation, and fused feature representation according to an exemplary embodiment of this application. For example... Figure 3 As shown, first obtain the first image X∈R Cin×H×W Image feature representation 310, image feature representation 310 is expressed as X'∈R Cmid×H×W , where C in H represents the number of channels in the input feature representation of the first image, W represents the height of the first image, and C represents the width of the first image. mid This indicates the number of channels in the image feature representation (310). Optionally, C can be converted using a preset channel transformation relationship. in Convert to C mid In some embodiments, X is transformed using a pre-trained transformation network to obtain X'. Optionally, C mid >C in For example, C mid =2C in Or, C mid =3C in.

[0119] For C mid Split to get C L C G and C I , where C L The number of channels corresponding to local branches, C G Implemented as the number of channels corresponding to the global branch, C I The implementation is the number of channels corresponding to the identity branch, and C L +C G +C I =C mid After decomposition, we obtain a first feature representation 321, a second feature representation 322, and an identity feature representation 323, where the first feature representation 321 is expressed as I. global ∈R CG×H×W The second feature, represented as 322, is expressed as I. local ∈R CL×H×W The identity feature is represented as I. identity ∈R CI×H×W .

[0120] In some embodiments, for C mid The splitting method is preset, such as: splitting C mid The first two dimensions are split into the first feature representation, and C is... mid The middle two dimensions are split into the second feature representation, and C is... mid The last two dimensions are split into identity feature representations. Among them, the channel dimensions in the first feature representation, the second feature representation, and the identity feature representation do not overlap.

[0121] A global analysis of the first feature representation 321 yields a global feature representation 331. In the global branch, the first feature representation 321 is divided into 1×1 feature blocks, resulting in a one-dimensional feature sequence. Then, the spatial blending operation in VRWKV is applied to this one-dimensional feature sequence, finally transforming it into a two-dimensional feature map. Illustratively, I is first... global Divide the feature into 1×1 feature blocks to obtain the feature sequence, i.e., I. unfolding =Unfolding(I global Unfolding() refers to the operation of dividing feature blocks. unfolding This is a characteristic sequence. I unfolding ∈R CG×N N represents the number of feature blocks.

[0122] Applying spatial blending operations in VRWKV to I unfolding Execution space mixing, that is, I mix=SpatialMix(LayerNorm(I unfolding ))+I unfolding LayerNorm() refers to the layer normalization operation, SpatialMix() refers to the spatial mixing operation, and I mix This refers to hybrid feature representation, I mix ∈R CG×N .

[0123] Transform the hybrid feature representation back into a two-dimensional feature map, i.e., I2 = Folding(I mix I2 represents the global feature representation, and Folding() refers to the operation of converting it into a two-dimensional feature map, where I2 ∈ R. CG×H×W .

[0124] Local analysis is performed on the second feature representation 322 to obtain local feature representation 332. For each local branch, convolution operations are performed on the second feature representation 322 using kernels of different sizes, and then the results of the different convolution operations are concatenated.

[0125] In some embodiments, the second feature representation 322 is split into four sub-feature representations, that is, in,

[0126] Convolution operations are performed on the four sub-feature representations, and the convolution results are concatenated to obtain the local feature representations. That is, I1 represents the local feature representation, Concat() refers to the concatenation operation, Conv() refers to the convolution operation, and I1∈R. CL×H×W 3×3, 5×5, 7×7, and 9×9 refer to the size of the convolution kernel.

[0127] By concatenating the global feature representation 331, the local feature representation 332, and the identity feature representation 323, a fused feature representation 340 is obtained. That is, I combined =Concat(I2, I1, I identity ).

[0128] In some embodiments, after obtaining the fused feature representation, local feature aggregation is performed on the fused feature representation through a depthwise separable convolution operation, and the aggregated feature representation is projected onto a preset output dimension.

[0129] Illustratively, a 5×5 depthwise separable convolution (DWConv) is used to perform local feature aggregation on the fused feature representation. That is, I aggregation =DWConv(I combined )+I combined .

[0130] Among them, Iaggregation ∈R Cmid×H’×W’ H′×W′ is determined by the stride of the depthwise separable convolution (DWConv). Finally, I aggregation Projecting onto a predefined output dimension yields the encoded feature representation, i.e., F = Conv 1×1 (I aggregation ).

[0131] Step 250: Predict the content of the first image based on the fused feature representation to obtain the segmentation prediction result of the first image content in the first image.

[0132] In some embodiments, the fused feature representation is decoded to predict the content of the first image, thereby obtaining a segmentation prediction result for the first image content in the first image.

[0133] In summary, the image segmentation method provided in this application generates a global feature representation of the first image and obtains a local feature representation of the first image through a recursive linear attention mechanism. This combines the global features of the first image with the local features of each image region to comprehensively analyze the content of the first image, thereby segmenting the content of the first image from the first image and obtaining a segmentation prediction result. This avoids the problem of a one-sided analysis of the first image and insufficient analysis of long-range features caused by only analyzing each image region. It considers both the local features of each image region and the correlation between relatively distant pixels in the first image, i.e., it introduces the long-range features of the first image, improving the accuracy of the analysis of the first image and the segmentation prediction accuracy of the content of the first image from the first image.

[0134] In some embodiments, the above image segmentation method is performed by an image segmentation model, which includes an encoder and a decoder.

[0135] The process of generating global and local feature representations is performed by the encoder; the process of predicting the content of the first image based on the fused feature representation is performed by the decoder.

[0136] In some embodiments, the image segmentation model includes m encoders and m decoders, where m is a positive integer. Each encoder corresponds to an encoding stage, and each decoder corresponds to a decoding stage.

[0137] Please refer to Figure 4 This illustrates a flowchart of an image segmentation method provided in another embodiment of this application. The method is implemented using a computer device (which can be configured as follows). Figure 1 The method, executed by the terminal 120 or server 140 shown, may include steps 210 to 460.

[0138] Step 210: Obtain the first image, which is the image whose content is to be obtained through image segmentation.

[0139] The first image is the image for which image segmentation is required. Optionally, the first image can be a camera image, a drawn image, a computer-generated image, a satellite image, a microscope image, an image acquired by a sensor (e.g., thermal imaging, X-ray image, ultrasound image, etc.), a radar image, etc., which are not limited here.

[0140] Step 420: In the k-th encoding stage, a global feature representation of the first image in the k-th encoding stage is generated through a recursive linear attention mechanism.

[0141] In some embodiments, during the first encoding stage, an image feature representation of the first image is first obtained. This image feature representation is obtained by transforming the color channels of the first image according to a preset channel transformation relationship. A global feature representation corresponding to the image feature representation is then generated using a recursive linear attention mechanism.

[0142] In some embodiments, in the initial stage, preliminary features are extracted to provide feature maps with preliminary convolution for subsequent stages. The encoder includes at least two encoding stages, i.e., m≥2. Schematic, an encoder with four encoding stages is used as an example. The first network layer of each encoding stage performs a convolution operation using a depthwise separable convolution (DWConv) with a stride of 2 to downsample the feature map. Subsequently, a stacked multi-scale perceptron (MSP) is used for feature transformation, and spatial blending operations from VRWKV are introduced.

[0143] For the input image X∈R 3×H0×W0 The initial features obtained from the encoder's output at the start stage are represented as F0∈R C0 ×H0×W0 .

[0144] In the first encoding stage, given a batch of input feature maps, X∈R is obtained from the above initial feature representation. Cin ×H×W First, the initial feature representation is normalized and projected onto the intermediate dimension Cmid using a 1×1 convolutional layer, and then layer normalization is performed to obtain the image feature representation, i.e., I = LayerNorm(Conv 1×1 (X)).

[0145] In some embodiments, in the second encoding stage, the feature representation output from the first encoding stage is used as the input feature representation for the second encoding stage, and the feature representations of the global branch, local branch, and identity branch are split again. Global analysis is performed on the feature representation of the global branch, and local analysis is performed on the feature representation of the local branch.

[0146] Schematic, the output features of the first encoding stage are represented as F1∈R C1×H1×W1 The channel dimension C1 is split to obtain the global branch feature representation, local branch feature representation, and identity feature representation of the second encoding stage. The global branch feature representation is then analyzed globally by dividing it into 1×1 blocks. The spatial blending operation in VRWKV is then applied, and finally the feature sequence is converted back to a two-dimensional feature map to obtain the global feature representation of the second encoding stage.

[0147] In the k-th encoding stage, the channel dimension of the output feature representation of the (k-1)-th encoding stage is split to obtain the global branch feature representation, local branch feature representation, and identity feature representation of the k-th encoding stage. The global branch feature representation is then analyzed globally by dividing it into 1×1 blocks. The spatial blending operation in VRWKV is then applied, and finally the feature sequence is converted back to a two-dimensional feature map to obtain the global feature representation of the k-th encoding stage.

[0148] Step 430: In the k-th encoding stage, the first image is convolved with a convolution kernel of at least one size to generate a local feature representation of the first image.

[0149] In some embodiments, in the first encoding stage, an image feature representation of a first image is first obtained. The image feature representation is a feature representation obtained by transforming the color channels of the first image according to a preset channel transformation relationship. The image feature representation is then convolved with a convolution kernel of at least one size to generate a local feature representation.

[0150] In some embodiments, in the second encoding stage, the feature representation output from the first encoding stage is used as the input feature representation for the second encoding stage, and the feature representations of the global branch, local branch, and identity branch are split again, and local analysis is performed on the feature representations of the local branch.

[0151] Schematic, the output features of the first encoding stage are represented as F1∈R C1×H1×W1 The channel dimension C1 is split to obtain the local branch feature representation of the second encoding stage. The local branch feature representation is then analyzed locally, that is, the local branch feature representation is convolved by a convolution kernel of at least one size to obtain the local feature representation of the second encoding stage.

[0152] In the k-th encoding stage, the channel dimension of the output feature representation of the (k-1)-th encoding stage is split to obtain the global branch feature representation, local branch feature representation, and identity feature representation of the k-th encoding stage. Local analysis is then performed on the local branch feature representation, that is, the local branch feature representation is convolved by a convolution kernel of at least one size to obtain the local feature representation of the k-th encoding stage.

[0153] Step 440: In the k-th encoding stage of the encoder, the global feature representation and local feature representation of the k-th encoding stage are fused to obtain the fused feature representation of the k-th encoding stage, where k is a positive integer.

[0154] In some embodiments, after obtaining the fused feature representation of the k-th encoding stage, local feature aggregation is performed on the fused feature representation through a depthwise separable convolution operation, and the aggregated feature representation is projected onto a preset output dimension to obtain the encoded feature representation of the k-th encoding stage.

[0155] In some embodiments, after obtaining the fusion feature representation of the k-th coding stage, it is determined whether k is equal to m, where m is the number of coding stages. That is, it is determined whether the k-th coding stage is the last coding stage. If so, step 450 is executed; otherwise, k is set to k+1, and step 440 is executed repeatedly.

[0156] Step 450: In the q-th decoding stage of the decoder, the fused feature representation of the (m+1-q)-th encoding stage and the decoding feature representation of the (q-1)-th decoding stage are fused to obtain the encoding-decoding fused feature representation of the q-th decoding stage. Then, the encoding-decoding fused feature representation of the q-th decoding stage is decoded to obtain the decoding feature representation of the q-th decoding stage.

[0157] m is the number of decoding stages, and q is a positive integer. In some embodiments, 1 < q ≤ m, that is, in the first decoding stage, there is no need to fuse the encoded feature representation and the decoded feature representation to obtain the encoded-decoded fused feature representation. In the first decoding stage, the fused feature representation output by the last encoding stage is decoded to obtain the decoded feature representation of the first decoding stage.

[0158] Optionally, in the q-th decoding stage of the decoder, the encoded feature representations of the (m+1-q)-th encoding stage and the decoded feature representations of the (q-1)-th decoding stage are fused to obtain the fused encoding / decoding feature representation of the q-th decoding stage. Optionally, the fused feature representations of at least two encoding stages are fused based on a channel mixing mode to obtain a fused encoding feature representation; the fused encoding feature representation is then split into at least two encoded feature representations, where the (m+1-q)-th encoding stage corresponds to the (m+1-q)-th encoded feature representation; the (m+1-q)-th encoded feature representation and the decoded feature representation of the (q-1)-th decoding stage are fused to obtain the fused encoding / decoding feature representation of the q-th decoding stage. Here, the channel mixing mode refers to the channel mixing mode of the fused feature representations of at least two encoding stages through GCM.

[0159] In some embodiments, the decoding stage that requires the fusion of encoded feature representations and decoded feature representations is pre-defined.

[0160] To illustrate, taking m as a value of 4 as an example, there are a total of 4 encoding stages and 4 decoding stages. In the first decoding stage, the encoded feature representation output from the fourth encoding stage is decoded to obtain the decoded feature representation of the first decoding stage. In the second decoding stage, the decoded feature representation of the first decoding stage is fused with the encoded feature representation of the third encoding stage to obtain the fused encoding / decoding feature representation of the second decoding stage. This fused encoding / decoding feature representation of the second decoding stage is then decoded to obtain the decoded feature representation of the second decoding stage. In the third decoding stage, the decoded feature representation of the second decoding stage is fused with the encoded feature representation of the second encoding stage to obtain the fused encoding / decoding feature representation of the third decoding stage. This fused encoding / decoding feature representation of the third decoding stage is then decoded to obtain the decoded feature representation of the third decoding stage. In the fourth decoding stage, the decoded feature representation of the third decoding stage is fused with the encoded feature representation of the first encoding stage to obtain the fused encoding / decoding feature representation of the fourth decoding stage. This fused encoding / decoding feature representation of the fourth decoding stage is then decoded to obtain the decoded feature representation of the fourth decoding stage.

[0161] The decoder takes the feature map fused from the last layer of the encoder and the GCM module as input, and generates the final segmentation result through progressive upsampling. Each decoder module consists of a 1×1 convolutional layer, a depthwise separable convolutional layer DWConv, an output 1×1 convolutional layer, and a bilinear interpolation upsampling operation. The decoding process is as follows: Assuming the input is feature map F... in ∈R Cin×H×W After the first 1×1 convolutional layer, the feature distribution is adjusted while maintaining the same number of input and output channels: F de1 =Conv 1×1 (F in)∈R Cin×H×W Spatial feature extraction was performed using depthwise separable convolutional layers (DWConv), yielding the following features: F de2 =DWConv(F de1 )∈R Cin×H×W After another 1×1 convolutional layer, the feature map is mapped to the target output channel of the decoder, F. de3 =Conv 1×1 (F de2 )∈R Cout×H×W .

[0162] After upsampling, the size of the feature map is increased to twice its original size, F out =Upsample(F de3 )∈R Cout×2H×2W .

[0163] The decoder consists of four stages, with each stage outputting F. out1 F out2 F out3 and F out4 .

[0164] The encoder's fourth-stage output is F4∈R C4×H4×W4 The encoded feature representations of the first encoding stage are denoted as F1, the second encoding stage as F2, and the third encoding stage as F3. The decoding process, which fuses the encoded and decoded feature representations using Global Channel Mix (GCM), is as follows:

[0165] F out1 =Decoder1(F4)∈R C3×H3×W3 ;

[0166] F out2 =Decoder2(Concat(F out1 ,F3))∈R C2×H2×W2 ;

[0167] F out3 =Decoder3(Concat(F out2 ,F2))∈R C1×H1×W1 ;

[0168] F out4 =Decoder4(Concat(F out3 ,F1))∈R C0×H0×W0 ;

[0169] Decoder1, Decoder2, Decoder3, and Decoder4 are the decoder modules for the four decoding stages. H0×W0 is the original image size, optionally H0=2H1=4H2=8H3=16H4, W0=2W1=4W2=8W3=16W4.

[0170] In some embodiments, when fusing coded and decoded feature representations based on GCM, the GCM design includes the following cases. Illustratively, using four encoding stages and four decoding stages as an example, the output feature map of the first encoding stage in the encoder, i.e., the coded feature representation of the first encoding stage, is defined as F1; the output feature map of the second encoding stage in the encoder, i.e., the coded feature representation of the second encoding stage, is defined as F2; ​​and the output feature map of the third encoding stage in the encoder, i.e., the coded feature representation of the third encoding stage, is defined as F3. F1, F2, and F3 have different spatial dimensions and channel numbers. F1∈R C1×H1×W1 F2∈R C2×H2×W2 F3∈R C3×H3×W3 Where H1 > H2 > H3, W1 > W2 > W3, and C3 > C2 > C1. To unify features for multi-scale fusion, this embodiment applies global average pooling to reduce each feature map to a single spatial unit, i.e. in, The pooled features are concatenated along the channel dimension to form a unified feature representation, that is... Among them, F cat ∈R (C1+C2+C3)×H1×W1 The above features are represented as F. cat The sequence representation is reshaped for channel mixing, i.e., F seq =Unfolding(F cat )∈R 1×(C1+C2+C3) .

[0171] Optionally, a channel hybrid operation with layer normalization and DropPath is applied, i.e., F mix =ChannelMix(LayerNorm(F seq ))+F seq ∈R 1×(C1+C2+C3) The features after channel blending are reshaped back into the spatial format, i.e., F. fold =Folding(F mix )∈R (C1+C2+C3)×1×1 Then F fold Decompose back into the original feature components, that is Each split feature is reshaped and expanded to the original space size, that is... Features after recovery Subsequently, the encoded feature representation is concatenated with the decoded feature representation from the third decoding stage of the decoder in the fourth decoding stage. Subsequently, the encoded feature representation is concatenated with the decoded feature representation from the second decoding stage of the decoder in the third decoding stage, and... Subsequently, the encoded feature representation is concatenated with the decoded feature representation of the decoder in the first decoding stage during the second decoding stage.

[0172] Figure 5 This is a schematic diagram of the encoding and decoding process provided in an exemplary embodiment of this application, as shown below. Figure 5 As shown, the encoding and decoding process includes encoders 510 and decoders 520, including four encoders 510 and four decoders 520.

[0173] After acquiring the first image 500, the image feature representation of the first image 500 is extracted. The image feature representation is encoded by the encoder 510 in the first encoding stage to obtain the encoded feature representation F1. Then, the image feature representation is encoded by the encoder 510 in the second encoding stage to obtain the encoded feature representation F2. The image feature representation is encoded by the encoder 510 in the third encoding stage to obtain the encoded feature representation F3. Finally, the image feature representation is encoded by the encoder 510 in the fourth encoding stage to obtain the encoded feature representation F4.

[0174] Among them, F1, F2, and F3 are respectively obtained by the above channel mixing and other operations to obtain F1', F2', and F3'.

[0175] In the first decoding stage, the decoder 520 of the first decoding stage decodes the encoded feature representation F4 of the fourth encoding stage to obtain the decoded feature representation F of the first decoding stage. out1 In the second decoding stage, the decoding feature representation F of the first decoding stage is... out1 The feature representation F3' obtained in the third encoding stage is concatenated with the feature representation F3' obtained in the second decoding stage to obtain the encoded-decoded fused feature representation. This feature representation is then decoded by the decoder 520 in the second decoding stage to obtain the decoded feature representation F3' in the second decoding stage. out2 In the third decoding stage, the decoding feature representation F of the second decoding stage is... out2 The feature representation F2' obtained in the second encoding stage is concatenated with the feature representation F2' obtained in the third decoding stage to obtain the encoded-decoded fused feature representation. This feature representation is then decoded by the decoder 520 in the third decoding stage to obtain the decoded feature representation F2'. out3 In the fourth decoding stage, the decoding feature representation F of the third decoding stage is... out3The feature representation F1' obtained in the first encoding stage is concatenated with the feature representation F1' obtained in the fourth decoding stage to obtain the encoded-decoded fused feature representation. This feature representation is then decoded by the decoder 520 in the fourth decoding stage to obtain the decoded feature representation F1'. out4 .

[0176] By establishing skip connections between the codec and decoder through GCM, the encoded feature representation and the decoded feature representation are fused, and the next decoding stage is performed based on the fused encoded and decoded feature representation, which improves the decoding accuracy and the accuracy of the segmentation prediction result of the output first image content.

[0177] In some embodiments, after obtaining the decoding feature representation of the q-th decoding stage, it is determined whether q is equal to m, where m is the number of decoding stages. That is, it is determined whether the q-th decoding stage is the last decoding stage. If so, step 460 is executed; otherwise, q is set to q+1, and step 450 is executed repeatedly.

[0178] Step 460: Based on the decoding feature representation of m decoding stages, predict the content of the first image to obtain the segmentation prediction result of the first image content in the first image.

[0179] The decoded feature representation output by the decoder passes through an output layer, and the final output of the network is: S1 = softmax(Conv 1×1 (F out1 S2 = softmax(Conv) 1×1 (F out2 S3 = softmax(Conv) 1×1 (F out3 S4 = softmax(Conv) 1×1 (F out4 )).

[0180] Based on S1, S2, S3, and S4, the segmentation prediction result of the first image content in the first image is determined. Schematically, based on S1, S2, S3, and S4, the weighted average of the probabilities of corresponding pixels belonging to the first image content is determined, the probability of pixels in the first image belonging to the first image content is determined, and the segmentation prediction result of the first image content is determined based on the probability of pixels belonging to the first image content.

[0181] In summary, the image segmentation method provided in this application generates a global feature representation of the first image and obtains a local feature representation of the first image through a recursive linear attention mechanism. This combines the global features of the first image with the local features of each image region to comprehensively analyze the content of the first image, thereby segmenting the content of the first image from the first image and obtaining a segmentation prediction result. This avoids the problem of a one-sided analysis of the first image and insufficient analysis of long-range features caused by only analyzing each image region. It considers both the local features of each image region and the correlation between relatively distant pixels in the first image, i.e., it introduces the long-range features of the first image, improving the accuracy of the analysis of the first image and the segmentation prediction accuracy of the content of the first image from the first image.

[0182] In some embodiments, the image segmentation method provided in this application is executed by an image segmentation model, wherein the image segmentation model is trained using sample images. Figure 6 This is a flowchart of an exemplary embodiment of an image segmentation method provided in this application, which is executed by a computer device, such as... Figure 6 As shown, the method includes the following steps 610 to 630.

[0183] Step 610: Obtain the sample image.

[0184] The sample images are labeled with reference content areas corresponding to the content of the first image.

[0185] In related technologies, semantic segmentation tasks typically involve a batch of multiple images containing P categories, where each sample is an image of shape R. B×C×H×W Let represent an image batch with B samples, C channels, H height, and W width. During the training phase, a neural network extracts the feature representation of each image, performs feature map encoding and decoding, and finally outputs the class label for each pixel. The class label for each pixel is predicted using a softmax activation function or other activation functions, resulting in S∈R. B×P×H×W , representing the probability distribution of the image output.

[0186] The corresponding truth label G usually has shape R B×H×W For semantic segmentation tasks, ground truth labels are typically transformed into a single-channel class index map (using sparse integer values ​​to represent class indices). That is, taking the first image content as an example, if a pixel belongs to the first image content, it is assigned a value of 1; if a pixel does not belong to the first image content, it is assigned a value of 0. These ground truth labels are used for supervised training and to calculate cross-entropy loss or other related loss functions.

[0187] Taking the segmentation task of the first image content as an example, the reference content region is used to indicate whether a pixel in the sample image belongs to the first image content. Illustratively, pixels within the reference content region have a value of 1, while pixels outside the reference content region have a value of 0.

[0188] Step 620: Predict the first image content in the sample image using an image segmentation model to obtain the predicted content region.

[0189] An image segmentation model is a model to be trained. In some embodiments, the image segmentation model has the same structure before and after training, but the parameters applied in the model are different.

[0190] When predicting the first image content in a sample image using an image segmentation model, the encoder in the above embodiments is used to encode the sample image, and the decoder in the above embodiments is used to decode the encoded feature representation to obtain the predicted content region of the first image content in the sample image.

[0191] In some embodiments, an image segmentation model is used to predict the probability that a pixel in a sample image belongs to the content of a first image, and the predicted content region in the sample image that belongs to the content of the first image is determined based on the probability.

[0192] That is, by predicting the first image content in the sample image through the image segmentation model, the predicted probability of the pixels in the sample image belonging to the first image content is obtained, and the pixels in the sample image belonging to the first image content are determined based on the predicted probability, thus obtaining the predicted content region.

[0193] Step 630: Train the image segmentation model based on the difference between the reference content region and the predicted content region.

[0194] In some embodiments, a loss value is obtained based on the difference between the reference content region and the predicted content region, and an image segmentation model is trained based on the loss value.

[0195] When obtaining the loss value based on the difference between the reference content region and the predicted content region, at least one of the following losses is included:

[0196] First, a first loss value is obtained based on the pixel differences between the reference content region and the predicted content region; then, an image segmentation model is trained based on the first loss value.

[0197] In some embodiments, a cross-entropy loss, also known as a first loss value, is calculated for the reference content region and the predicted content region. Illustratively, the calculation process for the cross-entropy loss is shown in Formula 1 below:

[0198] Formula 1:

[0199] Where N is the number of pixels in the sample image, i represents the i-th pixel, P is the number of categories to be classified, p is the p-th category, the content of the first image belongs to one of the categories to be classified, and k is the number of decoders, that is, the decoded feature obtained by the k-th decoder represents a prediction result of the corresponding sample image. G pi S represents the classification label of the i-th pixel in the p-th category. pik This represents the classification probability of the i-th pixel in the p-th category, output by the k-th decoder.

[0200] The cross-entropy loss is obtained as the first loss value based on the differences between pixels, which improves the efficiency of loss value calculation and the training efficiency of image segmentation model.

[0201] Second, obtain the number of intersection pixels between the reference content region and the predicted content region; obtain the number of union pixels between the reference content region and the predicted content region; obtain a second loss value based on the number of intersection pixels and the number of union pixels; and train an image segmentation model based on the second loss value.

[0202] In some embodiments, a similarity loss (Dice Loss), also known as a second loss value, is calculated for the reference content region and the predicted content region. The calculation process for the similarity loss is illustrated in Formula 2 below:

[0203] Formula 2:

[0204] Wherein, ∈ is a smoothing factor to prevent the numerator or denominator from being 0.

[0205] The second loss value is determined by the intersection and union of pixels in the reference content region and the predicted content region. The segmentation accuracy of the image segmentation model is expressed from the pixel overlap relationship, which improves the training accuracy of the image segmentation model.

[0206] Third, obtain the first edge pixel of the reference content region; and obtain the second edge pixel of the predicted content region. Obtain a third loss value based on the distance between the first edge pixel and the second edge pixel, and train the image segmentation model based on the third loss value.

[0207] In some embodiments, the edge loss between the reference content region and the predicted content region, i.e., the third loss value, is obtained. For the edge loss, firstly, edge detection operators (such as the Canny operator, or, in deep learning, differentiable edge detection methods such as gradient-based edge detection) are used to extract edges. Let the extracted edges of the predicted content region be E(G). pi The edge of the reference content area is E(S). pikE() refers to the edge extraction operation. For each category p, the average distance between the predicted edge and the reference edge is calculated. In this embodiment, the distance between the pixels of the predicted edge and the reference edge is calculated, and the total number of pixels in the sample is N = H × W.

[0208] The calculation process for edge loss is shown in Formula 3 below:

[0209] Formula 3:

[0210] Among them, E(G pi E(S) represents the coordinates of the predicted content region of the sample image at the i-th pixel and the p-th category. pik Let represent the coordinates of the reference content region of the sample image at the i-th pixel and the p-th category, and d() represent the distance function used to calculate the distance between two edge points. In some embodiments, the distance function includes the Euclidean distance function, the Manhattan distance function, etc., and this application embodiment does not limit it to this.

[0211] The edge loss value is determined by the difference between the edge of the predicted content region and the edge of the reference content region. This reduces the number of pixels required for distance calculation and also expresses the error of the predicted content region obtained from the perspective of the outer contour, further improving the training accuracy of the image segmentation model.

[0212] In some embodiments, after obtaining the first loss value, the second loss value, and the third loss value, the first loss value, the second loss value, and the third loss value are fused to obtain a total loss value, and an image segmentation model is trained based on the total loss value.

[0213] To illustrate, the first loss value, the second loss value, and the third loss value are weighted and combined to obtain the total loss value.

[0214] As an illustration, the combination loss is shown in Formula 4 below:

[0215] Formula 4: L k =αL ck +βL Dk +γL Rk k = 1, 2, 3, 4

[0216] Here, α, β, and γ are pre-defined weighting coefficients used to balance the three loss values. The final loss value includes the sum of the segmentation prediction results from the four decoder outputs. That is,

[0217] In summary, the method provided in this application efficiently fuses long-range and local information from the encoder, improving segmentation performance. Specifically, it introduces a VRWKV structure into the encoder architecture, endowing it with the ability to capture long-range dependency information of the image. This capture of long-range dependency information is crucial for understanding the global semantics of an image. Simultaneously, by combining the fine-grained representation advantages of convolutional neural networks (CNNs) in local feature extraction, the encoder can accurately grasp the detailed information of the image at a microscopic level. In this way, a deep fusion and collaborative modeling of global semantic information and local detailed information is achieved. This collaborative modeling mechanism can more comprehensively and accurately characterize image features, thereby significantly improving the accuracy of segmentation results in image segmentation tasks.

[0218] The design of the MSP module in the encoder improves the model performance. That is, the embodiment of this application designs an efficient encoder with an MSP module. This module includes a global branch based on VRWKV spatial mixing, a local branch using convolution with different kernel sizes, and an identity branch, thereby effectively improving the overall performance of the model without significantly increasing the amount of computation.

[0219] The innovation of the global channel fusion module enhances segmentation capabilities. Specifically, the method provided in this application proposes a novel global channel fusion module that optimizes skip connections, promotes the fusion of global channels, significantly enhances the model's ability to capture long-distance dependencies, and thus greatly improves the accuracy and efficiency of segmentation, enhancing the model's segmentation performance in complex scenes.

[0220] The image segmentation method provided in this application can be applied to medical image recognition scenarios, obstacle recognition tasks for road condition images in autonomous driving scenarios, agricultural area recognition tasks for satellite images in satellite image analysis scenarios, and specific part recognition tasks for human images in portrait processing scenarios, etc., without limitation.

[0221] In some optional embodiments, a medical image recognition scenario is taken as an example. Illustratively, medical images, such as X-ray images and ultrasound images, are acquired. The medical image is the image to be segmented to obtain the lesion region. The medical image is convolved with a kernel of at least one size to generate a local feature representation of the medical image; a global feature representation of the medical image is generated through a recursive linear attention mechanism, which expresses the global features of the medical image in the spatial dimension; the global feature representation and the local feature representation are fused to obtain a fused feature representation; the lesion region is predicted based on the fused feature representation to obtain the segmentation prediction result of the lesion region in the segmented medical image.

[0222] Indicative, Figure 7 This is a schematic diagram of a lesion area identification process provided in an exemplary embodiment of this application, as shown below. Figure 7 As shown, a medical image 700 is acquired, and convolved with a kernel of at least one size to obtain a local feature representation 710. A global feature representation 720 of the medical image 700 is generated through a recursive linear attention mechanism. The local feature representation 710 and the global feature representation 720 are fused to obtain a fused feature representation 730. The lesion region is predicted based on the fused feature representation 730. The segmentation prediction result 740 of the lesion region in the segmented medical image is obtained.

[0223] In some embodiments, an image feature representation of a medical image is obtained, which is a feature representation obtained by converting the color channels of the medical image according to a preset channel conversion relationship; a global feature representation of the image feature representation is generated through a recursive linear attention mechanism.

[0224] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a first feature representation containing at least one channel dimension, the first feature representation being the feature representation applied to the global branch; and a global feature representation corresponding to the first feature representation is generated through a recursive linear attention mechanism.

[0225] In some embodiments, the first feature representation is divided into at least two feature blocks of a preset feature size; adjacent feature blocks in the at least two feature blocks are cross-mixed to obtain a hybrid feature representation of the medical image; the hybrid feature representation is transformed based on a bidirectional global attention mechanism to obtain a global feature representation of the medical image.

[0226] In some embodiments, an image feature representation of a medical image is obtained, which is a feature representation obtained by converting the color channels of the medical image according to a preset channel conversion relationship; the image feature representation is convolved by a convolution kernel of at least one size to obtain a local feature representation of the medical image.

[0227] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a second feature representation containing at least one channel dimension, the second feature representation being a feature representation applied to local branches; the second feature representation is convolved with a convolution kernel of at least one size to obtain a local feature representation of the medical image.

[0228] In some embodiments, the second feature representation is split into n sub-feature representations, where n is an integer greater than 1; the i-th sub-feature representation is convolved with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; the convolutional feature representations corresponding to the n sub-feature representations are obtained respectively; and the convolutional feature representations corresponding to the n sub-feature representations are concatenated to obtain the local feature representation of the medical image.

[0229] In some embodiments, the image segmentation method for medical images is performed by an image segmentation model, which includes an encoder and a decoder. In the k-th encoding stage of the encoder, the global and local feature representations of the k-th encoding stage are fused to obtain a fused feature representation for the k-th encoding stage, where k is a positive integer. In the q-th decoding stage of the decoder, the fused feature representations of the (m+1-q)-th encoding stages and the decoded feature representations of the (q-1)-th decoding stages are fused to obtain a fused encoding-decoding feature representation for the q-th decoding stage. The fused encoding-decoding feature representation for the q-th decoding stage is then decoded to obtain a decoded feature representation for the q-th decoding stage, where m is the number of decoding stages and q is a positive integer. Based on the decoded feature representations of the m-th decoding stages, the medical image content is predicted to obtain a segmentation prediction result for the medical image content in the segmented medical image.

[0230] Optionally, based on the channel mixing mode, the fused feature representations of at least two coding stages are fused to obtain the coded fused feature representation; the coded fused feature representation is then split into at least two coded feature representations, where the (m+1-q)th coding stage corresponds to the (m+1-q)th coded feature representation; the (m+1-q)th coded feature representation and the decoding feature representation of the (q-1)th decoding stage are fused to obtain the coded-decoded fused feature representation of the qth decoding stage. Here, the channel mixing mode refers to the channel mixing mode applied to the fused feature representations of at least two coding stages using GCM.

[0231] In some embodiments, an image feature representation of a medical image is obtained, the image feature representation including at least two channel dimensions; the at least two channel dimensions are split to obtain an identity feature representation including at least one channel dimension; the global feature representation, the local feature representation and the identity feature representation are fused to obtain a fused feature representation.

[0232] In some embodiments, the image segmentation method for medical images is performed by an image segmentation model. The training process of the image segmentation model includes: acquiring sample images, which are labeled with reference lesion regions corresponding to medical image content; predicting the medical image content in the sample images using the image segmentation model to obtain predicted lesion regions; and training the image segmentation model based on the difference between the reference lesion regions and the predicted lesion regions.

[0233] In some embodiments, the medical image content in the sample image is predicted by an image segmentation model to obtain the predicted probability that a pixel in the sample image belongs to the medical image content; the pixels in the sample image that belong to the medical image content are determined based on the predicted probability to obtain the predicted lesion region.

[0234] In some embodiments, a first loss value is obtained based on the pixel differences between the reference lesion region and the predicted lesion region; an image segmentation model is trained based on the first loss value.

[0235] In some embodiments, the number of intersection pixels between the reference lesion region and the predicted lesion region is obtained; the number of union pixels between the reference lesion region and the predicted lesion region is obtained; a second loss value is obtained based on the number of intersection pixels and the number of union pixels; and an image segmentation model is trained based on the second loss value.

[0236] In some embodiments, a first edge pixel of a reference lesion region is obtained; and a second edge pixel of a predicted lesion region is obtained; a third loss value is obtained based on the distance between the first edge pixel and the second edge pixel; and an image segmentation model is trained based on the third loss value.

[0237] In summary, the image segmentation method provided in this application generates a global feature representation of the medical image and obtains a local feature representation of the medical image through a recursive linear attention mechanism. This combines the global features of the medical image with the local features of each image region to comprehensively analyze the medical image content, thereby segmenting the medical image content and obtaining a segmentation prediction result. This avoids the problem of one-sided analysis and insufficient long-range feature analysis caused by only analyzing each image region. By considering both the local features of each image region and the correlation between relatively distant pixels in the medical image (i.e., introducing long-range features), the accuracy of medical image analysis and the segmentation prediction accuracy of medical image content segmentation are improved.

[0238] In some optional embodiments, taking an autonomous driving scenario as an example, road images are acquired, such as road images captured by an onboard camera or road cameras. The road image is the image to be segmented to obtain the obstacle region. The road image is convolved with a kernel of at least one size to generate a local feature representation of the road image; a global feature representation of the road image is generated through a recursive linear attention mechanism, which expresses the global features of the road image in the spatial dimension; the global feature representation and the local feature representation are fused to obtain a fused feature representation; based on the fused feature representation, obstacle regions are predicted to obtain the segmentation prediction result of the obstacle region in the segmented road image.

[0239] In some embodiments, an image feature representation of a road image is obtained, which is a feature representation obtained by transforming the color channels of the road image according to a preset channel transformation relationship; a global feature representation of the image feature representation is generated through a recursive linear attention mechanism.

[0240] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a first feature representation containing at least one channel dimension, the first feature representation being the feature representation applied to the global branch; and a global feature representation corresponding to the first feature representation is generated through a recursive linear attention mechanism.

[0241] In some embodiments, the first feature representation is divided into at least two feature blocks of a preset feature size; adjacent feature blocks in the at least two feature blocks are cross-mixed to obtain a mixed feature representation of the road image; the mixed feature representation is transformed based on a bidirectional global attention mechanism to obtain a global feature representation of the road image.

[0242] In some embodiments, an image feature representation of a road image is obtained, which is a feature representation obtained by converting the color channels of the road image according to a preset channel conversion relationship; the image feature representation is convolved by a convolution kernel of at least one size to obtain a local feature representation of the road image.

[0243] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a second feature representation containing at least one channel dimension, the second feature representation being a feature representation applied to local branches; the second feature representation is convolved with a convolution kernel of at least one size to obtain a local feature representation of the road image.

[0244] In some embodiments, the second feature representation is split into n sub-feature representations, where n is an integer greater than 1; the i-th sub-feature representation is convolved with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; the convolutional feature representations corresponding to the n sub-feature representations are obtained respectively; and the convolutional feature representations corresponding to the n sub-feature representations are concatenated to obtain the local feature representation of the road image.

[0245] In some embodiments, the image segmentation method for the road image is performed by an image segmentation model, which includes an encoder and a decoder. In the k-th encoding stage of the encoder, the global and local feature representations of the k-th encoding stage are fused to obtain a fused feature representation for the k-th encoding stage, where k is a positive integer. In the q-th decoding stage of the decoder, the fused feature representations of the (m+1-q)-th encoding stages and the decoded feature representations of the (q-1)-th decoding stages are fused to obtain a fused encoding / decoding feature representation for the q-th decoding stage. The fused encoding / decoding feature representation for the q-th decoding stage is then decoded to obtain a decoded feature representation for the q-th decoding stage, where m is the number of decoding stages and q is a positive integer. Based on the decoded feature representations of the m-th decoding stages, the road image content is predicted to obtain a segmentation prediction result for the road image content in the segmented road image.

[0246] Optionally, based on the channel mixing mode, the fused feature representations of at least two coding stages are fused to obtain the coded fused feature representation; the coded fused feature representation is then split into at least two coded feature representations, where the (m+1-q)th coding stage corresponds to the (m+1-q)th coded feature representation; the (m+1-q)th coded feature representation and the decoding feature representation of the (q-1)th decoding stage are fused to obtain the coded-decoded fused feature representation of the qth decoding stage. Here, the channel mixing mode refers to the channel mixing mode applied to the fused feature representations of at least two coding stages using GCM.

[0247] In some embodiments, an image feature representation of a road image is obtained, the image feature representation including at least two channel dimensions; the at least two channel dimensions are split to obtain an identity feature representation containing at least one channel dimension; the global feature representation, the local feature representation and the identity feature representation are fused to obtain a fused feature representation.

[0248] In some embodiments, the image segmentation method for road images is performed by an image segmentation model. The training process of the image segmentation model includes: acquiring sample images, which are labeled with reference obstacle regions corresponding to the road image content; predicting the road image content in the sample images using the image segmentation model to obtain predicted obstacle regions; and training the image segmentation model based on the difference between the reference obstacle regions and the predicted obstacle regions.

[0249] In some embodiments, the road image content in the sample image is predicted by an image segmentation model to obtain the predicted probability that a pixel in the sample image belongs to the road image content; the pixels in the sample image that belong to the road image content are determined based on the predicted probability to obtain the predicted obstacle region.

[0250] In some embodiments, a first loss value is obtained based on the pixel differences between the reference obstacle region and the predicted obstacle region; an image segmentation model is trained based on the first loss value.

[0251] In some embodiments, the number of intersection pixels between the reference obstacle region and the predicted obstacle region is obtained; the number of union pixels between the reference obstacle region and the predicted obstacle region is obtained; a second loss value is obtained based on the number of intersection pixels and the number of union pixels; and an image segmentation model is trained based on the second loss value.

[0252] In some embodiments, a first edge pixel of a reference obstacle region is obtained; and a second edge pixel of a predicted obstacle region is obtained; a third loss value is obtained based on the distance between the first edge pixel and the second edge pixel; and an image segmentation model is trained based on the third loss value.

[0253] In summary, the image segmentation method provided in this application generates a global feature representation of a road image and obtains a local feature representation of the road image through a recursive linear attention mechanism. This combines the global features of the road image with the local features of each image region to comprehensively analyze the road image content, thereby segmenting the road image content and obtaining a segmentation prediction result. This avoids the problem of one-sided analysis and insufficient long-range feature analysis caused by only analyzing each image region of the road image. It considers both the local features of each image region and the correlation between relatively distant pixels in the road image, i.e., it introduces long-range features of the road image, improving the accuracy of road image analysis and the segmentation prediction accuracy of road image content segmentation.

[0254] In some optional embodiments, a portrait processing scenario is taken as an example. Illustratively, a captured image is obtained, such as a video frame from a captured video of a person, or a captured static image of a person. The captured image is the image from which the person region is to be obtained through image segmentation. The captured image is convolved with a kernel of at least one size to generate a local feature representation of the captured image; a global feature representation of the captured image is generated through a recursive linear attention mechanism, which expresses the global features of the captured image in the spatial dimension; the global feature representation and the local feature representation are fused to obtain a fused feature representation; based on the fused feature representation, the person region is predicted, resulting in a segmentation prediction result for the person region in the captured image.

[0255] In some embodiments, an image feature representation of the captured image is obtained, which is a feature representation obtained by transforming the color channels of the captured image according to a preset channel transformation relationship; a global feature representation of the image feature representation is generated through a recursive linear attention mechanism.

[0256] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a first feature representation containing at least one channel dimension, the first feature representation being the feature representation applied to the global branch; and a global feature representation corresponding to the first feature representation is generated through a recursive linear attention mechanism.

[0257] In some embodiments, the first feature representation is divided into at least two feature blocks of a preset feature size; adjacent feature blocks in the at least two feature blocks are cross-mixed to obtain a mixed feature representation of the captured image; the mixed feature representation is transformed based on a bidirectional global attention mechanism to obtain a global feature representation of the captured image.

[0258] In some embodiments, an image feature representation of the captured image is obtained, which is a feature representation obtained by converting the color channels of the captured image according to a preset channel conversion relationship; the image feature representation is convolved by a convolution kernel of at least one size to obtain a local feature representation of the captured image.

[0259] In some embodiments, the image feature representation includes at least two channel dimensions; the at least two channel dimensions are split to obtain a second feature representation containing at least one channel dimension, the second feature representation being a feature representation applied to a local branch; the second feature representation is convolved with a convolution kernel of at least one size to obtain a local feature representation of the captured image.

[0260] In some embodiments, the second feature representation is split into n sub-feature representations, where n is an integer greater than 1; the i-th sub-feature representation is convolved with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; the convolutional feature representations corresponding to the n sub-feature representations are obtained respectively; and the convolutional feature representations corresponding to the n sub-feature representations are concatenated to obtain the local feature representation of the captured image.

[0261] In some embodiments, the image segmentation method for the captured image is performed by an image segmentation model, which includes an encoder and a decoder. In the k-th encoding stage of the encoder, the global feature representation and local feature representation of the k-th encoding stage are fused to obtain a fused feature representation of the k-th encoding stage, where k is a positive integer. In the q-th decoding stage of the decoder, the fused feature representation of the (m+1-q)-th encoding stage and the decoded feature representation of the (q-1)-th decoding stage are fused to obtain a fused encoding / decoding feature representation of the q-th decoding stage. The fused encoding / decoding feature representation of the q-th decoding stage is then decoded to obtain a decoded feature representation of the q-th decoding stage, where m is the number of decoding stages and q is a positive integer. Based on the decoded feature representations of the m-th decoding stages, the captured image content is predicted to obtain a segmentation prediction result for segmenting the captured image content in the captured image.

[0262] Optionally, based on the channel mixing mode, the fused feature representations of at least two coding stages are fused to obtain the coded fused feature representation; the coded fused feature representation is then split into at least two coded feature representations, where the (m+1-q)th coding stage corresponds to the (m+1-q)th coded feature representation; the (m+1-q)th coded feature representation and the decoding feature representation of the (q-1)th decoding stage are fused to obtain the coded-decoded fused feature representation of the qth decoding stage. Here, the channel mixing mode refers to the channel mixing mode applied to the fused feature representations of at least two coding stages using GCM.

[0263] In some embodiments, an image feature representation of the captured image is obtained, the image feature representation including at least two channel dimensions; the at least two channel dimensions are split to obtain an identity feature representation containing at least one channel dimension; the global feature representation, the local feature representation and the identity feature representation are fused to obtain a fused feature representation.

[0264] In some embodiments, the image segmentation method for the captured image is performed by an image segmentation model. The training process of the image segmentation model includes: acquiring sample images, which are labeled with reference person regions corresponding to the content of the captured image; predicting the content of the captured image in the sample images using the image segmentation model to obtain predicted person regions; and training the image segmentation model based on the difference between the reference person regions and the predicted person regions.

[0265] In some embodiments, the image segmentation model is used to predict the content of the captured image in the sample image, thereby obtaining the predicted probability that a pixel in the sample image belongs to the captured image content; based on the predicted probability, the pixels in the sample image that belong to the captured image content are determined, thereby obtaining the predicted person region.

[0266] In some embodiments, a first loss value is obtained based on the pixel differences between the reference person region and the predicted person region; an image segmentation model is trained based on the first loss value.

[0267] In some embodiments, the number of intersection pixels between the reference figure region and the predicted figure region is obtained; the number of union pixels between the reference figure region and the predicted figure region is obtained; a second loss value is obtained based on the number of intersection pixels and the number of union pixels; and an image segmentation model is trained based on the second loss value.

[0268] In some embodiments, a first edge pixel of a reference person region is obtained; and a second edge pixel of a predicted person region is obtained; a third loss value is obtained based on the distance between the first edge pixel and the second edge pixel; and an image segmentation model is trained based on the third loss value.

[0269] In summary, the image segmentation method provided in this application generates a global feature representation of the captured image and obtains a local feature representation of the captured image through a recursive linear attention mechanism. This combines the global features of the captured image with the local features of each image region to comprehensively analyze the content of the captured image, thereby segmenting the captured image content from the captured image and obtaining a segmentation prediction result. This avoids the problem of a one-sided analysis of the captured image and insufficient analysis of long-range features caused by only analyzing each image region. It considers both the local features of each image region and the correlation between relatively distant pixels in the captured image, i.e., it introduces long-range features of the captured image, improving the accuracy of the image analysis and the segmentation prediction accuracy of the captured image content.

[0270] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0271] Please refer to Figure 8 This illustrates a block diagram of an image segmentation apparatus provided in one embodiment of this application. Figure 8 As shown, the device includes:

[0272] The acquisition module 810 is used to acquire a first image, wherein the first image is an image for which image segmentation is performed to acquire the content of the first image;

[0273] The convolution module 820 is used to convolve the first image with a convolution kernel of at least one size to generate a local feature representation of the first image;

[0274] The generation module 830 is used to generate a global feature representation of the first image through a recursive linear attention mechanism, wherein the global feature representation is used to express the global features of the first image in the spatial dimension.

[0275] The fusion module 840 is used to fuse the global feature representation and the local feature representation to obtain a fused feature representation;

[0276] The prediction module 850 is used to predict the content of the first image based on the fused feature representation, and to obtain the segmentation prediction result of the first image content in the first image.

[0277] In an optional embodiment, the acquisition module 810 is further configured to acquire the image feature representation of the first image, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship;

[0278] The generation module 830 is further configured to generate the global feature representation of the image feature representation through a recursive linear attention mechanism.

[0279] In an optional embodiment, the image feature representation includes at least two channel dimensions;

[0280] The generation module 830 is further configured to split the at least two channel dimensions to obtain a first feature representation containing at least one channel dimension, wherein the first feature representation is a feature representation applied to the global branch; and generate the global feature representation corresponding to the first feature representation through a recursive linear attention mechanism.

[0281] In an optional embodiment, the generation module 830 is further configured to divide the first feature representation into at least two feature blocks of a preset feature size; cross-mix adjacent feature blocks in the at least two feature blocks to obtain a mixed feature representation; and transform the mixed feature representation based on a bidirectional global attention mechanism to obtain the global feature representation.

[0282] In an optional embodiment, the generation module 830 is further configured to obtain an image feature representation of the first image, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship; and to obtain the local feature representation of the first image by convolving the image feature representation with a convolution kernel of at least one size.

[0283] In an optional embodiment, the image feature representation includes at least two channel dimensions;

[0284] The generation module 830 is further configured to split the at least two channel dimensions to obtain a second feature representation containing at least one channel dimension, wherein the second feature representation is a feature representation applied to a local branch;

[0285] The convolution module 820 is further configured to convolve the second feature representation with a convolution kernel of the at least one size to obtain the local feature representation of the first image.

[0286] In an optional embodiment, the convolution module 820 is further configured to: split the second feature representation into n sub-feature representations, where n is an integer greater than 1; convolve the i-th sub-feature representation with a convolution kernel of size i to obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; obtain the convolutional feature representations corresponding to the n sub-feature representations respectively; and concatenate the convolutional feature representations corresponding to the n sub-feature representations respectively to obtain the local feature representation of the first image.

[0287] In an optional embodiment, the apparatus includes an image segmentation model, which includes an encoder and a decoder;

[0288] The fusion module 840 is further configured to fuse the global feature representation and the local feature representation of the k-th encoding stage in the k-th encoding stage to obtain the fused feature representation of the k-th encoding stage, where k is a positive integer;

[0289] The prediction module 850 is further configured to, in the q-th decoding stage of the decoder, fuse the fusion feature representation of the (m+1-q)-th encoding stage and the decoding feature representation of the (q-1)-th decoding stage to obtain the encoding-decoding fusion feature representation of the q-th decoding stage, and decode the encoding-decoding fusion feature representation of the q-th decoding stage to obtain the decoding feature representation of the q-th decoding stage, where m is the number of decoding stages and q is a positive integer; predict the first image content based on the decoding feature representation of the m-th decoding stages to obtain the segmentation prediction result of segmenting the first image content in the first image.

[0290] In an optional embodiment, the prediction module is further configured to fuse the fusion feature representations of the at least two encoding stages based on the channel mixing mode to obtain an encoding fusion feature representation; split the encoding fusion feature representation into at least two encoding feature representations, wherein the (m+1-q)th encoding stage corresponds to the (m+1-q)th encoding feature representation; and fuse the (m+1-q)th encoding feature representation and the decoding feature representation of the (q-1)th decoding stage to obtain the encoding-decoding fusion feature representation of the qth decoding stage.

[0291] In an optional embodiment, the acquisition module 810 is further configured to acquire an image feature representation of the first image, the image feature representation including at least two channel dimensions;

[0292] The generation module 830 is further configured to split the at least two channel dimensions to obtain an identity feature representation containing at least one channel dimension;

[0293] The fusion module 840 is further configured to fuse the global feature representation, the local feature representation, and the identity feature representation to obtain the fused feature representation.

[0294] In an optional embodiment, the apparatus includes an image segmentation model;

[0295] The acquisition module 810 is further configured to acquire a sample image, wherein the sample image is marked with a reference content area corresponding to the content of the first image;

[0296] The prediction module 850 is further configured to predict the first image content in the sample image through the image segmentation model to obtain the predicted content region;

[0297] like Figure 9 As shown, the device further includes:

[0298] Training module 860 is used to train the image segmentation model based on the difference between the reference content region and the predicted content region.

[0299] In an optional embodiment, the prediction module 850 is further configured to predict the first image content in the sample image through the image segmentation model, and obtain the predicted probability that the pixels in the sample image belong to the first image content; and determine the pixels in the sample image that belong to the first image content based on the predicted probability, thereby obtaining the predicted content region.

[0300] In an optional embodiment, the training module 860 is further configured to obtain a first loss value based on the pixel difference between the reference content region and the predicted content region; and train the image segmentation model based on the first loss value.

[0301] In an optional embodiment, the training module 860 is further configured to obtain the number of intersection pixels between the reference content region and the predicted content region; obtain the number of union pixels between the reference content region and the predicted content region; obtain a second loss value based on the number of intersection pixels and the number of union pixels; and train the image segmentation model based on the second loss value.

[0302] In an optional embodiment, the training module 860 is further configured to: acquire a first edge pixel of the reference content region; acquire a second edge pixel of the predicted content region; acquire a third loss value based on the distance between the first edge pixel and the second edge pixel; and train the image segmentation model based on the third loss value.

[0303] In summary, the apparatus provided in this application generates a global feature representation of a first image and obtains a local feature representation of the first image through a recursive linear attention mechanism. This combines the global features of the first image with the local features of each image region to comprehensively analyze the content of the first image, thereby segmenting the content of the first image from the first image and obtaining a segmentation prediction result. This avoids the problem of a one-sided analysis of the first image and insufficient analysis of long-range features caused by analyzing only each image region. It considers both the local features of each image region and the correlation between relatively distant pixels in the first image, i.e., it introduces the long-range features of the first image, improving the accuracy of the analysis of the first image and the segmentation prediction accuracy of segmenting the content of the first image from the first image.

[0304] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0305] Please refer to Figure 10 This diagram illustrates a structural block diagram of a computer device 1000 provided in one embodiment of this application. The computer device 1000 may be... Figure 1 The terminal or server in the computer system shown is used to implement the image segmentation method provided in the above embodiments. Specifically:

[0306] Typically, computer device 1000 includes a processor 1001 and a memory 1002.

[0307] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0308] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store a computer program configured to be executed by one or more processors to implement the image segmentation method described above.

[0309] In some embodiments, the computer device 1000 may also optionally include other components 1003: a peripheral device interface and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: radio frequency circuitry, a display screen, audio circuitry, and a power supply.

[0310] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the computer device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0311] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the image segmentation method described above. Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0312] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the image segmentation method described above.

[0313] It should be noted that the collection and processing of relevant data (such as medical images) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0314] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0315] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: Acquire a first image, which is an image whose content is to be obtained through image segmentation; A global feature representation of the first image is generated through a recursive linear attention mechanism, wherein the global feature representation is used to express the global features of the first image in the spatial dimension; The first image is convolved with a convolution kernel of at least one size to generate a local feature representation of the first image; The global feature representation and the local feature representation are fused to obtain the fused feature representation; Based on the fusion feature representation, the content of the first image is predicted, and the segmentation prediction result of the first image content in the first image is obtained.

2. The method according to claim 1, characterized in that, The generation of the global feature representation of the first image through a recursive linear attention mechanism includes: The image feature representation of the first image is obtained, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship; The global feature representation of the image feature representation is generated through a recursive linear attention mechanism.

3. The method according to claim 2, characterized in that, The image feature representation includes at least two channel dimensions; The generation of the global feature representation of the image feature representation through a recursive linear attention mechanism includes: The at least two channel dimensions are split to obtain a first feature representation containing at least one channel dimension, and the first feature representation is a feature representation applied to the global branch. The global feature representation corresponding to the first feature representation is generated through a recursive linear attention mechanism.

4. The method according to claim 3, characterized in that, The step of generating the global feature representation corresponding to the first feature representation through a recursive linear attention mechanism includes: The first feature representation is divided into at least two feature blocks of a preset feature size; By cross-mixing adjacent feature blocks in the at least two feature blocks, a mixed feature representation is obtained; The hybrid feature representation is transformed based on a bidirectional global attention mechanism to obtain the global feature representation.

5. The method according to any one of claims 1 to 4, characterized in that, The step of convolving the first image with a convolution kernel of at least one size to generate a local feature representation of the first image includes: The image feature representation of the first image is obtained, wherein the image feature representation is a feature representation obtained by converting the color channels of the first image according to a preset channel conversion relationship; The local feature representation of the first image is obtained by convolving the image feature representation with a convolution kernel of at least one size.

6. The method according to claim 5, characterized in that, The image feature representation includes at least two channel dimensions; The step of convolving the image feature representation with a convolution kernel of at least one size to obtain the local feature representation of the first image includes: The at least two channel dimensions are split to obtain a second feature representation containing at least one channel dimension, and the second feature representation is a feature representation applied to the local branch. The local feature representation of the first image is obtained by convolving the second feature representation with a convolution kernel of at least one size.

7. The method according to claim 6, characterized in that, The step of convolving the second feature representation with a convolution kernel of at least one size to obtain the local feature representation of the first image includes: The second feature representation is split into n sub-feature representations, where n is an integer greater than 1; By convolving the i-th sub-feature representation with the i-th size convolution kernel, we obtain the convolutional feature representation corresponding to the i-th sub-feature representation, where 0 < i ≤ n; Obtain the convolutional feature representations corresponding to the n sub-feature representations respectively; and concatenate the convolutional feature representations corresponding to the n sub-feature representations respectively to obtain the local feature representation of the first image.

8. The method according to any one of claims 1 to 4, characterized in that, The method is performed by an image segmentation model, which includes an encoder and a decoder; The fusion of the global feature representation and the local feature representation to obtain the fused feature representation includes: In the k-th encoding stage of the encoder, the global feature representation and the local feature representation of the k-th encoding stage are fused to obtain the fused feature representation of the k-th encoding stage, where k is a positive integer; The step of predicting the content of the first image based on the fused feature representation to obtain the segmentation prediction result of the first image content in the first image includes: In the q-th decoding stage of the decoder, the fused feature representation of the (m+1-q)-th encoding stage and the decoding feature representation of the (q-1)-th decoding stage are fused to obtain the encoding-decoding fused feature representation of the q-th decoding stage. Then, the encoding-decoding fused feature representation of the q-th decoding stage is decoded to obtain the decoding feature representation of the q-th decoding stage. Here, m is the number of decoding stages and q is a positive integer. Based on the decoding feature representation of m decoding stages, the content of the first image is predicted to obtain the segmentation prediction result of the first image content in the first image.

9. The method according to claim 8, characterized in that, In the q-th decoding stage of the decoder, the fused feature representations of the (m+1-q)-th encoding stage and the decoding feature representations of the (q-1)-th decoding stage are fused to obtain the encoding-decoding fused feature representation of the q-th decoding stage, including: Based on the channel mixing mode, the fusion feature representations of at least two coding stages are fused to obtain the coding fusion feature representation; The coded fusion feature representation is split into at least two coded feature representations, wherein the (m+1-q)th coding stage corresponds to the (m+1-q)th coded feature representation; By fusing the (m+1-q)th encoding feature representation and the (q-1)th decoding feature representation, the encoding-decoding fusion feature representation of the qth decoding stage is obtained.

10. The method according to any one of claims 1 to 4, characterized in that, Before fusing the global feature representation and the local feature representation to obtain the fused feature representation, the method further includes: Obtain the image feature representation of the first image, wherein the image feature representation includes at least two channel dimensions; By splitting the at least two channel dimensions, an identity feature representation containing at least one channel dimension is obtained; The fusion of the global feature representation and the local feature representation to obtain the fused feature representation includes: The fused feature representation is obtained by fusing the global feature representation, the local feature representation, and the identity feature representation.

11. The method according to any one of claims 1 to 4, characterized in that, The method is performed by an image segmentation model; The training process of the image segmentation model includes: Acquire a sample image, wherein the sample image is labeled with a reference content area corresponding to the content of the first image; The image segmentation model is used to predict the content of the first image in the sample image to obtain the predicted content region. The image segmentation model is trained based on the difference between the reference content region and the predicted content region.

12. The method according to claim 11, characterized in that, The step of predicting the first image content in the sample image using the image segmentation model to obtain the predicted content region includes: The image segmentation model is used to predict the first image content in the sample image, thereby obtaining the predicted probability that a pixel in the sample image belongs to the first image content. Based on the predicted probability, the pixels in the sample image that belong to the content of the first image are determined, and the predicted content region is obtained.

13. The method according to claim 11, characterized in that, Training the image segmentation model based on the difference between the reference content region and the predicted content region includes: A first loss value is obtained based on the pixel differences between the reference content region and the predicted content region. The image segmentation model is trained based on the first loss value.

14. The method according to claim 11, characterized in that, Training the image segmentation model based on the difference between the reference content region and the predicted content region includes: Obtain the number of intersection pixels between the reference content region and the predicted content region; Obtain the number of pixels in the union between the reference content region and the predicted content region; A second loss value is obtained based on the number of intersection pixels and the number of union pixels; The image segmentation model is trained based on the second loss value.

15. The method according to claim 11, characterized in that, Training the image segmentation model based on the difference between the reference content region and the predicted content region includes: Obtain the first edge pixel of the reference content region; and obtain the second edge pixel of the predicted content region; A third loss value is obtained based on the distance between the first edge pixel and the second edge pixel; The image segmentation model is trained based on the third loss value.

16. An image segmentation apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first image, which is an image to be segmented to obtain the content of the first image; A convolution module is used to convolve the first image with a convolution kernel of at least one size to generate a local feature representation of the first image; The generation module is used to generate a global feature representation of the first image through a recursive linear attention mechanism, wherein the global feature representation is used to express the global features of the first image in the spatial dimension. The fusion module is used to fuse the global feature representation and the local feature representation to obtain a fused feature representation; The prediction module is used to predict the content of the first image based on the fused feature representation, and to obtain the segmentation prediction result of the first image content in the first image.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the image segmentation method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image segmentation method as described in any one of claims 1 to 15.

19. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, which a processor reads from and executes to implement the image segmentation method as described in any one of claims 1 to 15.

Images