Image segmentation method, device and equipment and computer readable storage medium

Abstract

The invention provides an image segmentation method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-segmented image; performing content extraction processing on the to-be-segmented image to obtain a content feature vector of the to-be-segmented image; the content feature vector is used for representing feature information and semantic information of the to-be-segmented image; performing bidirectional updating on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; and according to the updated query vector and the output feature vector, performing image segmentation processing on the to-be-segmented image to obtain a segmentation result of the to-be-segmented image. According to the embodiment of the invention, the accuracy and robustness of image segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and relates to but is not limited to an image segmentation method, apparatus, device and computer-readable storage medium. Background Art

[0002] At present, query vectors are often used in target detection (Detection Transformer, DETR). Current general image segmentation algorithms use query vectors to achieve better segmentation effects, such as Mask2Former. When segmenting an image, the image segmentation algorithm will go through two steps: feature extraction and query vector update. Among them, the feature extraction part mainly uses the backbone network to extract the features of each pixel in the image; the query vector update part mainly uses the features in the feature extraction part to gradually transfer the semantic information to the randomly initialized query vector, thereby achieving the purpose of updating the query vector.

[0003] However, with the widespread use of mobile phones, when image segmentation algorithms are applied to mobile phones, the accuracy of image segmentation will be affected when a lightweight image segmentation model is used for image segmentation. Summary of the invention

[0004] The embodiments of the present application provide an image segmentation method, apparatus, device and computer-readable storage medium, which can improve the accuracy of image segmentation.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image segmentation method, the method comprising:

[0007] Obtain the image to be segmented;

[0008] Performing content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented;

[0009] bidirectionally updating the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector;

[0010] Image segmentation processing is performed on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

[0011] In a second aspect, an embodiment of the present application provides an image segmentation device, the image segmentation device comprising an acquisition unit, an extraction unit, an update unit and a segmentation unit; wherein:

[0012] The acquisition unit is used to acquire the image to be segmented;

[0013] The extraction unit is used to perform content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented;

[0014] The updating unit is used to perform bidirectional updating on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector;

[0015] The segmentation unit is used to perform image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, characterized in that it includes:

[0017] A memory for storing executable instructions;

[0018] The processor is used to implement the above-mentioned image segmentation method when executing the executable instructions stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores executable instructions. When the executable instructions are executed by one or more processors, the image segmentation method as described above is implemented.

[0020] The embodiment of the present application provides an image segmentation method, apparatus, device and computer-readable storage medium, the method comprising: first, obtaining an image to be segmented; then, performing content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; then, bidirectionally updating the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; finally, performing image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented. On the one hand, since the content feature vector is used to characterize the feature information and semantic information of the image to be segmented, updating the initial query vector according to the content feature vector having the feature information and semantic information can promote full interaction between the query vector and the feature information and semantic information of the image to be segmented, thereby improving the accuracy of image segmentation. On the other hand, the strategy of mutually updating the initial query vector and the content feature vector makes the image segmentation process more stable and reliable, reduces the possibility of error accumulation, and thus improves the accuracy and robustness of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0023] Figure 1 A schematic diagram of an optional image segmentation method provided in an embodiment of the present application Figure 1 ;

[0024] Figure 2 A schematic diagram of an optional image segmentation method provided in an embodiment of the present application Figure 2 ;

[0025] Figure 3 A system block diagram of an optional image segmentation method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of an optional image segmentation model provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of an optional image segmentation method provided in an embodiment of the present application Figure 3 ;

[0028] Figure 6 A schematic diagram of an optional image segmentation method provided in an embodiment of the present application Figure 4 ;

[0029] Figure 7 A schematic diagram of an optional visual representation to semantic representation model provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of an optional semantic representation to visual representation model provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of an optional bidirectional Transformer decoder module provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the structure of an optional image segmentation device provided in an embodiment of the present application;

[0033] Fig.11 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0035] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] Existing general image segmentation algorithms often use large parameters and computational complexity to achieve excellent segmentation results. For example, Mask2Former performs very well, but it is difficult to apply to resource-constrained mobile devices, such as embedded devices and mobile phones, which limits the application of the algorithm in practice.

[0038] In addition, some image segmentation algorithms for mobile applications, although reducing the number of model parameters and computational complexity, often sacrifice performance in other aspects, such as TopFormer. Although it has advantages in speed, it is limited to semantic segmentation tasks and does not support multi-task prediction. For instance segmentation and panoramic segmentation tasks, this model cannot be used to process them. In actual applications, complex scenes and multi-task requirements are often faced, which makes it impossible to apply existing mobile algorithms. Therefore, existing general image segmentation algorithms for mobile terminals still face challenges.

[0039] Almost all existing mobile algorithms use the Transformer structure, such as TopFormer. The application of the Transformer structure can achieve relatively good segmentation effects, but the key lies in how to reduce the amount of computation in the Transformer structure to achieve efficient operation on mobile devices. The usual approach is to reduce the amount of computation of the attention mechanism in the Transformer structure, while using a lightweight backbone network to reduce the overall complexity of the model. In other words, when the image segmentation algorithm is applied to the mobile phone, the accuracy of the image segmentation will be affected when a lightweight image segmentation model is used for image segmentation.

[0040] The image segmentation method provided in the embodiment of the present application is executed by an image segmentation device and an electronic device, wherein the image segmentation device can be stored in the electronic device in the form of a software function model, the image segmentation device can also be integrated in the electronic device as a hardware function module, and the image segmentation device can also be combined with the electronic device in software and hardware to implement the image segmentation method. The present application does not impose any limitations on this.

[0041] In an embodiment of the present application, the electronic device may be a server, which may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but the embodiments of the present application do not limit this.

[0042] In addition, the electronic device can also be a terminal device, which can be a mobile phone, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smart watch, a wearable device (WD) or an autonomous driving vehicle, etc., which is not limited to the embodiments of the present application.

[0043] The embodiment of the present application provides an image segmentation method, the basic idea of ​​which is: first, obtain an image to be segmented; then, perform content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; then, perform bidirectional update on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; finally, perform image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented. On the one hand, since the content feature vector is used to characterize the feature information and semantic information of the image to be segmented, updating the initial query vector according to the content feature vector having feature information and semantic information can promote full interaction between the query vector and the feature information and semantic information of the image to be segmented, thereby improving the accuracy of image segmentation. On the other hand, the strategy of mutually updating the initial query vector and the content feature vector makes the image segmentation process more stable and reliable, reduces the possibility of error accumulation, and thus improves the accuracy and robustness of image segmentation.

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0045] Figure 1 A schematic diagram of an optional model optimization method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, including S101 to S104:

[0046] S101, obtaining an image to be segmented.

[0047] In an embodiment of the present application, an electronic device obtains an image to be segmented.

[0048] In the embodiment of the present application, the image to be segmented is an object that needs to be segmented, and the image to be segmented includes multiple instances.

[0049] S102 , performing content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented.

[0050] In the embodiment of the present application, after acquiring the image to be segmented, the electronic device performs content extraction processing on the image to be segmented to obtain a content feature vector representing feature information and semantic information of the image to be segmented.

[0051] In an embodiment of the present application, the feature information of the image to be segmented can be represented by a global feature vector and / or a local feature vector. The global feature vector is a vector representation that encodes the features of the image to be segmented. The global feature vector covers the overall information of the image to be segmented and is used to represent the global properties and overall characteristics of the image. For example, for an image, the global feature vector can be represented as the overall statistical information, color distribution, texture features, etc. of the image.

[0052] In the embodiment of the present application, the local feature vector is a vector representation that encodes the features of the local area or subset of the image to be segmented, and the local feature vector focuses on part of the information of the image. For example, for an image, the local feature vector can be represented as the color, texture, edge and other features of a certain area of ​​the image.

[0053] In an embodiment of the present application, the semantic information of the image to be segmented can be represented by a global semantic vector, which can represent the high-level semantic content contained in the image, such as objects, scenes, relationships, etc., rather than just low-level pixel information.

[0054] In some embodiments of the present application, the content extraction process includes feature extraction process and semantic extraction process; the content feature vector includes a local feature vector and a global semantic vector.

[0055] In some embodiments of the present application, the local feature vector is used to characterize the features of each pixel in the image to be segmented, and the features of different scales contain the details and structural information of the image. The global semantic vector is used to characterize the semantic information in the image to be segmented, which facilitates the understanding of the entire image.

[0056] In some embodiments of the present application, Figure 2 As shown, the implementation of performing content extraction processing on the image to be segmented in S102 to obtain the content feature vector of the image to be segmented may include S1021 to S1022:

[0057] S1021, performing feature extraction processing on the image to be segmented to obtain a local feature vector;

[0058] S1022. Perform semantic extraction processing on the local feature vector to obtain a global semantic vector.

[0059] In the embodiment of the present application, the electronic device performs feature extraction processing on the image to be segmented through a feature pyramid module to obtain a local feature vector of the image to be segmented.

[0060] In the embodiment of the present application, the electronic device performs semantic extraction processing on the local feature vector through a semantic extraction module to obtain a global semantic vector of the image to be segmented.

[0061] In the embodiment of the present application, the feature pyramid module is a backbone network part, which is used to perform feature extraction processing on the input image.

[0062] In an embodiment of the present application, the semantic extraction module is used to utilize features (local feature vectors) of different scales output by the backbone network part (feature pyramid module) to obtain a global semantic vector from features of different scales.

[0063] S103 . Perform bidirectional updates on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector.

[0064] In the embodiment of the present application, after the electronic device obtains the content feature vector of the image to be segmented, the electronic device performs bidirectional updates on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector.

[0065] In the embodiment of the present application, the electronic device bidirectionally updates the initial query vector and the content feature vector through a bidirectional Transformer decoder module to obtain an updated query vector and an output feature vector.

[0066] In an embodiment of the present application, a bidirectional Transformer decoder module is used to mutually update a first input vector (initial query vector) and a second input vector (content feature vector), thereby promoting full interaction between the query vector and the feature information and semantic information of the image to be segmented, thereby improving the accuracy of image segmentation.

[0067] It can be understood that by performing bidirectional updates on the initial query vector and the content feature vector, the communication between the initial query vector and the content feature vector can be enhanced, thereby accelerating the iterative convergence of the network and thus improving the training efficiency and accuracy of the model.

[0068] In the embodiment of the present application, the initial query vector may be a randomly generated vector, or a vector obtained according to a content feature vector.

[0069] In some embodiments of the present application, feature mapping processing is performed according to semantic information in a content feature vector to obtain an initial query vector.

[0070] It can be understood that the initial query vector can be initialized by utilizing the semantic information (i.e., semantic features) in the content feature vector. In this way, rich semantic embedding information can be introduced into the initial query vector, thereby reducing the iterative update burden of the bidirectional Transformer decoder, thereby accelerating the convergence of the network model and improving the training efficiency of the image segmentation model.

[0071] S104 . Perform image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

[0072] In the embodiment of the present application, the electronic device performs image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector through the segmentation head part to obtain a segmentation result of the image to be segmented.

[0073] In the embodiment of the present application, the segmentation head part is used to realize object recognition and segmentation.

[0074] In the embodiment of the present application, the segmentation head part can be: a full convolutional segmentation head (Fully Convolutional Head), a transpose convolution segmentation head (Transpose Convolution Head), a self-attention segmentation head (Self-Attention Head), a multi-scale segmentation head (Multi-Scale Head), etc. The embodiment of the present application does not impose any limitation on the segmentation head part, and the specific selection can be made according to the actual application scenario.

[0075] In some embodiments of the present application, image segmentation is performed on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented, including:

[0076] The updated query vector is subjected to data classification processing through a preset classification model to obtain a category vector corresponding to the updated query vector; wherein the preset classification model is used to classify according to the features of the input vector;

[0077] Determine a mask feature vector based on the updated query vector and the output feature vector;

[0078] According to the mask feature vector and the category vector, image segmentation of the image to be segmented is achieved, thereby obtaining a segmentation result of the image to be segmented.

[0079] In the embodiment of the present application, the category vector is used to represent the label for classifying the pixels in the image to be segmented. A category label can be assigned to each pixel in the image to be segmented, indicating which class of object the pixel belongs to. For example, the category label can be "car", "person", "tree", etc. The category label can also distinguish different instances, such as: "the first car", "the second person", etc. The category vector is usually represented by integer encoding or one-hot encoding. The embodiment of the present application does not impose any limitation on the representation method of the category vector.

[0080] In the embodiment of the present application, the mask feature vector is a vector representation for marking the pixels in the image to be segmented, which is used to indicate whether each pixel belongs to the target object (segmentation result). The mask feature vector is usually represented by pixel-level binary, where a pixel value of 1 indicates that the pixel belongs to the segmentation result, and a pixel value of 0 indicates that the pixel does not belong to the segmentation result. It can be seen that the mask feature vector can accurately indicate the position and shape of the segmentation result in the image to be segmented.

[0081] In some embodiments of the present application, determining a mask feature vector according to an updated query vector and an output feature vector includes:

[0082] Perform dot multiplication on the updated query vector and the output feature vector to obtain an intermediate mask feature vector;

[0083] The intermediate mask feature vector is optimized through a preset optimization model to obtain a mask feature vector; wherein the preset optimization model is used to optimize the edge portion and the texture portion of the input vector.

[0084] In the embodiment of the present application, it is considered that the image segmentation method provided in the embodiment of the present application is for the instance segmentation task and the semantic segmentation task of the image. For the instance segmentation task, the quality requirement of the mask feature vector is higher, so it is possible to consider adding a small network (i.e., a preset optimization model) that optimizes the mask quality to further refine the edge of the instance mask feature vector. For simple semantic segmentation tasks, the preset optimization model is not required to refine the mask features, so as to achieve the purpose of optimizing the segmentation mask quality without increasing the amount of calculation as much as possible.

[0085] For example, Figure 3 A system block diagram of an optional image segmentation method provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the image segmentation model mainly includes: feature pyramid module, semantic extraction module, bidirectional Transformer decoder module, and segmentation head. First, the feature pyramid module is used to extract features from the image to be segmented (input image) to obtain a local feature vector; then, the semantic extraction module is used to extract semantics from the local feature vector to obtain a global semantic vector; then, the bidirectional Transformer decoder module is used to update the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; finally, the segmentation head is used to segment the image to be segmented according to the updated query vector and the output feature vector to obtain the segmentation result of the image to be segmented.

[0086] In some embodiments of the present application, the image segmentation process includes: at least one of semantic segmentation process, instance segmentation process and panorama segmentation process; the segmentation result includes: at least one of semantic segmentation result, instance segmentation result and panorama segmentation result, and the above method also includes at least one of the following:

[0087] According to the updated query vector and the output feature vector, semantic segmentation processing is performed on the image to be segmented to obtain a semantic segmentation result of the image to be segmented;

[0088] According to the updated query vector and the output feature vector, the image to be segmented is processed for instance segmentation to obtain an instance segmentation result of the image to be segmented;

[0089] According to the updated query vector and the output feature vector, a panoramic segmentation process is performed on the image to be segmented to obtain a panoramic segmentation result of the image to be segmented.

[0090] In an embodiment of the present application, first, an image to be segmented is acquired; then, a content extraction process is performed on the image to be segmented to obtain a content feature vector of the image to be segmented; subsequently, the initial query vector and the content feature vector are bidirectionally updated to obtain an updated query vector and an output feature vector; finally, according to the updated query vector and the output feature vector, an image segmentation process is performed on the image to be segmented to obtain a segmentation result of the image to be segmented. On the one hand, since the content feature vector is used to characterize the feature information and semantic information of the image to be segmented, updating the initial query vector according to the content feature vector having the feature information and semantic information can promote full interaction between the query vector and the feature information and semantic information of the image to be segmented, thereby improving the accuracy of image segmentation. On the other hand, the strategy of mutually updating the initial query vector and the content feature vector makes the image segmentation process more stable and reliable, reduces the possibility of error accumulation, and thus improves the accuracy and robustness of image segmentation.

[0091] In some embodiments of the present application, the feature extraction process includes M convolution processes, and the local feature vector includes M first feature vectors; wherein M is a positive integer greater than or equal to 1.

[0092] In the embodiment of the present application, the feature extraction process includes M convolution processes, for example, M=4.

[0093] In the embodiment of the present application, the convolution process is a downsampling process.

[0094] In the embodiment of the present application, when the feature extraction process includes M convolution processes, the feature pyramid module includes at least M convolution layers.

[0095] The following is some explanation of the convolution layer. The convolution layer can include many convolution operators, which are also called kernels. Their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can essentially be a feature vector. In the process of convolution operation on the image, the feature vector is usually processed one pixel after another along the horizontal direction on the input image to complete the work of extracting specific features from the image. The size of the feature vector should be related to the size of the image. It should be noted that the depth dimension of the feature vector is the same as the depth dimension of the input image. In the process of convolution operation, the feature vector will extend to the entire depth of the input image. Therefore, convolution with a single feature vector will produce a convolution output with a single depth dimension, but in most cases, a single feature vector is not used, but multiple feature vectors of the same size (row × column), that is, multiple isomorphic matrices, are used. The values ​​in these feature vectors need to be obtained through a lot of training in practical applications. The feature vectors obtained through training can be used to extract information from the input image, so that the network model can make correct predictions.

[0096] In the embodiment of the present application, the step size corresponding to each convolution process can be 1, 2, 3, 5, etc. The embodiment of the present application does not impose any limitation on the step size of the convolution process.

[0097] Exemplarily, in the embodiment of the present application, the step size of the convolution processing is 2.

[0098] In the embodiment of the present application, the size of the convolution kernel corresponding to each convolution process can be 1, 2, 3, 5, etc. The embodiment of the present application does not impose any limitation on the size of the convolution kernel.

[0099] Exemplarily, in the embodiment of the present application, the size of the convolution kernel is 3.

[0100] In the embodiment of the present application, the sizes of the M first eigenvectors decrease in sequence. That is, the size of the i+1th first eigenvector is smaller than the size of the ith first eigenvector.

[0101] In some embodiments of the present application, the implementation of performing feature extraction processing on the image to be segmented to obtain a local feature vector in S1021 may include:

[0102] Perform the i-th convolution process on the image to be segmented to obtain the i-th first eigenvector; where i is a positive integer less than M;

[0103] Continue to perform the i+1th convolution process on the i-th first eigenvector to obtain the i+1th first eigenvector, until i+1 is equal to M, and obtain M first eigenvectors obtained by M convolution processes; wherein the size of the i+1th first eigenvector is smaller than the size of the i-th first eigenvector.

[0104] In the embodiment of the present application, the feature pyramid module includes at least M convolutional layers, and the i+1th convolutional layer processes the i-th first feature vector to obtain the i+1th first feature vector. The i-th first feature vector is the output vector of the i-th convolutional layer.

[0105] In an embodiment of the present application, the convolution layer in the feature pyramid module sequentially performs convolution processing on the output vector of the previous convolution layer to obtain first feature vectors of different scales.

[0106] It can be understood that by performing M convolutions on the image to be segmented, M first feature vectors of different sizes are obtained. In this way, through a series of feature vectors of different scales, multi-scale details and structural information in the image to be segmented can be captured. When the initial query vector is updated using the multi-scale local feature vectors, the update efficiency and accuracy can be improved, thereby improving the accuracy of image segmentation.

[0107] In some embodiments of the present application, the global semantic vector includes M second feature vectors; wherein M is a positive integer greater than or equal to 1.

[0108] In an embodiment of the present application, the semantic extraction module includes at least one pooling layer, and the pooling layer is used to perform pooling processing on the input vector; wherein the pooling processing includes average pooling processing and / or maximum pooling processing.

[0109] The following is some explanation of the pooling layer. Since it is often necessary to reduce the number of training parameters, the pooling layer often needs to be periodically introduced after the convolution layer. The pooling layer can also be called a downsampling layer. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0110] In the embodiment of the present application, the number of the first feature vectors is consistent with the number of the second feature vectors, so that the first feature vectors and the corresponding second feature vectors can be subjected to subsequent feature fusion processing.

[0111] In some embodiments of the present application, the implementation of performing semantic extraction processing on the local feature vector to obtain the global semantic vector in S1022 may include:

[0112] Performing pooling processing on the M first feature vectors in the local feature vectors respectively to obtain M first intermediate feature vectors;

[0113] According to the channel dimension, the M first intermediate feature vectors are concatenated to obtain a second intermediate feature vector;

[0114] The second intermediate feature vector is semantically extracted through at least one global feature interaction model to obtain a global semantic vector; wherein the global feature interaction model includes at least a multi-head attention model and a feedforward network model, the multi-head attention model is used to extract multi-level context information from the input vector, and the feedforward network model is used to perform nonlinear transformation processing on the input vector.

[0115] In the embodiment of the present application, in order to reduce the computational complexity of the semantic extraction module, the M first feature vectors obtained by the feature pyramid module are respectively pooled to obtain M first intermediate feature vectors.

[0116] In an embodiment of the present application, after obtaining M first intermediate feature vectors, the M first intermediate feature vectors are concatenated in the channel dimension to obtain a second intermediate feature vector.

[0117] Furthermore, after obtaining the second intermediate feature vector, semantic extraction processing is performed on the second intermediate feature vector through at least one global feature interaction model (ie, Transformer block) to obtain a global semantic vector, wherein the global semantic vector includes M second feature vectors.

[0118] In the embodiment of the present application, the global feature interaction model is also referred to as a Transformer module or a Transformer model.

[0119] In this embodiment of the present application, the number of global feature interaction models is 4.

[0120] It should be noted that the Transformer model is a deep learning architecture that is commonly used for natural language processing tasks such as machine translation. The Transformer model introduces the self-attention mechanism to process sequence data. It has the advantages of strong parallel computing capabilities and capturing long-distance dependencies. It has gradually been widely used in computer vision, speech processing and other fields. The core idea of ​​the Transformer model is to divide the input data into multiple representations, and then use the self-attention mechanism to calculate the attention weight at each position, so as to capture the relationship between different positions in the entire sequence. This mechanism allows the model to consider global context information while processing the sequence.

[0121] In the embodiment of the present application, the global feature interaction model (Transformer model) includes at least a multi-head attention model and a feed-forward neural network model; wherein the multi-head attention model calculates multiple self-attentions in parallel, then concatenates and linearly transforms them, thereby capturing information from different representation spaces, and then extracting multi-level context information from the input vector. The feed-forward neural network model is used to perform nonlinear transformations on features at each position of the input vector.

[0122] It should be noted that the global feature interaction model also includes other components, such as:

[0123] 1) Embedding Layer: Convert the elements in the input sequence (such as words, characters, pixels, etc.) into vector representations. For image data, position embedding and feature embedding can be used;

[0124] 2) Encoder: The encoder is composed of multiple encoder layers with the same structure. Each encoder layer contains a multi-head self-attention mechanism and a feedforward neural network. The encoder is responsible for converting the input sequence into a series of context-aware feature representations.

[0125] 3) Decoder: The decoder is also composed of multiple decoder layers with the same structure. In addition to the self-attention mechanism and feedforward neural network, each decoder layer also includes an encoder-decoder attention mechanism to focus on the information in the input sequence;

[0126] 4) Self-Attention Mechanism: The self-attention mechanism calculates the attention weights of each position in the sequence to other positions, and then applies these weights to the features of the corresponding positions. This allows the model to consider global dependencies at the same time;

[0127] 5) Residual connection and normalization: Residual connection and normalization are used in Transformer to accelerate training convergence and improve model performance.

[0128] It should be noted that the components of the global feature interaction model in the embodiment of the present application are only used as an example, and the embodiment of the present application does not impose any limitation on this.

[0129] It can be understood that by performing pooling processing on the M first feature vectors to obtain M first intermediate feature vectors, the computational complexity of the semantic extraction module can be effectively reduced, thereby improving the efficiency of model training.

[0130] For example, Figure 4 A schematic diagram of the structure of an optional image segmentation model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the image segmentation model mainly includes: a feature pyramid module, a semantic extraction module, and a bidirectional Transformer decoder module. It can be seen that the feature pyramid module includes 4 convolutional layers, each of which is used to downsample the input vector, and finally obtains 4 first feature vectors (i.e., local feature vectors). The sizes of these 4 first feature vectors are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the size of the image to be segmented, respectively. Then, these 4 first feature vectors are pooled respectively to obtain 4 first intermediate feature vectors. The size of these 4 first intermediate feature vectors is 1 / 64 of the size of the image to be segmented. Then, these 4 first intermediate feature vectors are spliced ​​according to the channel dimension to obtain the second intermediate feature vector. Subsequently, the second intermediate feature vector is passed through at least one global feature interaction model (Transformer module) for semantic extraction processing, thereby obtaining a global semantic vector including 4 second feature vectors. Furthermore, the initial query vector is updated through the bidirectional Transformer decoder according to the four first feature vectors output by the feature pyramid module and the four second feature vectors output by the semantic extraction module to obtain an updated query vector and an output feature vector. Finally, the category vector of the image to be segmented is determined according to the updated query vector, and the updated query vector and the output feature vector are multiplied to obtain the mask feature vector of the image to be segmented. Then, the image to be segmented is segmented according to the mask feature vector and the category vector to obtain the segmentation result of the image to be segmented (i.e., the segmentation result).

[0131] In some embodiments of the present application, when the content feature vector includes a local feature vector and a global semantic vector, the implementation of bidirectionally updating the initial query vector and the content feature vector in S103 to obtain the updated query vector and the output feature vector may include:

[0132] The local feature vector and the global semantic vector are subjected to feature fusion processing to obtain a fused feature vector, and the fused feature vector and the initial query vector are bidirectionally updated to obtain an updated query vector and an output feature vector.

[0133] In an embodiment of the present application, the local feature vector includes M first feature vectors, and the global semantic vector includes M second feature vectors. When the local feature vector and the global semantic vector are subjected to feature fusion processing, the first feature vector and the corresponding second feature vector may be subjected to feature fusion processing. It should be noted that the sizes of the first feature vector and the corresponding second feature vector are consistent.

[0134] In the embodiment of the present application, feature fusion processing is performed on the local feature vector and the global semantic vector through a feature fusion module (model), and the feature fusion module is also called an Injector module.

[0135] It can be understood that by performing feature fusion processing on the local feature vector and the global semantic vector, the fused feature vector has the feature information and semantic information of the segmented image to be updated. In this way, when the query vector is updated using the fused feature vector, the feature information and semantic information of the image can be fully understood, thereby improving the learning ability of the model and further improving the accuracy and efficiency of image segmentation.

[0136] In some embodiments of the present application, Figure 5 As shown, the local feature vector and the global semantic vector are subjected to feature fusion processing to obtain a fused feature vector, and the fused feature vector and the initial query vector are bidirectionally updated to obtain an updated query vector and an output feature vector, which may include S201 to S204:

[0137] S201. Perform feature fusion processing on the i-th first feature vector in the local feature vector and the i-th second feature vector corresponding to the i-th first feature vector in the global semantic vector to obtain the i-th fused feature vector; wherein i is a positive integer greater than or equal to 1 and less than M.

[0138] In some embodiments of the present application, feature fusion processing includes: convolution processing, normalization processing, data mapping processing and weighted summation processing.

[0139] In an embodiment of the present application, the feature fusion module includes a convolution layer, a normalization layer, a data mapping processing layer and a weighted layer.

[0140] In the embodiment of the present application, the data mapping processing layer is used to perform a Sigmoid operation on the input vector, wherein the Sigmoid function is used to map the value of the input vector to between 0 and 1.

[0141] It is understandable that the Sigmoid operation can solve the problems of gradient disappearance and gradient explosion, and can improve the training efficiency and accuracy of the image segmentation model.

[0142] In some embodiments of the present application, the implementation of performing feature fusion processing on the i-th first feature vector in the local feature vector and the i-th second feature vector corresponding to the i-th first feature vector in the global semantic vector in S201 to obtain the i-th fused feature vector may include:

[0143] Performing convolution processing and normalization processing on the i-th first eigenvector and the i-th second eigenvector respectively to obtain a current fourth output vector of the i-th first eigenvector and a current fifth output vector of the i-th second eigenvector;

[0144] Performing data mapping processing on the current fifth output vector to obtain the current sixth output vector of the i-th second feature vector; the current sixth output vector represents the semantic weight of the i-th second feature vector;

[0145] Multiply the current sixth output vector and the current fourth output vector to obtain the current seventh output vector of the i-th first eigenvector;

[0146] Performing convolution processing and normalization processing on the i-th second eigenvector to obtain a current eighth output vector of the i-th second eigenvector;

[0147] The current seventh output vector and the current eighth output vector are weightedly summed to obtain the i-th fused feature vector.

[0148] In the embodiment of the present application, first, the electronic device performs convolution processing with a convolution kernel of 1×1 and batch normalization processing on the i-th first feature vector and the i-th second feature vector respectively, and obtains the current fourth output vector of the i-th first feature vector and the current fifth output vector of the i-th second feature vector; then, the current fifth output vector continues to be processed by Sigmoid to obtain the current sixth output vector representing the semantic weight of the i-th second feature vector; further, the current sixth output vector and the current fourth output vector are correspondingly multiplied to obtain the current seventh output vector of the i-th first feature vector; in addition, the i-th second feature vector is subjected to convolution processing with a convolution kernel of 1×1 and batch normalization processing to obtain the current eighth output vector; finally, the current seventh output vector and the current eighth output vector are weighted summed to obtain the i-th fused feature vector. At this point, the feature fusion processing of the i-th first feature vector in the local feature vector and the i-th second feature vector in the global semantic vector is realized.

[0149] It is understandable that due to the semantic gap between the global semantic vector and the local feature vector, the two cannot be directly used to update the query vector. Therefore, by fusing the global semantic vector and the local feature vector, the obtained fused feature vector contains rich semantic information, feature information and detail information, which helps to generate more accurate image segmentation results.

[0150] S202 . Perform bidirectional updates on the i th fused feature vector and the current query vector to obtain a current updated query vector and a current first output vector.

[0151] In some embodiments of the present application, Figure 6 As shown, in S202, the i-th fused feature vector and the current query vector are updated bidirectionally to obtain the current updated query vector and the current first output vector, which may include S2021 to S2022:

[0152] S2021. Update the current query vector according to the i-th fused feature vector through the visual representation to semantic representation model to obtain the current updated query vector; the visual representation to semantic representation model is used to convert the visual information of the input vector into a vector representation with semantic meaning.

[0153] In the embodiment of the present application, the visual representation to semantic representation model is also referred to as a visual representation to semantic representation module (Semantic to Visual Block, S2V).

[0154] In an embodiment of the present application, the visual representation to semantic representation model is used to update the current query vector using the feature information and semantic information of the i-th fused feature vector, so that the obtained current updated query vector can represent the semantic information of the image.

[0155] In some embodiments of the present application, the visual representation to semantic representation model includes: a cross-attention model, a self-attention model and a feedforward network model; the cross-attention model is used to perform correlation feature analysis on different input sequences; the self-attention model is used to perform correlation feature analysis on each element in the input sequence.

[0156] In some embodiments of the present application, in S2021, the current query vector is updated according to the i-th fused feature vector through the visual representation to semantic representation model to obtain the current updated query vector, which may include S301 to S306:

[0157] S301. Perform correlation feature analysis on the i-th fused feature vector and the current query vector through a cross attention model to obtain a first query vector.

[0158] In an embodiment of the present application, the cross-attention model introduces a cross-attention mechanism, and the core of the cross-attention mechanism lies in the query vector (Query, Q), the key vector (Key, K) and the value vector (Value, V). Among them, the Q vector is used to calculate the vector of the attention mechanism weight, which is usually a representation of a specific position or feature that needs to be paid attention to. The Q vector and the K vector perform similarity calculations to obtain the attention weight of each position for the specific position of attention. The K vector contains the information of the original features, and the K vector and the Q vector perform similarity calculations to obtain the correlation between features at different positions. The V vector is used to store a vector of special diagnostic information, which contains the original feature information and is used to calculate weights in the attention mechanism.

[0159] It can be understood that in the image segmentation task, the cross-attention model can effectively capture the correlation information between different regions in the image and use this correlation information to improve the accuracy of image segmentation. The cross-attention model can effectively interact and transfer information between different parts of the image, thereby improving the quality of the image segmentation results.

[0160] S302: Add the first query vector and the current query vector to obtain a second query vector, and normalize the second query vector to obtain a third query vector.

[0161] S303: Perform correlation feature analysis on the third query vector through a self-attention model to obtain a fourth query vector.

[0162] In an embodiment of the present application, a self-attention mechanism is introduced into the self-attention model, which can help the network model focus on local information inside the image and capture the association with surrounding pixels. In this way, the network model can better understand the overall semantic information of the image while retaining details.

[0163] S304: Add the fourth query vector and the third query vector to obtain a fifth query vector, and normalize the fifth query vector to obtain a sixth query vector.

[0164] S305 . Perform nonlinear transformation processing on the sixth query vector through a feedforward network model to obtain a seventh query vector.

[0165] S306 . Add the seventh query vector and the sixth query vector to obtain an eighth query vector, and normalize the eighth query vector to obtain a current updated query vector.

[0166] For example, Figure 7 A schematic diagram of an optional visual representation to semantic representation model provided in an embodiment of the present application, such as Figure 7As shown, the visual representation to semantic representation model mainly includes: a cross attention model, a self-attention model and a feedforward network model. First, the i-th fused feature vector and the current query vector are subjected to feature conversion processing to obtain the key vector (K) and value vector (V) of the i-th fused feature vector, and the query vector (Q) of the current query vector; wherein the K vector represents the degree of association between the features in the i-th fused feature vector, the V vector represents the feature information in the i-th fused feature vector; and the Q vector represents the key information of the feature information in the current query vector. Then, through the cross attention model, the K vector and value vector of the i-th fused feature vector, and the Q vector of the current query vector are subjected to correlation feature analysis to obtain the first query vector. Further, the first query vector and the current query vector are added and normalized to obtain the third query vector. Subsequently, through the self-attention model, the key vector (K), value vector (V) and query vector (Q) of the third query vector are subjected to correlation feature analysis to obtain the fourth query vector. Further, the fourth query vector and the third query vector are added and normalized to obtain the sixth query vector. Then, the sixth query vector is subjected to nonlinear transformation processing through the feedforward network model to obtain a seventh query vector. Finally, the seventh query vector and the sixth query vector are added and normalized to complete the update of the current query vector, thereby obtaining a current updated query vector.

[0167] S2022. Update the i-th fused feature vector according to the current updated query vector through the semantic representation to visual representation model to obtain the current first output vector; the semantic representation to visual representation model is used to convert the semantic information of the input vector into a vector representation with visual meaning.

[0168] In the embodiment of the present application, the semantic representation to visual representation model is also referred to as a semantic representation to visual representation module (Visual to Semantic Block, V2S).

[0169] In an embodiment of the present application, the semantic representation to visual representation model is used to guide the update of the i-th fused feature vector using the semantic information of the currently updated query vector.

[0170] In some embodiments of the present application, the semantic representation to visual representation model includes: a cross-attention model.

[0171] In some embodiments of the present application, the implementation of updating the i-th fused feature vector according to the current updated query vector through the semantic representation to the visual representation model in S2022 to obtain the current first output vector may include S401 to S405:

[0172] S401. Perform correlation feature analysis on the current updated query vector and the i-th fused feature vector through a cross attention model to obtain a ninth query vector.

[0173] S402: Perform convolution processing and nonlinear transformation processing on the ninth query vector to obtain a tenth query vector.

[0174] In the application embodiment, when performing convolution processing, the parameters such as the convolution kernel size and step size used are not limited. For example, the convolution kernel size is 1×1.

[0175] In the application embodiment, when performing nonlinear conversion processing, the Relu activation function can be used, which can effectively solve the problem of gradient disappearance that may occur in model training, making the model easier to train and obtaining better model performance.

[0176] S403 , performing convolution processing and nonlinear transformation processing on the i th fused feature vector to obtain a first intermediate feature vector.

[0177] S404: Multiply the tenth query vector and the intermediate feature vector to obtain a second intermediate feature vector.

[0178] S405 . Perform convolution processing and nonlinear transformation processing on the second intermediate feature vector to obtain a current first output vector.

[0179] For example, Figure 8 A schematic diagram of an optional semantic representation to visual representation model provided in an embodiment of the present application, such as Figure 8 As shown, the semantic representation to visual representation model mainly includes: a cross-attention model. First, the i-th fused feature vector and the current query vector are subjected to feature conversion processing to obtain the query vector (Q) of the i-th fused feature vector, as well as the key vector (K) and value vector (V) of the current query vector. Subsequently, the Q of the i-th fused feature vector, as well as the K vector and V vector of the current query vector are subjected to correlation feature analysis through the cross-attention model to obtain the ninth query vector. Further, the ninth query vector is subjected to convolution processing and nonlinear transformation processing to obtain the tenth query vector. At the same time, the i-th fused feature vector is subjected to convolution processing and nonlinear transformation processing to obtain the first intermediate feature vector. Then, the tenth query vector and the intermediate feature vector are multiplied to obtain the second intermediate feature vector. Finally, the second intermediate feature vector is subjected to convolution processing and nonlinear transformation processing to complete the update of the i-th fused feature vector, thereby obtaining the current first output vector.

[0180] It can be understood that by updating the i-th fused feature vector and the current query vector, the query vector and the semantic feature vector are fully interacted with each other. The mutual updating strategy makes the prediction process more stable and reliable, reduces the possibility of error accumulation, and improves the accuracy and robustness of the prediction.

[0181] S203 . Determine an i+1 th fused feature vector based on the current first output vector and the i+1 th second feature vector in the global semantic vector corresponding to the i+1 th first feature vector.

[0182] In some embodiments of the present application, the implementation of determining the i+1th fused feature vector based on the current first output vector and the i+1th second feature vector corresponding to the i+1th first feature vector in the global semantic vector in S203 may include:

[0183] Performing upsampling processing on the current first output vector to obtain a current second output vector;

[0184] Add the current second output vector and the (i+1)th first eigenvector to obtain a current third output vector;

[0185] The current third output vector and the (i+1)th second feature vector are subjected to feature fusion processing to obtain an (i+1)th fused feature vector.

[0186] In an embodiment of the present application, in order to utilize multi-scale image features, after obtaining the current first output vector, the current first output vector is upsampled and then added to the i+1th first feature vector in the local feature vector as the current third output vector. Further, feature fusion processing is performed based on the current third output vector and the i+1th second feature vector to obtain the i+1th fused feature vector, thereby realizing iterative update of the query vector and semantic features, thereby improving the accuracy and robustness of the image segmentation model.

[0187] S204. Continue to perform bidirectional updating on the current updated query vector according to the i+1th fused feature vector to obtain the next updated query vector and the next first output vector, until i+1 is equal to M, and use the next updated query vector as the updated query vector and the next first output vector as the output feature vector.

[0188] In an embodiment of the present application, M first feature vectors in the local feature vector and M second feature vectors in the global semantic vector are used to implement cyclic bidirectional updating of the query vector and the fused feature vector, thereby improving the accuracy of the updated query vector and the output feature vector, thereby improving the accuracy and robustness of the image segmentation model.

[0189] For example, Fig. 9 A schematic diagram of an optional bidirectional Transformer decoder module provided in an embodiment of the present application, such as Fig. 9 As shown in the figure, the bidirectional Transformer decoder module mainly includes: feature fusion module (model), S2V module (visual representation to semantic representation model), V2S module (semantic representation to visual representation model). First, the local feature vector and the global semantic vector are subjected to feature fusion processing through the feature fusion module. Then, the initial query vector is updated by the fused features (fused feature vector) through the S2V module, and the fused feature vector with semantic information is optimized in turn through the V2S module using the updated query vector, and the update is repeated iteratively. The updated query vector finally obtained can be used for category prediction, thereby realizing the distinction between different targets in the image. In this way, the initial query vector and the fused feature vector can fully interact with each other, and the mutual updating strategy makes the prediction process more stable and reliable, reduces the possibility of error accumulation, improves the accuracy and robustness of the prediction, and thus improves the accuracy of the image segmentation model.

[0190] Exemplarily, the specific processing steps of the bidirectional Transformer decoder module may include:

[0191] 1) The current query vector and the fused feature vector are first passed through the s2v module in the bidirectional Transformer decoder. This module uses the rich semantic and detail information in the fused feature vector to update the current query vector and obtain the current updated query vector, so that the current updated query vector gradually contains image semantic information and local feature information. Secondly, the current updated query vector in turn updates the fused feature vector, and the v2s module is used to use the semantic information in the current updated query vector to guide the update of the fused feature vector and obtain the current output feature vector;

[0192] 2) In order to utilize multi-scale image features, the current output feature vector is added to the local feature vector output by the feature pyramid module after upsampling as a new local vector, and the feature fusion process is repeated to achieve iterative update of the query vector and semantic features.

[0193] by Figure 4 Taking the image segmentation model shown as an example, in order to balance the computational complexity and segmentation accuracy of the network, the local feature vector only utilizes 1 / 32, 1 / 16, and 1 / 8 of the features and is fused with the global semantic vector. The output feature vector is generated by adding the feature vector updated by the last layer v2s module and the local feature vector, and then fusing it with the global semantic vector through the feature fusion module.

[0194] In the embodiment of the present application, the backbone network (feature pyramid module) adopts a lightweight structure and uses a bidirectional Transformer decoder to achieve universal image segmentation on mobile devices while providing better segmentation effects.

[0195] It is understandable that in the embodiment of the present application, the image feature vectors of each scale of the image to be segmented are obtained through the feature pyramid module, including local feature vectors and global semantic vectors; the bidirectional Transformer decoder also speeds up the update of the image segmentation model and effectively avoids the problem of excessive calculation of the model. The embodiment of the present application can realize lightweight and effective general image segmentation, solving the limitations of the prior art.

[0196] In some embodiments of the present application, the segmentation result of the image to be segmented is obtained by performing image segmentation on the image to be segmented by an image segmentation model, and the image segmentation model is obtained by training an initial image segmentation model. The method further includes:

[0197] Acquire sample data; the sample data includes original image samples and instance samples corresponding to the original image samples;

[0198] Perform image segmentation on the original image sample through the initial image segmentation model to obtain candidate results of the original image sample;

[0199] Input the candidate results of the original image sample and the corresponding instance sample into the preset loss function to obtain the current loss result;

[0200] The model parameters in the initial image segmentation model are modified according to the current loss result to obtain the current intermediate image segmentation model;

[0201] Continue to perform image segmentation on the original image sample through the current intermediate image segmentation model to obtain the next loss result, and modify the model parameters in the current intermediate image segmentation model according to the next loss result to obtain the next intermediate image segmentation model, until the next loss result corresponding to the next intermediate image segmentation model meets the cutoff condition, and use the next intermediate image segmentation model as the image segmentation model.

[0202] In an embodiment of the present application, the preset loss function is pre-set. Exemplarily, the preset loss function can be the pixel difference between the candidate result of the original image sample and the corresponding instance sample. The embodiment of the present application does not impose any limitation on the preset loss function.

[0203] In an embodiment of the present application, the cutoff condition may include at least one of the following: the loss result is less than or equal to a threshold, and the number of iterations is met. The embodiment of the present application does not impose any limitation on the cutoff condition.

[0204] In some embodiments of the present application, the implementation of performing image segmentation on the original image sample by using the initial image segmentation model to obtain the candidate result of the original image sample may include:

[0205] Through the initial image segmentation model, the original image sample is subjected to content extraction processing to obtain the content feature vector of the original image sample;

[0206] Bidirectionally updating the initial query vector and the content feature vector of the original image sample to obtain an updated query vector and an output feature vector of the original image sample;

[0207] According to the updated query vector and the output feature vector, the original image sample is segmented to obtain the candidate result of the original image sample.

[0208] The image segmentation method provided in the embodiment of the present application is explained in detail in a specific embodiment below.

[0209] The image segmentation method of the embodiment of the present application is implemented by an image segmentation model, wherein the image segmentation model includes: a feature pyramid module, a semantic extraction module, a bidirectional Transformer decoder module and a segmentation head part. Through the collaboration of these three modules, the versatility and application scope of the model are expanded while reducing the number of model parameters and the amount of calculation, thereby achieving efficient calculation.

[0210] Feature pyramid module: that is, the backbone network part. The embodiment of the present application utilizes the advantages of TopFormer, that is, the structure of the backbone network part, to simplify the MobileNetv2 model to extract the features of each pixel in the input image (image to be segmented). These features of different scales contain details and structural information.

[0211] Semantic extraction module: In the embodiment of the present application, the semantic extraction module is composed of four stacked Transformer modules (global feature interaction model). These four Transformer modules use the features of different scales (local feature vectors) output by the backbone network part (feature pyramid module) to obtain global semantic information (semantic feature vectors) from the features of different scales, so as to understand the semantic information of the image to be segmented.

[0212] Bidirectional Transformer decoder module: includes feature fusion module, query vector and feature vector (fused feature vector) mutual update operation. In the bidirectional Transformer decoder module, considering that there is a semantic gap between the features extracted by the feature extraction module (feature pyramid module) and the semantic extraction module, the two cannot be used directly. Therefore, in this module, the local feature vector rich in detail information and the global semantics rich in semantic information are first fused, and then used to update the initial query vector, that is, to propagate the semantic information to the query vector. At the same time, in order to enhance the communication between the query vector and the feature vector, the feature vector is updated through the updated query vector, so as to accelerate the convergence of network iteration.

[0213] Segmentation head part: Generates segmentation results (segmentation results) according to the query vector (updated query vector) containing rich semantic information.

[0214] In the embodiment of the present application, while controlling the amount of calculation and the amount of parameters, the network structure of multi-task prediction can be realized through the image segmentation model, and the image segmentation effect can be guaranteed.

[0215] In summary, the embodiment of the present application provides an image segmentation method that can effectively extract image features, making the segmentation result more accurate and robust. By combining the feature pyramid module, the semantic extraction module and the bidirectional Transformer decoder, efficient image segmentation is achieved, providing a new solution for the field of image processing.

[0216] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0217] Based on the above embodiments, the present application provides a schematic diagram of an optional image segmentation device, as shown in FIG. Fig.10 As shown, the image segmentation device 10 includes an acquisition unit 11, an extraction unit 12, an update unit 13, a segmentation unit 14 and a training unit 15; wherein,

[0218] The acquisition unit 11 is used to acquire the image to be segmented;

[0219] The extraction unit 12 is used to perform content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented;

[0220] The updating unit 13 is used to perform bidirectional updating on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector;

[0221] The segmentation unit 14 is used to perform image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

[0222] In some embodiments of the present application, the content extraction process includes feature extraction process and semantic extraction process; the content feature vector includes a local feature vector and a global semantic vector.

[0223] In some embodiments of the present application, the extraction unit 12 is further used to perform feature extraction processing on the image to be segmented to obtain the local feature vector; and perform semantic extraction processing on the local feature vector to obtain the global semantic vector.

[0224] In some embodiments of the present application, the feature extraction process includes M convolution processes, and the local feature vector includes M first feature vectors; wherein M is a positive integer greater than or equal to 1; the extraction unit 12 is also used to perform the i-th convolution process on the image to be segmented to obtain the i-th first feature vector; wherein i is a positive integer less than M; continue to perform the i+1-th convolution process on the i-th first feature vector to obtain the i+1-th first feature vector, until i+1 is equal to M, to obtain the M first feature vectors obtained by the M convolution processes; wherein the size of the i+1-th first feature vector is smaller than the size of the i-th first feature vector.

[0225] In some embodiments of the present application, the extraction unit 12 is also used to perform pooling processing on the M first feature vectors in the local feature vectors respectively to obtain M first intermediate feature vectors; concatenate the M first intermediate feature vectors according to the channel dimension to obtain a second intermediate feature vector; perform semantic extraction processing on the second intermediate feature vector through at least one global feature interaction model to obtain the global semantic vector; wherein the global semantic vector includes M second feature vectors, M is a positive integer greater than or equal to 1; the global feature interaction model includes at least a multi-head attention model and a feedforward network model, the multi-head attention model is used to extract multi-level context information in the input vector, and the feedforward network model is used to perform nonlinear transformation processing on the input vector.

[0226] In some embodiments of the present application, the updating unit 13 is further used to perform feature fusion processing on the local feature vector and the global semantic vector to obtain a fused feature vector, and to perform bidirectional updating on the fused feature vector and the initial query vector to obtain an updated query vector and an output feature vector.

[0227] In some embodiments of the present application, the updating unit 13 is further used to perform feature fusion processing on the i-th first feature vector in the local feature vector and the i-th second feature vector corresponding to the i-th first feature vector in the global semantic vector to obtain the i-th fused feature vector; wherein i is a positive integer greater than or equal to 1 and less than M; bidirectionally update the i-th fused feature vector and the current query vector to obtain a current updated query vector and a current first output vector; determine the i+1-th fused feature vector based on the current first output vector and the i+1-th second feature vector corresponding to the i+1-th first feature vector in the global semantic vector; continue to bidirectionally update the current updated query vector according to the i+1-th fused feature vector to obtain a next updated query vector and a next first output vector, until i+1 is equal to M, and use the next updated query vector as the updated query vector, and use the next first output vector as the output feature vector.

[0228] In some embodiments of the present application, the update unit 13 is also used to upsample the current first output vector to obtain a current second output vector; add the current second output vector and the i+1th first feature vector to obtain a current third output vector; perform feature fusion processing on the current third output vector and the i+1th second feature vector to obtain the i+1th fused feature vector.

[0229] In some embodiments of the present application, the feature fusion processing includes: convolution processing, normalization processing, data mapping processing and weighted summation processing; the update unit 13 is also used to perform convolution processing and normalization processing on the i-th first feature vector and the i-th second feature vector, respectively, to obtain the current fourth output vector of the i-th first feature vector and the current fifth output vector of the i-th second feature vector; perform data mapping processing on the current fifth output vector to obtain the current sixth output vector of the i-th second feature vector; the current sixth output vector represents the semantic weight of the i-th second feature vector; multiply the current sixth output vector and the current fourth output vector to obtain the current seventh output vector of the i-th first feature vector; perform convolution processing and normalization processing on the i-th second feature vector to obtain the current eighth output vector of the i-th second feature vector; perform weighted summation on the current seventh output vector and the current eighth output vector to obtain the i-th fused feature vector.

[0230] In some embodiments of the present application, the updating unit 13 is further used to update the current query vector according to the i-th fused feature vector through a visual representation to semantic representation model to obtain the current updated query vector; the visual representation to semantic representation model is used to convert the visual information of the input vector into a vector representation with semantic meaning; through the semantic representation to visual representation model, the i-th fused feature vector is updated according to the current updated query vector to obtain the current first output vector; the semantic representation to visual representation model is used to convert the semantic information of the input vector into a vector representation with visual meaning.

[0231] In some embodiments of the present application, the visual representation to semantic representation model includes: a cross-attention model, a self-attention model and a feedforward network model; the cross-attention model is used to perform correlation feature analysis on different input sequences; the self-attention model is used to perform correlation feature analysis on each element in the input sequence;

[0232] In some embodiments of the present application, the updating unit 13 is further used to perform correlation feature analysis on the i-th fused feature vector and the current query vector through the cross-attention model to obtain a first query vector; add the first query vector and the current query vector to obtain a second query vector, and normalize the second query vector to obtain a third query vector; perform correlation feature analysis on the third query vector through the self-attention model to obtain a fourth query vector; add the fourth query vector and the third query vector to obtain a fifth query vector, and normalize the fifth query vector to obtain a sixth query vector; perform nonlinear transformation processing on the sixth query vector through the feedforward network model to obtain a seventh query vector; add the seventh query vector and the sixth query vector to obtain an eighth query vector, and normalize the eighth query vector to obtain the current updated query vector.

[0233] In some embodiments of the present application, the semantic representation to visual representation model includes: a cross-attention model; the update unit 13 is also used to perform correlation feature analysis on the current updated query vector and the i-th fused feature vector through the cross-attention model to obtain a ninth query vector; perform convolution processing and non-linear transformation processing on the ninth query vector to obtain a tenth query vector; perform convolution processing and non-linear transformation processing on the i-th fused feature vector to obtain a first intermediate feature vector; multiply the tenth query vector and the intermediate feature vector to obtain a second intermediate feature vector; perform convolution processing and non-linear transformation processing on the second intermediate feature vector to obtain the current first output vector.

[0234] In some embodiments of the present application, the segmentation unit 14 is further used to perform data classification processing on the updated query vector through a preset classification model to obtain a category vector corresponding to the updated query vector; wherein the preset classification model is used to perform classification according to the characteristics of the input vector; a mask feature vector is determined based on the updated query vector and the output feature vector; and image segmentation of the image to be segmented is achieved based on the mask feature vector and the category vector, thereby obtaining the segmentation result of the image to be segmented.

[0235] In some embodiments of the present application, the segmentation unit 14 is also used to perform dot multiplication processing on the updated query vector and the output feature vector to obtain an intermediate mask feature vector; the intermediate mask feature vector is optimized through a preset optimization model to obtain the mask feature vector; wherein the preset optimization model is used to optimize the edge part and texture part of the input vector.

[0236] In some embodiments of the present application, the extraction unit 12 is further configured to perform feature mapping processing according to semantic information in the content feature vector to obtain the initial query vector.

[0237] In some embodiments of the present application, the acquisition unit 11 is further used to acquire sample data; the sample data includes an original image sample and an instance sample corresponding to the original image sample.

[0238] In some embodiments of the present application, the training unit 15 is used to perform image segmentation on the original image sample through the initial image segmentation model to obtain candidate results of the original image sample; input the candidate results of the original image sample and the corresponding instance sample into a preset loss function to obtain a current loss result; modify the model parameters in the initial image segmentation model according to the current loss result to obtain a current intermediate image segmentation model; continue to perform image segmentation on the original image sample through the current intermediate image segmentation model to obtain the next loss result, and modify the model parameters in the current intermediate image segmentation model according to the next loss result to obtain the next intermediate image segmentation model, until the next loss result corresponding to the next intermediate image segmentation model meets the cutoff condition, and the next intermediate image segmentation model is used as the image segmentation model.

[0239] In some embodiments of the present application, the training unit 15 is further used to perform content extraction processing on the original image sample through the initial image segmentation model to obtain a content feature vector of the original image sample; to bidirectionally update the initial query vector and the content feature vector of the original image sample to obtain an updated query vector and an output feature vector of the original image sample; and to perform image segmentation on the original image sample based on the updated query vector and the output feature vector to obtain the candidate result of the original image sample.

[0240] In some embodiments of the present application, the image segmentation processing includes: at least one of semantic segmentation processing, instance segmentation processing and panoramic segmentation processing; the segmentation result includes: at least one of semantic segmentation result, instance segmentation result and panoramic segmentation result; the segmentation unit 14 is also used to perform semantic segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain the semantic segmentation result of the image to be segmented; perform instance segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain the instance segmentation result of the image to be segmented; perform panoramic segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain the panoramic segmentation result of the image to be segmented.

[0241] It should be noted that the division of modules or units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0242] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0243] Fig.11 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application, such as Fig.11 As shown, an embodiment of the present application provides an electronic device 20, including:

[0244] A memory 21, used for storing executable instructions;

[0245] The processor 22 is used to implement the above-mentioned image segmentation method when executing the executable instructions stored in the memory 21.

[0246] It can be understood that the memory 21 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 21 of the system and method described in the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0247] The processor 22 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the above method in combination with its hardware.

[0248] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units or combinations thereof for performing functions described in the present application. For software implementation, the technology described in the present application can be implemented by a module (such as a process, function, etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in a processor or outside a processor.

[0249] An embodiment of the present application provides a computer storage medium, which stores executable instructions. When the executable instructions are executed, the processor is used to execute the image segmentation method described in one or more of the above embodiments.

[0250] In some embodiments, the computer-readable storage medium can be applied to the electronic device in this embodiment, and when the computer program is executed by at least one processor, it implements the corresponding processes implemented by the electronic device in each method of this embodiment. For the sake of brevity, it will not be repeated here.

[0251] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the image segmentation method provided in the above method embodiment.

[0252] In some embodiments, the computer program product can be applied to the electronic device in this embodiment, and the computer program instructions enable the computer to execute the corresponding processes implemented by the electronic device in each method of this embodiment, which will not be repeated here for the sake of brevity.

[0253] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0254] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0255] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this application will not repeat them.

[0256] The term "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, object A and / or object B may represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0257] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0259] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0260] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0261] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0262] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0263] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0264] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0265] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image segmentation method, characterized in that: The method comprises: Obtain the image to be segmented; Performing content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented; bidirectionally updating the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; Image segmentation processing is performed on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

2. The method according to claim 1, characterized in that: The content extraction process includes feature extraction process and semantic extraction process; the content feature vector includes a local feature vector and a global semantic vector; The step of performing content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented includes: Performing feature extraction processing on the image to be segmented to obtain the local feature vector; The local feature vector is subjected to semantic extraction processing to obtain the global semantic vector.

3. The method according to claim 2, characterized in that The feature extraction process includes M convolution processes, and the local feature vector includes M first feature vectors; wherein M is a positive integer greater than or equal to 1; The step of performing feature extraction processing on the image to be segmented to obtain the local feature vector includes: Performing an i-th convolution process on the image to be segmented to obtain an i-th first eigenvector; wherein i is a positive integer less than M; Continue to perform the i+1th convolution process on the i-th first eigenvector to obtain the i+1th first eigenvector, until i+1 is equal to M, and obtain the M first eigenvectors obtained by the M-times convolution process; wherein the size of the i+1th first eigenvector is smaller than the size of the i-th first eigenvector.

4. The method according to claim 2 or 3, characterized in that: The performing semantic extraction processing on the local feature vector to obtain the global semantic vector includes: Performing pooling processing on the M first feature vectors in the local feature vectors respectively to obtain M first intermediate feature vectors; According to the channel dimension, the M first intermediate feature vectors are concatenated to obtain a second intermediate feature vector; The second intermediate feature vector is semantically extracted through at least one global feature interaction model to obtain the global semantic vector; wherein the global semantic vector includes M second feature vectors, and M is a positive integer greater than or equal to 1; the global feature interaction model includes at least a multi-head attention model and a feedforward network model, the multi-head attention model is used to extract multi-level context information from the input vector, and the feedforward network model is used to perform nonlinear transformation processing on the input vector.

5. The method according to any one of claims 1 to 4, characterized in that: The content feature vector includes a local feature vector and a global semantic vector; The step of bidirectionally updating the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector includes: The local feature vector and the global semantic vector are subjected to feature fusion processing to obtain a fused feature vector, and the fused feature vector and the initial query vector are bidirectionally updated to obtain an updated query vector and an output feature vector.

6. The method according to claim 5, characterized in that The performing feature fusion processing on the local feature vector and the global semantic vector to obtain a fused feature vector, and bidirectionally updating the fused feature vector and the initial query vector to obtain an updated query vector and an output feature vector, includes: Performing feature fusion processing on the i-th first feature vector in the local feature vector and the i-th second feature vector corresponding to the i-th first feature vector in the global semantic vector to obtain an i-th fused feature vector; wherein i is a positive integer greater than or equal to 1 and less than M; Performing bidirectional updates on the i-th fused feature vector and the current query vector to obtain a current updated query vector and a current first output vector; Determine an i+1th fused feature vector based on the current first output vector and an i+1th second feature vector in the global semantic vector corresponding to the i+1th first feature vector; Continue to bidirectionally update the current updated query vector according to the i+1th fused feature vector to obtain a next updated query vector and a next first output vector, until i+1 is equal to M, and use the next updated query vector as the updated query vector and the next first output vector as the output feature vector.

7. The method according to claim 6, characterized in that The determining, based on the current first output vector and the (i+1)th second feature vector in the global semantic vector corresponding to the (i+1)th first feature vector, the (i+1)th fused feature vector comprises: Performing upsampling processing on the current first output vector to obtain a current second output vector; Add the current second output vector and the (i+1)th first eigenvector to obtain a current third output vector; Perform feature fusion processing on the current third output vector and the (i+1)th second feature vector to obtain the (i+1)th fused feature vector.

8. The method according to claim 6 or 7, characterized in that: The feature fusion processing includes: convolution processing, normalization processing, data mapping processing and weighted summation processing; The step of performing feature fusion processing on the i-th first feature vector in the local feature vector and the i-th second feature vector in the global semantic vector corresponding to the i-th first feature vector to obtain the i-th fused feature vector includes: Performing convolution processing and normalization processing on the i-th first eigenvector and the i-th second eigenvector respectively to obtain a current fourth output vector of the i-th first eigenvector and a current fifth output vector of the i-th second eigenvector; Performing data mapping processing on the current fifth output vector to obtain a current sixth output vector of the i-th second feature vector; the current sixth output vector represents the semantic weight of the i-th second feature vector; Multiplying the current sixth output vector and the current fourth output vector to obtain a current seventh output vector of the i-th first eigenvector; Performing convolution processing and normalization processing on the i-th second eigenvector to obtain a current eighth output vector of the i-th second eigenvector; Perform a weighted summation on the current seventh output vector and the current eighth output vector to obtain the i-th fused feature vector.

9. The method according to claim 6 or 7, characterized in that: The step of bidirectionally updating the i-th fused feature vector and the current query vector to obtain a current updated query vector and a current first output vector includes: The current query vector is updated according to the i-th fused feature vector through a visual representation to semantic representation model to obtain the current updated query vector; the visual representation to semantic representation model is used to convert visual information of an input vector into a vector representation with semantic meaning; Through the semantic representation to visual representation model, the i-th fused feature vector is updated according to the current updated query vector to obtain the current first output vector; the semantic representation to visual representation model is used to convert the semantic information of the input vector into a vector representation with visual meaning.

10. The method according to claim 9, characterized in that The visual representation to semantic representation model includes: a cross attention model, a self-attention model and a feedforward network model; the cross attention model is used to perform correlation feature analysis on different input sequences; the self-attention model is used to perform correlation feature analysis on each element in the input sequence; The updating of the current query vector according to the i-th fused feature vector through the visual representation to semantic representation model to obtain the current updated query vector includes: By using the cross attention model, performing correlation feature analysis on the i-th fused feature vector and the current query vector to obtain a first query vector; Adding the first query vector and the current query vector to obtain a second query vector, and normalizing the second query vector to obtain a third query vector; Performing correlation feature analysis on the third query vector by using the self-attention model to obtain a fourth query vector; Adding the fourth query vector and the third query vector to obtain a fifth query vector, and normalizing the fifth query vector to obtain a sixth query vector; Performing nonlinear transformation processing on the sixth query vector through the feedforward network model to obtain a seventh query vector; The seventh query vector and the sixth query vector are added to obtain an eighth query vector, and the eighth query vector is normalized to obtain the current updated query vector.

11. The method according to claim 9, characterized in that The semantic representation to visual representation model includes: a cross attention model; The updating of the i-th fused feature vector according to the current updated query vector through the semantic representation to visual representation model to obtain the current first output vector includes: By using the cross attention model, performing correlation feature analysis on the current updated query vector and the i-th fused feature vector to obtain a ninth query vector; Performing convolution processing and nonlinear transformation processing on the ninth query vector to obtain a tenth query vector; Performing convolution processing and nonlinear transformation processing on the i-th fused feature vector to obtain a first intermediate feature vector; Multiplying the tenth query vector and the intermediate feature vector to obtain a second intermediate feature vector; The second intermediate feature vector is subjected to convolution processing and nonlinear transformation processing to obtain the current first output vector.

12. The method according to any one of claims 1 to 11, characterized in that: The performing image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented includes: The updated query vector is subjected to data classification processing by a preset classification model to obtain a category vector corresponding to the updated query vector; wherein the preset classification model is used to perform classification according to features of the input vector; Determining a mask feature vector based on the updated query vector and the output feature vector; Image segmentation processing of the image to be segmented is implemented according to the mask feature vector and the category vector, thereby obtaining the segmentation result of the image to be segmented.

13. The method according to claim 12, characterized in that The step of determining a mask feature vector according to the updated query vector and the output feature vector comprises: Performing a dot multiplication process on the updated query vector and the output feature vector to obtain an intermediate mask feature vector; The intermediate mask feature vector is optimized through a preset optimization model to obtain the mask feature vector; wherein the preset optimization model is used to optimize the edge portion and texture portion of the input vector.

14. The method according to any one of claims 1 to 11, characterized in that Before bidirectionally updating the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector, the method further includes: Feature mapping processing is performed according to the semantic information in the content feature vector to obtain the initial query vector.

15. The method according to claim 1, characterized in that The segmentation result of the image to be segmented is obtained by performing image segmentation on the image to be segmented by an image segmentation model, and the image segmentation model is obtained by training an initial image segmentation model; the method further includes: Acquire sample data; the sample data includes an original image sample and an instance sample corresponding to the original image sample; Performing image segmentation on the original image sample by using the initial image segmentation model to obtain a candidate result of the original image sample; Inputting the candidate result of the original image sample and the corresponding instance sample into a preset loss function to obtain a current loss result; Modifying the model parameters in the initial image segmentation model according to the current loss result to obtain a current intermediate image segmentation model; Continue to perform image segmentation on the original image sample through the current intermediate image segmentation model to obtain the next loss result, and modify the model parameters in the current intermediate image segmentation model according to the next loss result to obtain the next intermediate image segmentation model, until the next loss result corresponding to the next intermediate image segmentation model meets the cutoff condition, and use the next intermediate image segmentation model as the image segmentation model.

16. The method according to claim 15, characterized in that The performing image segmentation on the original image sample by using the initial image segmentation model to obtain a candidate result of the original image sample includes: Performing content extraction processing on the original image sample through the initial image segmentation model to obtain a content feature vector of the original image sample; Bidirectionally updating the initial query vector and the content feature vector of the original image sample to obtain an updated query vector and an output feature vector of the original image sample; Image segmentation is performed on the original image sample according to the updated query vector and the output feature vector to obtain the candidate result of the original image sample.

17. The method according to any one of claims 1 to 16, characterized in that: The image segmentation process includes: at least one of semantic segmentation process, instance segmentation process and panorama segmentation process; the segmentation result includes: at least one of semantic segmentation result, instance segmentation result and panorama segmentation result; The performing image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented includes at least one of the following: According to the updated query vector and the output feature vector, semantic segmentation processing is performed on the image to be segmented to obtain the semantic segmentation result of the image to be segmented; According to the updated query vector and the output feature vector, performing instance segmentation processing on the image to be segmented to obtain the instance segmentation result of the image to be segmented; According to the updated query vector and the output feature vector, a panoramic segmentation process is performed on the image to be segmented to obtain the panoramic segmentation result of the image to be segmented.

18. An image segmentation device, characterized in that: The image segmentation device comprises an acquisition unit, an extraction unit, an update unit and a segmentation unit; wherein, The acquisition unit is used to acquire the image to be segmented; The extraction unit is used to perform content extraction processing on the image to be segmented to obtain a content feature vector of the image to be segmented; the content feature vector is used to represent feature information and semantic information of the image to be segmented; The updating unit is used to perform bidirectional updating on the initial query vector and the content feature vector to obtain an updated query vector and an output feature vector; The segmentation unit is used to perform image segmentation processing on the image to be segmented according to the updated query vector and the output feature vector to obtain a segmentation result of the image to be segmented.

19. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the image segmentation method described in any one of claims 1 to 17 when executing the executable instructions stored in the memory.

20. A computer-readable storage medium, characterized in that: The storage medium stores executable instructions, and when the executable instructions are executed by one or more processors, the image segmentation method according to any one of claims 1 to 17 is implemented.

Images