Target segmentation model establishment method and device
By encoding images and keywords, using self-attention and cross-attention networks to calculate cross features, and generating target segmentation mask images, the problem of poor ease of use of target segmentation methods in the prior art is solved, and segmentation efficiency and flexibility are improved sex.
Patent Information
- Application Number
- CN202310731052.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-19
- Publication Date
- 2024-12-20
AI Technical Summary
The existing target segmentation method has poor ease of use in multi-target area annotation and automation, and requires multiple detection and providing target information, reducing segmentation efficiency.
By encoding the image and the keywords of the target to be segmented, the self-attention network and the cross-attention network calculate the cross-features of text to image and image to text, perform correlation fusion, generate the predicted target segmentation mask image, and Update the model parameters through the loss function until the model converges.
It improves the efficiency and flexibility of target segmentation, simplifies the segmentation process of multi-target areas, and enhances the ease of use of segmentation method.
Smart Images
Figure CN119169401A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for establishing a target segmentation model and a method and device for establishing a target segmentation model. Background Art
[0002] Segmentation is the process of selecting a region of interest from a specified image. As a basic and important field, segmentation has many applications in medical imaging, autonomous driving, video conferencing, semi-automatic annotation, and other fields.
[0003] The interactive segmentation method first prompts through points, rectangular boxes or selected areas, and then the algorithm segments the target area. If there are multiple target areas in the image, multiple annotations are required, and the target information needs to be detected and provided during the automated use process, which greatly reduces the ease of use of the segmentation method. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for establishing a target segmentation model and a method and device for target segmentation to improve the efficiency of target segmentation.
[0005] The technical solution of the embodiment of the present invention is achieved as follows:
[0006] A method for establishing a target segmentation model, the method comprising:
[0007] A1. Encode the current sample image to obtain image encoding features, and encode the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features;
[0008] B1. Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain the text relevance enhancement features;
[0009] C1, input the text relevance enhancement features and image encoding features into the first cross attention network of the target segmentation model for calculation to obtain the cross features from text to image;
[0010] D1, input the cross features from text to image and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the cross features from image to text;
[0011] E1. Perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain the predicted target segmentation mask image;
[0012] F1, calculate the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain the loss function value;
[0013] G1, use the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network and the second cross-attention network;
[0014] H1. Reselect the current sample image and return to step A1 until the target segmentation model converges, and use the converged target segmentation model as the final target segmentation model.
[0015] After obtaining the cross-features from text to image in step C1 and before inputting the cross-features from text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation in step D1, the method further includes:
[0016] The cross features from text to image are input into the multi-layer perception network of the target segmentation model for calculation to obtain the spatial fusion features of the cross features from text to image;
[0017] Furthermore, in step D1, the cross-features from text to image and the image encoding features are input into the second cross-attention network of the target segmentation model for calculation, including:
[0018] The spatial fusion features of the cross features from text to image and the image encoding features are input into the second cross attention network of the target segmentation model for calculation;
[0019] The step G1 uses the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network, and the second cross-attention network, including:
[0020] The loss function value is used to update the neural network parameters of the self-attention network, the first criss-cross attention network, the multi-layer perception network and the second criss-cross attention network.
[0021] When the resolution of the predicted target segmentation mask image in step E1 is smaller than the resolution of the current sample image,
[0022] After obtaining the predicted target segmentation mask image in step E1 and before calculating the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image in step F1, the method further includes:
[0023] Input the predicted target segmentation mask image into the context upsampling network of the target segmentation model for calculation to obtain a final predicted target segmentation mask image, wherein the resolution of the final predicted target segmentation mask image is the same as the resolution of the current sample image;
[0024] The step F1 calculates the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image, including:
[0025] The loss function is calculated based on the final predicted target segmentation mask image and the actual target segmentation mask image corresponding to the current sample image.
[0026] The step A1 of encoding the current sample image includes:
[0027] Input the current sample image into the pre-trained first language-image pre-training model for calculation;
[0028] Step A1 encodes the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features, including:
[0029] The keywords corresponding to the target to be segmented in the current sample image are input into the pre-trained second language-image pre-training model for calculation.
[0030] Step A1: The first language-image pre-training model and the second language-image pre-training model are obtained through the following training process:
[0031] I1. Input the current sample image into the first language-image pre-training model to be trained for calculation to obtain image coding features;
[0032] Input the keywords corresponding to the current sample image into the second language-image pre-training model to be trained for calculation to obtain text encoding features;
[0033] Calculate the matching degree between the image encoding features and the text encoding features, and calculate the loss function value based on the matching degree;
[0034] Adjusting neural network parameters of the first language-image pre-trained model and the second language-image pre-trained model according to the loss function value;
[0035] J1. Reselect the current sample image and return to step I1 until the first language-image pre-training model and the second language-image pre-training model converge, and use the converged first language-image pre-training model and the second language-image pre-training model as the first language-image pre-training model and the second language-image pre-training model to be finally used.
[0036] After obtaining the cross-features from image to text in step D1 and before performing correlation fusion calculation on the cross-features from text to image and the cross-features from image to text in step E1, the method further includes:
[0037] Determine whether steps B1 to D1 have been executed for a preset number of times for the current sample image. If so, execute step E1; otherwise, update the image encoding feature with the latest image-to-text cross-feature, and update the text encoding feature with the latest text-to-image cross-feature, and return to step B1;
[0038] Furthermore, the step E1 of performing correlation fusion calculation on the cross features from text to image and the cross features from image to text includes:
[0039] The latest text-to-image cross-features and the latest image-to-text cross-features are correlated and fused.
[0040] A target segmentation method, the method comprising:
[0041] A2, encoding the current image to obtain image encoding features, and encoding the keywords corresponding to the target to be segmented in the current image to obtain text encoding features;
[0042] B2. Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain the text relevance enhancement features;
[0043] C2, input the text relevance enhancement features and image encoding features into the first cross attention network of the target segmentation model for calculation to obtain the cross features from text to image;
[0044] D2, input the cross features from text to image and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the cross features from image to text;
[0045] E2. Perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain the target segmentation mask image.
[0046] After obtaining the cross-features from text to image in step C2 and before inputting the cross-features from text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation in step D2, the method further includes:
[0047] The cross features from text to image are input into the multi-layer perception network of the target segmentation model for calculation to obtain the spatial fusion features of the cross features from text to image;
[0048] Furthermore, in step D2, the cross-features from text to image and the image encoding features are input into the second cross-attention network of the target segmentation model for calculation, including:
[0049] The spatial fusion features of the cross features from text to image and the image encoding features are input into the second cross attention network of the target segmentation model for calculation.
[0050] When the resolution of the target segmentation mask image in step E2 is smaller than the resolution of the current image,
[0051] After obtaining the target segmentation mask image in step E2, the method further includes:
[0052] The target segmentation mask image is input into the context upsampling network of the target segmentation model for calculation to obtain a final target segmentation mask image, wherein the resolution of the final target segmentation mask image is the same as the resolution of the current image.
[0053] The step A2 of encoding the current image includes:
[0054] Input the current image into the pre-trained first language-image pre-training model for calculation;
[0055] Step A2 encodes the keywords corresponding to the target to be segmented in the current image to obtain text encoding features, including:
[0056] The keywords corresponding to the target to be segmented in the current image are input into the pre-trained second language-image pre-training model for calculation.
[0057] After obtaining the cross-features from image to text in step D2 and before performing correlation fusion calculation on the cross-features from text to image and the cross-features from image to text in step E2, the method further includes:
[0058] Determine whether steps B2 to D2 have been executed for the current image for a preset number of times. If so, execute step E1; otherwise, update the image encoding feature with the latest image-to-text cross-feature, and update the text encoding feature with the latest text-to-image cross-feature, and return to step B1;
[0059] Furthermore, the step E1 of performing correlation fusion calculation on the cross features from text to image and the cross features from image to text includes:
[0060] The latest text-to-image cross-features and the latest image-to-text cross-features are correlated and fused.
[0061] A target segmentation model building device, the device comprising:
[0062] The encoding module is used to encode the current sample image to obtain image encoding features, and to encode the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features;
[0063] The self-attention module is used to perform self-attention calculation on the text encoding features to obtain text relevance enhancement features;
[0064] The first cross attention module is used to perform cross attention calculation on the text relevance enhancement feature and the image encoding feature to obtain the cross feature from text to image;
[0065] The second cross-attention module is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text;
[0066] A correlation fusion module is used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a predicted target segmentation mask image;
[0067] The parameter adjustment module is used to calculate the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain the loss function value; the loss function value is used to update the neural network parameters of the self-attention module, the first cross-attention module and the second cross-attention module until the target segmentation model converges, and the converged target segmentation model is used as the final target segmentation model.
[0068] A target segmentation device, the device comprising:
[0069] The encoding module is used to encode the current image to obtain image encoding features, and to encode keywords corresponding to the target to be segmented in the current image to obtain text encoding features;
[0070] The self-attention module is used to perform self-attention calculation on the text encoding features to obtain text relevance enhancement features;
[0071] The first cross attention module is used to perform cross attention calculation on the text relevance enhancement feature and the image encoding feature to obtain the cross feature from text to image;
[0072] The second cross-attention module is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text;
[0073] The correlation fusion module is used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a target segmentation mask image.
[0074] In an embodiment of the present invention, the current sample image and the keywords corresponding to the target to be segmented in the image are encoded, and then self-attention calculation is performed based on the obtained text encoding features to obtain text relevance enhancement features, and then text-to-image cross-attention calculation and image-to-text cross-attention calculation are performed based on the image encoding features and the text relevance enhancement features to obtain text-to-image cross-features and image-to-text cross-features, and correlation fusion calculation is performed on the two features to obtain a predicted target segmentation mask image, and a loss function is calculated based on the predicted target segmentation mask image and the actual target segmentation mask image, and finally a converged target segmentation model is obtained, thereby improving the efficiency of target segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0076] Figure 1 A flowchart of a method for establishing a target segmentation model provided by an embodiment of the present invention;
[0077] Figure 2 A flow chart of a target segmentation method provided by an embodiment of the present invention;
[0078] Figure 3 A schematic diagram of the structure of a device for establishing a target segmentation model provided by an embodiment of the present invention;
[0079] Figure 4 A schematic diagram of the structure of a target segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0082] The inventor of the present application has found that the segmentation method based on input text can segment the area of interest by inputting words or phrases such as "car" or "blue sky". This method can improve the flexibility and interpretability of the segmentation method, because people only pay attention to specific categories in many scenarios. The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0083] Figure 1 This is a flow chart of a method for establishing a target segmentation model provided by an embodiment of the present invention, and the specific steps are as follows:
[0084] Step 101: perform image encoding on the current sample image to obtain image encoding features, and encode keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features.
[0085] In an optional embodiment, in step 101, image encoding is performed on the current sample image, including: inputting the current sample image into a pre-trained first language-image pre-training model for calculation.
[0086] In an optional embodiment, in step 101, the keywords corresponding to the target to be segmented in the current sample image are encoded to obtain text encoding features, including: inputting the keywords corresponding to the target to be segmented in the current sample image into a pre-trained second language-image pre-training model for calculation.
[0087] The first language-image pre-training model and the second language-image pre-training model are models that can jointly model language and image data. This model can learn shared multimodal representations through pre-training on a large number of image and text pairs.
[0088] In an optional embodiment, the first language-image pre-training model and the second language-image pre-training model may adopt a CLIP (Contrastive Language-Image Pre-Training) model, or a ViLBERT (Vision-Language Bidirectional Encoder Representation from Transformers) model, or a BLIP (Bootstrapping Language-Image Pre-training) model, etc.
[0089] In an optional embodiment, in step 101, the first language-image pre-training model and the second language-image pre-training model are obtained through the following training process:
[0090] Step 1011: input the current sample image into the first language-image pre-training model to be trained for calculation to obtain image coding features; input the keywords corresponding to the current sample image into the second language-image pre-training model to be trained for calculation to obtain text coding features.
[0091] In practical applications, the first language-image pre-training model and the second language-image pre-training model may include: a convolutional network, a self-attention network and an MLP (Multi-Layer Perceptron) network, respectively. The convolutional network is used to efficiently extract features, the self-attention network is used to perform full-range spatial feature fusion, and the MLP is used to perform channel-range feature fusion.
[0092] In step 1011, the specific process of inputting the current sample image into the first language-image pre-training model to be trained for calculation is as follows:
[0093] First, the current sample image is divided into multiple non-overlapping blocks. For example, if the current sample image I is an RGB image, the dimension of image I is H×W×3, where H is the height of image I, W is the width of image I, and 3 is the number of channels of image I. Then, image I is divided into multiple non-overlapping blocks. The size of each block can be: 16×16, or 32×32, etc. Then, multiple blocks are input into the convolutional network for initial feature extraction to obtain the initial image feature f. For example, if the size of each block is 16×16, the dimension of f can be: 1×76 8×(H / 16)×(W / 16) (where 1 represents 1 frame of image, 768 is the feature dimension, and here each pixel is represented by a vector of length 768); then, the initial image feature f is added with the preset position encoding feature, and the superimposed feature is input into the self-attention network, where the dimension of the position encoding feature is the same as f, and the specific value can be set according to experience, etc. The number of layers of the self-attention network can be set according to experience, such as: 12 layers; then the feature output by the self-attention network is input into the MLP network to obtain the image encoding feature Q image , Q image The dimension can be: 1×256×(H / 16)×(W / 16) (where 1 represents 1 frame of image and 256 is the feature dimension).
[0094] In step 1011, the specific process of inputting the keywords corresponding to the current sample image into the second language-image pre-training model to be trained for calculation is as follows:
[0095] First, the keywords are converted into vectors of fixed length according to the word correspondence table. If the number of words is insufficient, "0" is used to fill it. For example, "a glass" is converted to: [49406, 320, 3313, 49407, 0, 0, ....] (where a corresponds to 49406, space corresponds to 320, glass corresponds to 3313, 49407 indicates the end character, and 0, 0, .... correspond to the filler when the number of words is insufficient); then, the converted vector is added with a preset position encoding vector, and the superimposed vector is input into the self-attention network, where the dimension of the position encoding vector is the same as the dimension of the vector after the keyword conversion, and the specific value of the position encoding vector can be set according to experience, etc. The number of layers of the self-attention network can be set according to experience, or / and according to the capabilities of the computing hardware and considering the balance between computing accuracy and speed, such as: set to 11 layers; then, the feature vector output by the self-attention network is input into the MLP network to obtain the text encoding feature Q text , Q text The dimension can be: 1 × 256. The keywords can be either English text or Chinese text.
[0096] Among them, position encoding features are a technique to introduce absolute or relative position information into neural network sequence modeling. Position encoding features represent a front-back (1-dimensional) position or a spatial position (2-dimensional), and their function is to make the features have a positional relationship. For example: the simplest form is to use a unique hot code for each position. For a sequence of length m, m unique hot codes of size n are required as position encoding features. In summary, the position of each pixel in the image or the serial number of each word in the keyword is encoded into an n-dimensional vector, and then the image or text features are superimposed and input into the self-attention network.
[0097] Step 1012: Calculate the matching degree between the image coding feature and the text coding feature, and calculate the loss function value according to the matching degree.
[0098] In practical applications, the matching degree between the image coding feature and the text coding feature can be obtained by calculating the square of the difference between the corresponding vectors of the image coding feature and the text coding feature.
[0099] Step 1013: Adjust the neural network parameters of the first language-image pre-training model and the second language-image pre-training model according to the loss function value.
[0100] Step 1014: reselect the current sample image and return to step 1011 until the first language-image pre-trained model and the second language-image pre-trained model converge, and use the converged first language-image pre-trained model and the second language-image pre-trained model as the first language-image pre-trained model and the second language-image pre-trained model to be finally used.
[0101] Step 102: Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain text relevance enhancement features.
[0102] In practical applications, the text encoding feature Q text With the initial output encoding feature Q init Superposition, the superimposed feature Q = Q text +Q init Input self-attention network, where Q init The initial value of can be a random value, and then it is updated according to the loss function value. init Dimension and Q text same.
[0103] The calculation process of the attention network (including the self-attention network and the cross-attention network) can be shown as formula (1):
[0104]
[0105] Among them, Q, K, V are the inputs of the attention network, Attention(Q, K, V) is the output of the attention network, softmax() is the name of the activation function, and dk is the size of the second dimension of Q, that is, the number of channels. For the self-attention network in step 102, K=V=Q, and for the value of dk, for example, Q=2*256, where 256 is the size of the second dimension of Q, that is, the number of channels is 256, then dk=256.
[0106] Step 103: Input the text relevance enhancement features and the image encoding features into the first cross-attention network of the target segmentation model for calculation to obtain the cross-features from text to image.
[0107] Specifically, the first cross-attention network calculates the text relevance enhancement features and image encoding features according to formula (1). The calculation process is essentially to calculate the correlation value between each word (including spaces and terminators) in the keyword (i.e., text) and each pixel in the image, and then perform weighted summation on the correlation values to obtain the cross-features of the word and the image, and finally obtain the cross-features of all words and images.
[0108] The first cross-attention network associates the image with the keywords corresponding to the target to be segmented, so that the image can be focused according to the keywords, thereby semantically parsing the details in the image related to the keyword content.
[0109] The calculation process of the first cross-attention network is also shown in formula (1), with the difference that K and V are image encoding features, and Q is the output feature of the self-attention network, i.e., the text relevance enhancement feature in step 102.
[0110] For example, the dimension of the cross feature from text to image is 1×256.
[0111] Step 104: Input the text-to-image cross features and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the image-to-text cross features.
[0112] Specifically, the second cross-attention network calculates the cross-features from text to image and the image encoding features according to formula (1). The calculation process is essentially to calculate the correlation value between each pixel in the image and each word (including spaces and terminators) in the keyword (i.e., text), and then perform weighted summation on the correlation values to obtain the cross-features between the pixel and the keyword, and finally obtain the cross-features between all pixels and the keyword (i.e., text).
[0113] The calculation process of the second cross-attention network is also as shown in formula (1), the difference is that K and V are the output features of the first cross-attention network, that is, the cross-features from text to image in step 103, and Q is the image encoding feature.
[0114] For example, the dimension of the cross feature from image to text is: 1×256×(H / 16)×(W / 16).
[0115] Step 105: performing correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a predicted target segmentation mask image.
[0116] In practical applications, the cross-features from text to image and from image to text are represented by matrices respectively, and the correlation fusion calculation can be a matrix multiplication operation. For example, the dimension of the cross-features from text to image is 1×256, and the dimension of the cross-features from image to text is 1×256×(H / 16)×(W / 16), so the size of the predicted target segmentation mask image is (H / 16)×(W / 16).
[0117] After the correlation fusion calculation, for each pixel in the predicted target segmentation mask image, a predicted probability value of whether the pixel is the background or the target will be obtained.
[0118] Step 106: Calculate the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain a loss function value.
[0119] Step 107: Use the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network, and the second cross-attention network.
[0120] Step 108: reselect the current sample image and return to step 101 until the target segmentation model converges, and use the converged target segmentation model as the final target segmentation model.
[0121] In the above embodiment, the current sample image and the keywords corresponding to the target to be segmented in the image are encoded, and then self-attention calculation is performed based on the obtained text encoding features to obtain text relevance enhancement features, and then text-to-image cross-attention calculation and image-to-text cross-attention calculation are performed based on the image encoding features and the text relevance enhancement features to obtain text-to-image cross-features and image-to-text cross-features, and correlation fusion calculation is performed on the two features to obtain a predicted target segmentation mask image, and a loss function is calculated based on the predicted target segmentation mask image and the actual target segmentation mask image, and finally a converged target segmentation model is obtained, thereby improving the efficiency of target segmentation.
[0122] In an optional embodiment, after step 103 obtains the cross-features of text to image and before step 104 inputs the cross-features of text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation, the method further includes: inputting the cross-features of text to image into the MLP network of the target segmentation model for calculation to obtain the spatial fusion features of the cross-features of text to image;
[0123] The calculation process of the MLP network can be shown as formula (2):
[0124] output=w2×relu(w1*input+b1)+b2(2)
[0125] Among them, output is the output of the MLP network, input is the input of the MLP network, relu() is the name of the activation function, w1 and w2 are weights, and b1 and b2 are biases.
[0126] Furthermore, in step 104, the cross features of the text to the image and the image coding features are input into the second cross attention network of the target segmentation model for calculation, including: the spatial fusion features of the cross features of the text to the image and the image coding features are input into the second cross attention network of the target segmentation model for calculation;
[0127] The calculation process of the second cross-attention network is shown in formula (1), where K and V are the output features of the MLP network, i.e., the spatial fusion features of the cross features from text to image, and Q is the image encoding feature.
[0128] Moreover, step 107 uses the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network, and the second cross-attention network, including: using the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network, the multi-layer perception network, and the second cross-attention network.
[0129] In an optional embodiment, when the resolution of the target segmentation mask image predicted in step 105 is less than the resolution of the current sample image, after step 105 obtains the predicted target segmentation mask image and before step 106 calculates the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image, it further includes: inputting the predicted target segmentation mask image into the context upsampling network of the target segmentation model for calculation to obtain a final predicted target segmentation mask image, wherein the resolution of the final predicted target segmentation mask image is the same as the resolution of the current sample image;
[0130] After the predicted target segmentation mask image is input into the context upsampling network of the target segmentation model for calculation, the final predicted target segmentation mask image can better maintain the contour shape of the target and ensure better edge details of the segmented image.
[0131] The calculation process of the context upsampling network can be shown as formula (3):
[0132]
[0133] Among them, masks is the input of the context upsampling network, that is, the predicted target segmentation mask image; in is an intermediate variable, inc×scale 2 is the number of output channels of in, inc is the number of channels of masks, for example, inc=2, scale is the preset upsampling multiple; conv() is the convolution function; softmax() is the activation function name; dim means performing softmax() operation on the dim-th dimension of in; attn is the intermediate variable; is the matrix multiplication operator; up_mask is the output of the context upsampling network.
[0134] Furthermore, step 106 calculates the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image, including: calculating the loss function based on the final predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image.
[0135] Among them, the real target segmentation mask image corresponding to the current sample image can be manually annotated in advance.
[0136] In practical applications, the loss function can adopt the focal loss function of binary classification. The focal loss function expands the expressive power of cross entropy loss and allows a more flexible and controllable trade-off strategy. It is particularly suitable for unbalanced classification tasks. The calculation formula of the focal loss function is as follows:
[0137] loss = -(1-pt) gamma log(pt)(4)
[0138] Among them, loss is the loss function value, pt is the predicted probability value, and gamma is a preset adjustable parameter whose value range is [0,5].
[0139] In an embodiment of the present invention, for the final predicted target segmentation mask image, the predicted probability value of each pixel on the image being the target or the background is output, the loss of each pixel is calculated according to formula (4), and then the loss of all pixels is averaged, and the average value is the final loss function value.
[0140] In an optional embodiment, after step 104 obtains the cross-features of image to text and before step 105 performs correlation fusion calculation on the cross-features of text to image and image to text, the method further includes: determining whether steps 102 to 104 have been executed for a preset number of times, such as 4 times, for the current sample image; if so, executing step 105; otherwise, updating the image encoding feature with the latest cross-features of image to text, and updating the text encoding feature with the latest cross-features of text to image, and returning to step 102; wherein, when step 102 is executed for the second time in the loop, when the self-attention network performs calculation according to formula (1), the input Q does not need to consider the initial output encoding feature Q init , the latest cross-features from text to image can be directly used as Q.
[0141] Furthermore, step 105 performs correlation fusion calculation on the cross features from text to image and the cross features from image to text, including: performing correlation fusion calculation on the latest cross features from text to image and the latest cross features from image to text.
[0142] Figure 2 This is a flow chart of a target segmentation method provided by an embodiment of the present invention, and the specific steps are as follows:
[0143] Step 201: Encode the current image to obtain image encoding features, and encode the keywords corresponding to the target to be segmented in the current image to obtain text encoding features.
[0144] Step 202: Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain text relevance enhancement features.
[0145] Step 203: Input the text relevance enhancement features and the image encoding features into the first cross-attention network of the target segmentation model for calculation to obtain the cross-features from text to image.
[0146] Step 204: Input the text-to-image cross features and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the image-to-text cross features.
[0147] Step 205: performing correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a target segmentation mask image.
[0148] In the above embodiment, the keywords corresponding to the current image and the target to be segmented in the image are encoded, and then self-attention calculation is performed based on the obtained text encoding features to obtain text relevance enhancement features, and then text-to-image cross-attention calculation and image-to-text cross-attention calculation are performed based on the image encoding features and the text relevance enhancement features to obtain text-to-image cross-features and image-to-text cross-features, and correlation fusion calculation is performed on these two features to obtain a target segmentation mask image, thereby improving the efficiency of target segmentation.
[0149] In an optional embodiment, after step 203 obtains the cross-features of text to image and before step 204 inputs the cross-features of text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation, the method further includes: inputting the cross-features of text to image into the multi-layer perception network of the target segmentation model for calculation to obtain the spatial fusion features of the cross-features of text to image;
[0150] Furthermore, step 204 inputs the cross-features of text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation, including: inputting the spatial fusion features of the cross-features of text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation.
[0151] In an optional embodiment, when the resolution of the target segmentation mask image in step 205 is smaller than the resolution of the current image, after step 205 obtains the target segmentation mask image, it further includes: inputting the target segmentation mask image into the context upsampling network of the target segmentation model for calculation to obtain the final target segmentation mask image, wherein the resolution of the final target segmentation mask image is the same as the resolution of the current image.
[0152] In practical applications, when the context upsampling network outputs the final target segmentation mask image, it also outputs the confidence that each pixel is the target or the background.
[0153] In an optional embodiment, step 201 performs image encoding on the current image, including: inputting the current image into a pre-trained first language-image pre-training model for calculation;
[0154] Step 201 encodes the keywords corresponding to the target to be segmented in the current image to obtain text encoding features, including: inputting the keywords corresponding to the target to be segmented in the current image into a pre-trained second language-image pre-training model for calculation.
[0155] In an optional embodiment, after step 204 obtains the cross-features of image to text and before step 205 performs correlation fusion calculation on the cross-features of text to image and the cross-features of image to text, it further includes: determining whether steps 202 to 204 have been looped for a preset number of times for the current image, if so, executing step 205; otherwise, updating the image encoding feature with the latest cross-features of image to text, updating the text encoding feature with the latest cross-features of text to image, and returning to step 202;
[0156] Furthermore, step 205 performs correlation fusion calculation on the cross features from text to image and the cross features from image to text, including: performing correlation fusion calculation on the latest cross features from text to image and the latest cross features from image to text.
[0157] The embodiments of the present invention are not only applicable to the segmentation of one target, but also to the segmentation of multiple targets. For multiple targets, such as pedestrians, plants, pets, etc., the keywords corresponding to the multiple targets are encoded respectively to obtain multiple vectors, and the multiple vectors are input into the second language-image pre-training model in parallel, and the image is input into the first language-image pre-training model at the same time.
[0158] The execution subject of the embodiment of the present invention may be a GPU (Graphics Processing Unit).
[0159] Figure 3 The schematic diagram of the structure of the target segmentation model establishment device provided by the embodiment of the present invention mainly includes: an encoding module 31, a self-attention module 32, a first cross-attention module 33, a second cross-attention module 34, a correlation fusion module 35 and a parameter adjustment module 36, wherein:
[0160] The encoding module 31 is used to perform image encoding on the current sample image to obtain image encoding features, and to encode keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features.
[0161] Self-attention module 32: used to perform self-attention calculation on text encoding features to obtain text relevance enhancement features.
[0162] The first cross-attention module 33 is used to perform cross-attention calculation on the text relevance enhancement features and the image encoding features to obtain cross-features from text to image.
[0163] The second cross-attention module 34 is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text.
[0164] Correlation fusion module 35: used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a predicted target segmentation mask image.
[0165] Parameter adjustment module 36: used to calculate the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain the loss function value; use the loss function value to update the neural network parameters of the self-attention module, the first cross-attention module and the second cross-attention module until the target segmentation model converges, and use the converged target segmentation model as the final target segmentation model.
[0166] In an optional embodiment, the above-mentioned device further comprises: a multi-layer perception module, which is used to perform spatial fusion calculation on the cross features from text to image to obtain spatial fusion features of the cross features from text to image;
[0167] Furthermore, the second cross-attention module 34 performs cross-attention calculation on the cross-features of the text to the image and the image coding features, including: performing cross-attention calculation on the spatial fusion features and the image coding features of the cross-features of the text to the image;
[0168] Moreover, the parameter adjustment module 36 uses the loss function value to update the neural network parameters of the self-attention module, the first cross-attention module and the second cross-attention module, including: using the loss function value to update the neural network parameters of the self-attention module, the first cross-attention module, the multi-layer perception module and the second cross-attention module.
[0169] In an optional embodiment, when the resolution of the predicted target segmentation mask image obtained by the correlation fusion module 35 is smaller than the resolution of the current sample image, the above-mentioned device further includes: a context upsampling module;
[0170] Furthermore, the correlation fusion module 35 is further used to input the predicted target segmentation mask image into the context upsampling module;
[0171] The context upsampling module is used to perform context upsampling calculation on the predicted target segmentation mask image to obtain a final predicted target segmentation mask image, wherein the resolution of the final predicted target segmentation mask image is the same as the resolution of the current sample image;
[0172] Moreover, the parameter adjustment module 36 calculates the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image, including: calculating the loss function according to the final predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image.
[0173] In an optional embodiment, the encoding module 31 includes: a first language-image pre-training module and a second language-image pre-training module, wherein:
[0174] The first language-image pre-training module is used to perform language-image pre-training calculation on the current sample image to obtain image coding features.
[0175] The second language-image pre-training module is used to perform language-image pre-training calculations on keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features.
[0176] Figure 4 The schematic diagram of the structure of the target segmentation device provided by the embodiment of the present invention mainly includes: an encoding module 41, a self-attention module 42, a first cross-attention module 43, a second cross-attention module 44 and a correlation fusion module 45, wherein:
[0177] The encoding module 41 is used to encode the current image to obtain image encoding features, and to encode keywords corresponding to the target to be segmented in the current image to obtain text encoding features.
[0178] Self-attention module 42: used to perform self-attention calculation on text encoding features to obtain text relevance enhancement features.
[0179] The first cross-attention module 43 is used to perform cross-attention calculations on text relevance enhancement features and image encoding features to obtain cross-features from text to image.
[0180] The second cross-attention module 44 is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text.
[0181] Correlation fusion module 45: used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a target segmentation mask image.
[0182] In an optional embodiment, the above-mentioned device further comprises: a multi-layer perception module, which is used to perform spatial fusion calculation on the cross features from text to image to obtain spatial fusion features of the cross features from text to image;
[0183] Furthermore, the second cross-attention module 44 performs cross-attention calculation on the cross-features from text to image and the image coding features, including: performing cross-attention calculation on the spatial fusion features and the image coding features of the cross-features from text to image.
[0184] In an optional embodiment, when the resolution of the target segmentation mask image obtained by the correlation fusion module 45 is smaller than the resolution of the current image, the above-mentioned apparatus further comprises: a context upsampling module;
[0185] Furthermore, the correlation fusion module 45 is further used to input the target segmentation mask image into the context upsampling module;
[0186] Furthermore, the context upsampling module is used to perform context upsampling calculation on the target segmentation mask image to obtain a final target segmentation mask image, wherein the resolution of the final target segmentation mask image is the same as the resolution of the current image.
[0187] In an optional embodiment, the encoding module 41 includes: a first language-image pre-training module and a second language-image pre-training module, wherein:
[0188] The first language-image pre-training module is used to perform language-image pre-training calculations on the current four images to obtain image coding features.
[0189] The second language-image pre-training module is used to perform language-image pre-training calculations on keywords corresponding to the target to be segmented in the current image to obtain text encoding features.
[0190] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor causes the processor to execute the steps of the target segmentation model establishment method described in any embodiment of the present invention, or execute the steps of the target segmentation method described in any embodiment of the present invention.
[0191] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.
[0192] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.
Claims
1. A method for establishing a target segmentation model, characterized in that: The method includes: A1. Encode the current sample image to obtain image encoding features, and encode the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features; B1. Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain the text relevance enhancement features; C1, input the text relevance enhancement features and image encoding features into the first cross attention network of the target segmentation model for calculation to obtain the cross features from text to image; D1, input the cross features from text to image and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the cross features from image to text; E1. Perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain the predicted target segmentation mask image; F1, calculate the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain the loss function value; G1, use the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network and the second cross-attention network; H1. Reselect the current sample image and return to step A1 until the target segmentation model converges, and use the converged target segmentation model as the final target segmentation model.
2. The method according to claim 1, characterized in that After obtaining the cross-features from text to image in step C1 and before inputting the cross-features from text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation in step D1, the method further includes: The cross features from text to image are input into the multi-layer perception network of the target segmentation model for calculation to obtain the spatial fusion features of the cross features from text to image; Furthermore, in step D1, the cross-features from text to image and the image encoding features are input into the second cross-attention network of the target segmentation model for calculation, including: The spatial fusion features of the cross features from text to image and the image encoding features are input into the second cross attention network of the target segmentation model for calculation; The step G1 uses the loss function value to update the neural network parameters of the self-attention network, the first cross-attention network, and the second cross-attention network, including: The loss function value is used to update the neural network parameters of the self-attention network, the first criss-cross attention network, the multi-layer perception network and the second criss-cross attention network.
3. The method according to claim 1, characterized in that When the resolution of the predicted target segmentation mask image in step E1 is smaller than the resolution of the current sample image, After obtaining the predicted target segmentation mask image in step E1 and before calculating the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image in step F1, the method further includes: Input the predicted target segmentation mask image into the context upsampling network of the target segmentation model for calculation to obtain a final predicted target segmentation mask image, wherein the resolution of the final predicted target segmentation mask image is the same as the resolution of the current sample image; The step F1 calculates the loss function based on the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image, including: The loss function is calculated based on the final predicted target segmentation mask image and the actual target segmentation mask image corresponding to the current sample image.
4. The method according to claim 1, characterized in that: The step A1 of encoding the current sample image includes: Input the current sample image into the pre-trained first language-image pre-training model for calculation; Step A1 encodes the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features, including: The keywords corresponding to the target to be segmented in the current sample image are input into the pre-trained second language-image pre-training model for calculation.
5. The method according to claim 4, characterized in that Step A1: The first language-image pre-training model and the second language-image pre-training model are obtained through the following training process: I1. Input the current sample image into the first language-image pre-training model to be trained for calculation to obtain image coding features; Input the keywords corresponding to the current sample image into the second language-image pre-training model to be trained for calculation to obtain text encoding features; Calculate the matching degree between the image encoding features and the text encoding features, and calculate the loss function value according to the matching degree; Adjusting neural network parameters of the first language-image pre-trained model and the second language-image pre-trained model according to the loss function value; J1. Reselect the current sample image and return to step I1 until the first language-image pre-training model and the second language-image pre-training model converge, and use the converged first language-image pre-training model and the second language-image pre-training model as the first language-image pre-training model and the second language-image pre-training model to be finally used.
6. The method according to claim 1, characterized in that After obtaining the cross-features from image to text in step D1 and before performing correlation fusion calculation on the cross-features from text to image and the cross-features from image to text in step E1, the method further includes: Determine whether steps B1 to D1 have been executed for a preset number of times for the current sample image. If so, execute step E1; otherwise, update the image encoding feature with the latest image-to-text cross-feature, and update the text encoding feature with the latest text-to-image cross-feature, and return to step B1; Furthermore, the step E1 of performing correlation fusion calculation on the cross features from text to image and the cross features from image to text includes: The latest text-to-image cross-features and the latest image-to-text cross-features are correlated and fused.
7. A target segmentation method, characterized in that: The method includes: A2, encoding the current image to obtain image encoding features, and encoding the keywords corresponding to the target to be segmented in the current image to obtain text encoding features; B2. Input the text encoding features into the self-attention network of the target segmentation model for calculation to obtain the text relevance enhancement features; C2, input the text relevance enhancement features and image encoding features into the first cross attention network of the target segmentation model for calculation to obtain the cross features from text to image; D2, input the cross features from text to image and the image encoding features into the second cross attention network of the target segmentation model for calculation to obtain the cross features from image to text; E2. Perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain the target segmentation mask image.
8. The method according to claim 7, characterized in that After obtaining the cross-features from text to image in step C2 and before inputting the cross-features from text to image and the image encoding features into the second cross-attention network of the target segmentation model for calculation in step D2, the method further includes: The cross features from text to image are input into the multi-layer perception network of the target segmentation model for calculation to obtain the spatial fusion features of the cross features from text to image; Furthermore, in step D2, the cross-features from text to image and the image encoding features are input into the second cross-attention network of the target segmentation model for calculation, including: The spatial fusion features of the cross features from text to image and the image encoding features are input into the second cross attention network of the target segmentation model for calculation.
9. The method according to claim 7, characterized in that: When the resolution of the target segmentation mask image in step E2 is smaller than the resolution of the current image, After obtaining the target segmentation mask image in step E2, the method further includes: The target segmentation mask image is input into the context upsampling network of the target segmentation model for calculation to obtain a final target segmentation mask image, wherein the resolution of the final target segmentation mask image is the same as the resolution of the current image.
10. The method according to claim 7, characterized in that The step A2 of encoding the current image includes: Input the current image into the pre-trained first language-image pre-training model for calculation; Step A2 encodes the keywords corresponding to the target to be segmented in the current image to obtain text encoding features, including: The keywords corresponding to the target to be segmented in the current image are input into the pre-trained second language-image pre-training model for calculation.
11. The method according to claim 7, characterized in that After obtaining the cross-features from image to text in step D2 and before performing correlation fusion calculation on the cross-features from text to image and the cross-features from image to text in step E2, the method further includes: Determine whether steps B2 to D2 have been executed for the current image for a preset number of times. If so, execute step E1; otherwise, update the image encoding feature with the latest image-to-text cross-feature, and update the text encoding feature with the latest text-to-image cross-feature, and return to step B1; Furthermore, the step E1 of performing correlation fusion calculation on the cross features from text to image and the cross features from image to text includes: The latest text-to-image cross-features and the latest image-to-text cross-features are correlated and fused.
12. A target segmentation model building device, characterized in that: The device includes: The encoding module is used to encode the current sample image to obtain image encoding features, and to encode the keywords corresponding to the target to be segmented in the current sample image to obtain text encoding features; The self-attention module is used to perform self-attention calculation on the text encoding features to obtain text relevance enhancement features; The first cross attention module is used to perform cross attention calculation on the text relevance enhancement feature and the image encoding feature to obtain the cross feature from text to image; The second cross-attention module is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text; A correlation fusion module is used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a predicted target segmentation mask image; The parameter adjustment module is used to calculate the loss function according to the predicted target segmentation mask image and the real target segmentation mask image corresponding to the current sample image to obtain the loss function value; the loss function value is used to update the neural network parameters of the self-attention module, the first cross-attention module and the second cross-attention module until the target segmentation model converges, and the converged target segmentation model is used as the final target segmentation model.
13. A target segmentation device, characterized in that: The device includes: The encoding module is used to encode the current image to obtain image encoding features, and to encode keywords corresponding to the target to be segmented in the current image to obtain text encoding features; The self-attention module is used to perform self-attention calculation on the text encoding features to obtain text relevance enhancement features; The first cross attention module is used to perform cross attention calculation on the text relevance enhancement feature and the image encoding feature to obtain the cross feature from text to image; The second cross-attention module is used to perform cross-attention calculation on the cross-features from text to image and the image encoding features to obtain the cross-features from image to text; The correlation fusion module is used to perform correlation fusion calculation on the cross features from text to image and the cross features from image to text to obtain a target segmentation mask image.