Generative reference segmentation method and device based on implicit structural features

The generative reference segmentation method addresses the complexity of integrating computer vision tasks into generative frameworks by employing a pre-trained structured feature extraction network to enhance the accuracy of object segmentation and localization in images.

CN118570481BActive Publication Date: 2024-12-06INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411062005.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-06
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

The integration of computer vision localization tasks into generative frameworks is challenging due to the complex structure and high computational complexity of existing generative segmentation models, leading to poor target positioning accuracy.

Method used

A generative reference segmentation method based on implicit structural features, utilizing a pre-trained structured feature extraction network to extract real structural features and supervise a large language model, enabling accurate prediction of circumscribed polygon coordinates for object segmentation.

Benefits of technology

Improves target positioning accuracy by directly segmenting objects in images using a supervised large language model, reducing computational complexity and enhancing the precision of object localization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570481B_ABST
    Figure CN118570481B_ABST
Patent Text Reader

Abstract

The present invention provides a generative referential segmentation method and device based on implicit structural features, which relates to the technical field of image data processing. The method includes: obtaining an image to be segmented and a question text; encoding the question text to obtain text features, encoding the segmented image to obtain image features, and merging the text features with the image features to obtain a feature sequence; calling a pre-trained structured feature extraction network to extract real structural features, and inputting the feature sequence into a large language model, supervising the large language model through the real structural features, generating object categories and implicit structural features, and then predicting the circumscribed polygon contour coordinates of the referential object in the image to be segmented, which are used to segment the referential object. Through the present invention, the technical problem in the prior art that the positioning task of computer vision is difficult to unify into a generative framework, resulting in a complex structure of the generative segmentation model and high computational complexity, resulting in a low target positioning effect is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a generative reference segmentation method and device based on implicit structural features. Background Art

[0002] The combination of computer vision technology and generative frameworks (such as multimodal large language models) can perform localization tasks for segmented targets, such as object recognition and pixel-level coordinate positioning, and is therefore widely used in various visual application fields. However, target localization usually requires the generation of target location information, and generative frameworks are more suitable for generating text, images and other content. Therefore, it is difficult to unify some computer vision localization tasks into generative frameworks. The main difficulty lies in how to effectively solve the target localization problem in a generative framework, so there are challenges in integrating localization tasks into a generative framework.

[0003] The solutions proposed in the prior art generally propose generative segmentation models based on regression, region proposal, Anchor-Free and hybrid methods, and integrate the target positioning task through specific architecture design and optimization strategies. However, the generative segmentation model has a complex structure and high computational complexity, resulting in poor target positioning effect. Summary of the invention

[0004] The present invention provides a generative referential segmentation method and device based on implicit structural features, which is used to solve the technical problem in the prior art that the positioning task of computer vision is difficult to unify into a generative framework, resulting in a complex structure and high computational complexity of the generative segmentation model, resulting in poor target positioning effect.

[0005] The present invention provides a generative coreference segmentation method based on implicit structural features, comprising the following steps:

[0006] Acquire an image to be segmented and a question text, wherein the question text is used to describe a reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object;

[0007] Encoding the question text to obtain text features of the question text, calling a pre-trained visual model to encode the image to be segmented to obtain image features of the image to be segmented, and merging the text features with the image features to obtain a feature sequence;

[0008] Calling a pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and inputting the feature sequence into a large language model, supervising the large language model through the real structural features, and generating the object category and implicit structural features of the referred object;

[0009] The object category and the implicit structural features are predicted and processed to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

[0010] In some embodiments, the pre-trained structured feature extraction network is obtained by training the mask label, and the mask label is obtained by randomly masking the real contour coordinate label of the object, and the training process of the pre-trained structured feature extraction network includes:

[0011] Constructing a point token sample according to the mask label, wherein the point token sample includes a visible point token and an occluded point token;

[0012] Encoding each of the point token samples through the pre-trained structured feature extraction network to obtain an embedded position feature of the point token sample, and fusing the embedded position feature with the point token sample to obtain an encoded point token;

[0013] Calling a lightweight decoder to decode each of the code point tokens to obtain a to-be-reconstructed point token of the code point token and the number of tokens of the to-be-reconstructed point token;

[0014] Reconstructing the point tokens to be reconstructed according to the number of tokens to obtain a reconstructed point set, and determining a mean square error between the point token samples and the reconstructed point set;

[0015] The mean square error is used as a training loss value of the pre-trained structured feature extraction network, and the training loss value is used to perform back propagation in the pre-trained structured feature extraction network to update the parameters of the pre-trained structured feature extraction network.

[0016] In some embodiments, the coordinates of the circumscribed polygon outline of the reference object in the image to be segmented are predicted by calling a reference segmentation model, the reference segmentation model includes a pre-trained visual model and a large language model, and the training process of the reference segmentation model includes:

[0017] Acquire an image sample and a question description text corresponding to the image sample, wherein the question description text is used to indicate a referred object in the image sample;

[0018] Encoding the question description text to obtain question text features, and calling the pre-trained visual model to encode the image sample to obtain image sample features;

[0019] Merging the image sample features with the question text features to generate a graphic feature sequence, and inputting the graphic feature sequence into the large language model to output a descriptive feature text of the referred object, wherein the descriptive feature text includes an object prediction category and a predicted structural feature of the referred object;

[0020] Calling the large language model to perform position prediction on the predicted category of the object and the predicted structural features, and generating predicted circumscribed polygon coordinates of the referred object in the image sample;

[0021] A final loss value of the representative segmentation model is constructed based on the predicted circumscribed polygon coordinates, and the final loss value is used to perform back propagation in the representative segmentation model to update the parameters of the representative segmentation model.

[0022] In some embodiments, the image sample further includes a thought chain, the thought chain is an implicitly trained structural feature of the object being referred to, the implicitly trained structural feature is extracted from the image sample by calling a pre-trained structured feature extraction network, and the calling of the large language model predicts the position of the object prediction category and the predicted structured feature to generate the predicted circumscribed polygon coordinates of the object being referred to in the image sample, including:

[0023] Calling the large language model to perform reasoning processing on the implicit training structural features, the object prediction category and the predicted structural features to obtain each contour position coordinate point of the referred object;

[0024] The predicted circumscribed polygon coordinates of the designated object in the image sample are determined according to the contour position coordinate points.

[0025] In some embodiments, the descriptive feature text of the referred object carries a real text label, the image sample carries a real contour coordinate label of the referred object, and the final loss value of the reference segmentation model is constructed based on the predicted circumscribed polygon coordinates, including:

[0026] Based on the predicted structural features and the true contour coordinate labels, constructing a structural feature loss value representing the segmentation model;

[0027] Constructing a text cross entropy loss value of the reference segmentation model according to the descriptive feature text and the real text label;

[0028] Based on the predicted circumscribed polygon coordinates and the true contour coordinate labels, constructing a coordinate cross entropy loss value of the reference segmentation model;

[0029] The structural feature loss value is weighted by the structural feature weight to obtain a structural feature weighted loss value, the text cross entropy loss value is weighted by the text weight to obtain a text weighted cross entropy loss value, and the coordinate cross entropy loss value is weighted by the coordinate weight to obtain a coordinate weighted cross entropy loss value;

[0030] The structural feature weighted loss value, the text weighted cross entropy loss value, and the coordinate weighted cross entropy loss value are summed to obtain a final loss value of the reference segmentation model.

[0031] In some embodiments, constructing the structural feature loss value representing the segmentation model based on the predicted structural features and the true contour coordinate label includes:

[0032] Calling a pre-trained structured feature extraction network to extract structure features from the real contour coordinate label to obtain real structured features;

[0033] The mean square error between the true structured feature and the predicted structured feature is determined, and the mean square error is used as the structured feature loss value of the reference segmentation model.

[0034] The present invention also provides a generative referential segmentation device based on implicit structural features, which specifically includes the following modules:

[0035] An acquisition module, used for acquiring an image to be segmented and a question text, wherein the question text is used to describe a reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object;

[0036] An encoding module is used to encode the question text to obtain text features of the question text, call a pre-trained visual model to encode the image to be segmented to obtain image features of the image to be segmented, and merge the text features with the image features to obtain a feature sequence;

[0037] A generation module, used for calling a pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and inputting the feature sequence into a large language model, supervising the large language model through the real structural features, and generating the object category and implicit structural features of the object referred to;

[0038] A prediction module is used to predict the object category and the implicit structural features to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor implements any of the above-described generative reference segmentation methods based on implicit structural features.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described generative reference segmentation methods based on implicit structural features.

[0041] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned generative reference segmentation methods based on implicit structural features.

[0042] The generative referential segmentation method based on implicit structural features provided by the present invention generates the object category and structural features of the referent object through a feature sequence composed of the image features to be segmented and the question text features by a large language model. The pre-trained structured feature extraction network is called to extract the real structural features of the referent object, and under the supervision of the real structural features, the object category and the implicit structural features are predicted and processed to obtain the circumscribed polygon contour coordinates of the referent object in the image to be segmented, thereby segmenting the referent object in the image to be segmented. Therefore, only by designing a pre-trained structured feature extraction network, the large language model is supervised to directly complete the segmentation of the referent object, and the target positioning task for the image to be segmented is completed, which effectively improves the accuracy of target positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a flow chart of the generative reference segmentation method based on implicit structural features provided by the present invention.

[0045] Figure 2 It is a schematic diagram of the principle of the generative reference segmentation method based on implicit structural features provided by the present invention.

[0046] Figure 3 It is a training process diagram of the pre-trained structured feature extraction network provided by the present invention.

[0047] Figure 4 It is a structural schematic diagram of a generative reference segmentation device based on implicit structural features provided by the present invention.

[0048] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Combine the following Figure 1 Describe the generative coreference segmentation method based on implicit structural features of the present invention, Figure 1 is one of the flow charts of the generative referential segmentation method based on implicit structural features provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0051] In step 101, an image to be segmented and a question text are obtained, wherein the question text is used to describe a reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object.

[0052] First, obtain the image to be segmented and the question text. The question text is used to describe the object in the image to be segmented. The image to be segmented carries the real contour coordinate label of the object. Here, the question text tells the large language model which objects need to be identified from the image to be segmented, that is, to clarify the target positioning task of the large language model. Figure 2 , Figure 2 is a schematic diagram of the principle of the generative referential segmentation method based on implicit structural features provided by the present invention, such as Figure 2 As shown in the figure, the image to be segmented is an image of a bathroom, and the question text is "Please find a place in the picture where you can wash your hands." From this, it can be determined that the question text is telling the large language model that it needs to identify a place where you can wash your hands from the image of the bathroom, that is, the washbasin, and "washbasin" refers to the object. The real contour coordinate label is essentially a polygonal annotation used to mark the reference object in the image to be segmented. It specifically and in detail marks the real contour coordinates of the reference object as an auxiliary positioning. Therefore, the real contour of the washbasin is marked in the image of the bathroom.

[0053] In step 102, the question text is encoded to obtain text features of the question text, a pre-trained visual model is called to encode the image to be segmented to obtain image features of the image to be segmented, and the text features are merged with the image features to obtain a feature sequence.

[0054] Next is the feature extraction process, which requires extracting the features of the image to be segmented and the question text and inputting them into the large language model for generation. Here, the text encoder can be called to encode the question text to obtain the text features of the question text (i.e., semantic features, so that the large language model can understand the semantics of the question text), and then the pre-trained visual model can be called to encode the image to be segmented to obtain the image features of the image to be segmented. Figure 2 As shown, the pre-trained visual model here can be a visual encoder (image encoder), and the text encoder and the pre-trained visual model can be replaced by a CLIP model or a SAM model without limitation.

[0055] After extracting the image features and the text features of the question text respectively, the image features and the text features are merged to form a feature sequence. The merging process can be to directly concatenate the feature vectors corresponding to the image features and the text features.

[0056] In step 103, a pre-trained structured feature extraction network is called to extract the real structural features of the real contour coordinate labels, and the feature sequence is input into the large language model. The large language model is supervised by the real structural features to generate the object category and implicit structural features referring to the object.

[0057] After merging to form a feature sequence, the feature sequence is directly input into the large language model, and the large language model generates the object category and implicit structural features of the object. Since the implicit structural features are used to predict the contour coordinates of the washbasin, in order to further improve the accuracy of the contour coordinates and improve the target positioning accuracy, the present invention also calls a pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and supervises the large language model to generate through the real structural features, that is, supervises the large language model to generate the object category and implicit structural features of the object.

[0058] Here, after extracting the real structural features, the feature sequence is input into the large language model, and the large language model is supervised by the real structural features to generate the object category and implicit structural features referring to the object. The large language model here can be a model such as LLaMA, Mamba, etc., and then the parameters are fine-tuned using the LoRA algorithm, and it can be directly used for generation. The supervision process can be to input the real structural features and the feature sequence into the large language model together. The large language model understands the image features and the semantics of the prompt text (text features), and compares them under the guidance of the input real structural features, and finally outputs the implicit structural features.

[0059] like Figure 2As shown in the figure, the generated result of the large language model is essentially a description text describing the washbasin. The description text includes the object category and implicit structural features of the washbasin, that is, the object category is determined to be a washbasin. The implicit structural features, that is, the structural features of the washbasin, are generated and output under the supervision of the real structural features of the washbasin generated by the pre-trained structured feature extraction network. During training, the implicit structural features are compared with the real structural features. Through comparative learning, the real structural features can supervise the implicit structural features accurately output by the large language model. This result feature is used to predict the contour coordinates of the washbasin in the future.

[0060] In step 104, the object category and implicit structural features are predicted and processed to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

[0061] After the large language model outputs the object category and implicit structural features, it then predicts the object category and implicit structural features to obtain the circumscribed polygon outline coordinates of the object in the image to be segmented. Here, the large language model will combine the object category and implicit structural features for the next prediction, thereby obtaining each contour coordinate of the object. This coordinate refers to the position contour coordinate of the object in the image to be segmented. Ultimately, these position contour coordinates can form a circumscribed polygon outline, that is, the circumscribed polygon outline coordinates are obtained. Based on this circumscribed polygon outline coordinate, it can be used to segment the object in the image to be segmented.

[0062] like Figure 2 As shown in the figure, the large language model outputs multiple circumscribed polygon coordinates, i.e., position contour coordinates (such as Figure 2 By combining the coordinates of the circumscribed polygons, we can get the coordinates of the circumscribed polygon outline. The annotations are reflected on the image of the bathroom, and the washbasin can be segmented from the image of the bathroom. That is, the washbasin in the bathroom image can be located, thus completing the positioning task.

[0063] It should be noted that the coordinates of the circumscribed polygon are also generated under the supervised learning of the true label of the circumscribed polygon (that is, the true contour coordinate label of the image to be segmented). This supervision process is completed by constructing the coordinate cross entropy loss value, which will be explained below and will not be repeated here.

[0064] Through the embodiment of the present application, the feature sequence composed of the image features to be segmented and the question text features is used by the large language model to generate the object category and structural features of the object being referred to. The pre-trained structured feature extraction network is called to extract the real structural features of the object being referred to, and under the supervision of the real structural features, the object category and implicit structural features are predicted and processed to obtain the circumscribed polygon contour coordinates of the object being referred to in the image to be segmented, thereby segmenting the object being referred to in the image to be segmented. Therefore, only by designing a pre-trained structured feature extraction network, the large language model is supervised to directly complete the segmentation of the object being referred to, and the target positioning task for the image to be segmented is completed, which effectively improves the accuracy of target positioning.

[0065] The embodiments of the present invention can achieve the above technical effects, and the importance of the pre-trained structured feature extraction network is self-evident. This network plays a role of supervised generation when the large language model outputs implicit structural features, and is pre-trained and endowed with the ability to extract the implicit structural features of objects in the coordinate points. The training process of the pre-trained structured feature extraction network is introduced below.

[0066] First of all, the function of the structured feature extraction network is to extract features, and the network structure can be an encoder, specifically a Transformer network model. Since the structural features extracted by the structured feature extraction network are used to guide and ultimately generate contour coordinates, the present invention trains the structured feature extraction network through a set of coordinate points. Therefore, the mask labels obtained here are randomly masked for the real contour coordinate labels referring to the objects. The random masking also masks out some of the coordinate points and leaves some visible coordinate points. The purpose of masking is to train the structured feature extraction network to accurately predict the shape contour of the target even when there is no specific shape contour (because some of the coordinate points are masked), thereby extracting structured features. The mask labels can be used to train the structured feature extraction network, see Figure 3 , Figure 3 This is a training process diagram of the pre-trained structured feature extraction network provided by the present invention. Figure 3 Let’s introduce the specific training process.

[0067] like Figure 3 As shown, firstly, a point token sample is constructed according to the mask label, where the point token sample includes the visible point token ( Figure 3 ) and occlusion point tokens ( Figure 3The gray dots in the figure), a token is actually a vector, a point token is also a coordinate point vector, and each occluded point token here is a shared, learnable vector, and the visible point token is an actual fixed point coordinate vector. Since the encoder (Transformer network model) is only applicable to visible, unoccluded coordinate points (that is, visible point tokens), the encoder's task is to predict whether the missing point (that is, the occluded point token) exists.

[0068] Next, the point token samples are directly input into the pre-trained structured feature extraction network, and each point token sample is encoded through the pre-trained structured feature extraction network to obtain the embedded position features of the point token samples, and the embedded position features are fused with the point token samples to obtain the encoded point tokens.

[0069] like Figure 3 As shown in the figure, the point token samples are directly input into the encoder (Transformer network model). The encoder first determines the embedded position features of each point token sample by encoding, and then fuses the embedded position features with the point token sample, that is, integrates the position features into the point token sample to obtain the encoded point token. In this way, the encoder can determine the spatial relative position information of each point token sample based on the encoded point token.

[0070] After the encoder completes encoding, it calls a lightweight decoder to decode each coded point token to obtain the to-be-reconstructed point token of the coded point token and the number of tokens of the to-be-reconstructed point token. Here, the coded point tokens include the encoded visible point tokens and occluded point tokens, which can be directly input into the lightweight decoder for decoding. The decoder result can still be a Transformer network model. The reason why it is lightweight is that there is only one layer of Transformer in the Transformer network model, and the structure is simple, so it becomes a lightweight decoder. The decoder is only used here when training the structured feature extraction network to perform the coordinate point reconstruction task to obtain the training results to construct the loss function. In actual application, there is no decoder in the pre-trained structured feature extraction network, only an encoder.

[0071] Here, the decoder architecture can be flexibly designed and is independent of the encoder design. And only one layer of Transformer is used, which is not only simple in structure, but also narrower and shallower than the encoder, and can reduce the amount of computation of the decoder for each token to less than 10% of the encoder. Through this asymmetric design of the encoder and decoder, the lightweight decoder can process the complete set of point tokens, significantly reducing the training time of the structured feature extraction network.

[0072] After the decoder completes decoding, it reconstructs the tokens of the points to be reconstructed according to the number of tokens, obtains a set of reconstructed points, and determines the mean square error between the point token samples and the set of reconstructed points. Here, the decoder obtains the tokens of the points to be reconstructed and the number of tokens of the tokens of the points to be reconstructed. This number of tokens can be used to determine whether the encoder has correctly predicted the occluded point tokens, that is, whether all missing points have been predicted. Next, the decoder will reconstruct the tokens of the points to be reconstructed according to the number of tokens. The reconstruction process uses the last linear layer of the decoder to perform linear projection on the tokens of the points to be reconstructed. The final projection mapping obtains the predicted target, that is, the set of reconstructed points. This set of reconstructed points is used as the forward propagation result of the structured feature extraction network.

[0073] After the reconstruction process is completed, the mean square error between the determined point token sample and the reconstructed point set is calculated. Here, the difference between the actual result and the predicted result is compared, and the difference is reflected by the mean square error to construct the loss function.

[0074] Finally, the mean square error is used as the training loss value of the pre-trained structured feature extraction network, and the training loss value is back-propagated in the pre-trained structured feature extraction network to update the parameters of the pre-trained structured feature extraction network. This process specifically back-propagates the training loss value to the decoder and encoder, and updates the parameters of the decoder and encoder by calculating the gradient through the gradient descent algorithm. When the loss function converges or reaches the specified training round, the training ends. The trained encoder can be used as a pre-trained structured feature extraction network, which can be used to accurately extract the structured features of the object.

[0075] Therefore, when training the structured feature extraction network, the embodiment of the present invention designs an encoder and decoder with an asymmetric structure. The encoder only acts on part of the observed signal (unobstructed coordinate points), and the decoder is a lightweight structure. In addition, coordinate points are randomly obscured when constructing training samples: this method of randomly extracting obscured points follows a uniform distribution to eliminate redundancy and creates a task that cannot be simply extrapolated through visible neighboring points.

[0076] In some embodiments, the coordinates of the circumscribed polygon outline of the reference object in the image to be segmented can also be predicted by calling the reference segmentation model, that is, the image to be segmented and the question text are directly input into the reference segmentation model, and the model directly outputs the coordinates of the circumscribed polygon outline of the reference object in the image to be segmented. The reference segmentation model includes a pre-trained visual model and a large language model, and the large language model needs to be trained. The purpose of training is to enable the large language model to accurately generate implicit structural features. Therefore, training the reference segmentation model is also training the large language model. The training process of the reference segmentation model is introduced below.

[0077] First, obtain the image sample and the question description text corresponding to the image sample, where the question description text is used to indicate the referred object in the image sample. The question description text paradigm is "Please find {description / semantics} in the picture", where "picture" refers to the image sample, and "description / semantics" is the description of the instance of the referred object, for example: "Please find the place where you can wash your hands in the picture" or "Please find the red object in the picture".

[0078] Then, the text encoder can be called to encode the question description text to obtain the question text features, and the pre-trained visual model can be called to encode the image sample to obtain the image sample features. Although the pre-trained visual model here is part of the reference segmentation model, it does not need to be trained and the parameters are not updated.

[0079] Next, the image sample features are combined with the question text features to generate a sequence of image and text features, and the image and text feature sequence is input into the large language model to output the descriptive feature text of the object being referred to (denoted as ), which describes the feature text including the predicted object category of the object being referred to and the generated predicted structured features (that is, implicit features, denoted as ).

[0080] Then, the large language model is called to predict the location of the object prediction category and the predicted structural features, generating the predicted circumscribed polygon coordinates of the object in the image sample (denoted as ).

[0081] Of course, the predicted structured features here are still supervised by the pre-trained structured feature extraction network. The supervision method here is through the thought chain method. The pre-trained structured feature extraction network is first called to extract the implicit training structure features of the object being referred to from the image sample, and this implicit training structure feature is added to the image sample as a thought chain, as the implicit object structure feature in the image sample. The thought chain is input into the large language model along with the image and text feature sequence to guide the large language model to output accurate predicted structured features. In this process, the predicted structured features are compared and learned by using the thought chain (i.e., implicit training structure features) as a reference, so as to generate more accurate predicted structured features, which are recorded as .

[0082] When calling the large language model to predict the object category and the predicted structural features for position prediction, the large language model is used to infer the implicit training structural features, the object prediction category and the predicted structural features, and finally obtain each contour position coordinate point of the object being referred to. With the assistance of the implicit training structural features (i.e., the thinking chain), the accuracy of the contour position coordinate points can be effectively improved. After determining each contour position coordinate point, the predicted circumscribed polygon coordinates of the object being referred to in the image sample can be determined based on the contour position coordinate points, as the prediction result of the forward propagation of the reference segmentation model during the training process. Next, it is necessary to construct a loss function based on this prediction result, that is, to construct the final loss value L of the reference segmentation model based on the predicted circumscribed polygon coordinates.

[0083] In the embodiment of the present invention, the final loss value L of the segmentation model is represented by the structural feature loss value (denoted as ), text cross entropy loss value (denoted as ) and the coordinate cross entropy loss value (denoted as ). The following describes the process of constructing the final loss value.

[0084] First, based on the predicted structural features As well as the true contour coordinate label C, the structural feature loss value of the generation segmentation model , where the structural feature loss value is Structural features can be predicted computationally The mean square error of the real structured feature F is obtained. Since the image sample carries the real contour coordinate label C of the object being referred to in the embodiment of the present invention, the pre-trained structured feature extraction network is called here to extract the structural features of the real contour coordinate label C of the object being referred to, and the real structured feature F of the object being referred to is obtained. The extraction process is to directly call the pre-trained structured feature extraction network to extract features. Then determine the real structured feature F and the predicted structured feature The mean square error is used as the structural feature loss value of the segmentation model. The calculation process can be expressed as the following formula (1):

[0085] (1)

[0086] In the above formula (1), MSE represents a function for calculating the mean square error.

[0087] Then, the text cross entropy loss value of the reference segmentation model is constructed based on the descriptive feature text and the real text label. Specifically, during the training of the reference segmentation model, the descriptive feature text of the object being referred to also carries the real text label Y. The descriptive feature text here is the descriptive feature text of the object being referred to output by the large language model based on the image-text feature sequence. The real text label Y can be generated by calling other pre-trained reference segmentation models. Therefore, the descriptive feature text is calculated here. The difference between the real text label Y and the real text label Y is used to calculate the cross entropy value, and then the text cross entropy loss value of the reference segmentation model is constructed. , the calculation process can be expressed as the following formula (2):

[0088] (2)

[0089] In the above formula (2), CE represents a function for calculating the cross entropy value.

[0090] For the coordinate cross entropy loss value , the embodiment of the present invention is based on the predicted circumscribed polygon coordinates With the true contour coordinate label C, the coordinate cross entropy loss value of the segmentation model is constructed. The construction process is to first calculate the predicted circumscribed polygon coordinates The difference between the true contour coordinate label C and the cross entropy value is calculated based on this difference to construct the coordinate cross entropy loss value , the calculation process can be expressed as the following formula (3):

[0091] (3)

[0092] In the above formula (3), CE represents a function for calculating the cross entropy value.

[0093] After calculating the three loss values ​​through the above formulas (1) to (3), we can then use the structural feature weights to Loss value of structural features Weighted to get the weighted loss value of structural features , through the text weight Cross entropy loss value for text Weighted to get the text weighted cross entropy loss value , and by coordinate weights Cross entropy loss value for coordinates Weighted to get the coordinate weighted cross entropy loss value .

[0094] The goal of setting weights here is to focus on different learning objectives, so the weights set for each loss value are different. However, the loss calculation method for text and the loss calculation method for the circumscribed polygon contour coordinates are the same, both of which calculate cross entropy. The difference between them is that the model's prediction of the circumscribed polygon contour coordinates is more important for the entire task, so the coordinate cross entropy loss value Will be given a greater weight in loss calculation.

[0095] Finally, the weighted loss value of the structural feature, the weighted cross entropy loss value of the text, and the weighted cross entropy loss value of the coordinate are summed to obtain the final loss value L of the reference segmentation model, which is expressed as the following formula (4):

[0096] (4)

[0097] After the final loss value L is determined, the final loss value is used to back-propagate in the reference segmentation model during training to update the parameters of the reference segmentation model. Here, in each training round during the training process, the final loss value is used to back-propagate in the reference segmentation model, and the gradient descent algorithm is called to optimize the gradient. The parameters of the reference segmentation model are updated using the gradient update, that is, the parameters of the large language model are updated (the pre-trained visual model does not update the parameters). When the final loss function L converges or reaches the specified training round, the training ends. The trained large language model and the pre-trained visual language model can be used as a pre-trained reference segmentation model, which can accurately segment the reference object from the image to be segmented according to the question text describing the reference object, and annotate it to complete the positioning task.

[0098] In an embodiment of the present invention, a feature sequence composed of the features of the image to be segmented and the features of the question text is used by a large language model to generate an object category and structural features of the object being referred to. A pre-trained structured feature extraction network is called to extract the real structural features of the object being referred to, and under the supervision of the real structural features, the object category and implicit structural features are predicted and processed to obtain the coordinates of the circumscribed polygon contour of the object being referred to in the image to be segmented, thereby segmenting the object being referred to in the image to be segmented. Therefore, only by designing a pre-trained structured feature extraction network, the large language model is supervised to directly complete the segmentation of the object being referred to, and the target positioning task for the image to be segmented is completed, which effectively improves the accuracy of target positioning.

[0099] The following is a description of the generative referential segmentation device based on implicit structural features provided by the present invention. The generative referential segmentation device based on implicit structural features described below and the generative referential segmentation method based on implicit structural features described above can be referred to each other.

[0100] Figure 4 is a schematic diagram of the structure of a generative referential segmentation device based on implicit structural features provided by the present invention, such as Figure 4As shown, the present invention provides a generative referential segmentation device based on implicit structural features, and the device specifically includes: an acquisition module 401, an encoding module 402, a generation module 403, and a prediction module 404. The acquisition module 401 is used to acquire the image to be segmented and the question text, wherein the question text is used to describe the reference object in the image to be segmented, and the image to be segmented carries the real contour coordinate label of the reference object; the encoding module 402 is used to encode the question text to obtain the text feature of the question text, call the pre-trained visual model to encode the image to be segmented to obtain the image feature of the image to be segmented, and merge the text feature with the image feature to obtain a feature sequence; the generation module 403 is used to call the pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and input the feature sequence into the large language model, supervise the large language model through the real structural features, and generate the object category and implicit structural features of the reference object; the prediction module 404 is used to predict the object category and the implicit structural features to obtain the circumscribed polygon contour coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon contour coordinates are used to segment the reference object in the image to be segmented.

[0101] Figure 5 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the generative reference segmentation method based on implicit structural features, which includes: obtaining an image to be segmented and a question text, wherein the question text is used to describe the reference object in the image to be segmented, and the image to be segmented carries the real contour coordinate label of the reference object; encoding the question text to obtain the text feature of the question text, calling the pre-trained visual model to encode the image to be segmented to obtain the image feature of the image to be segmented, and merging the text feature with the image feature to obtain a feature sequence; calling the pre-trained structured feature extraction network to extract the real structural feature of the real contour coordinate label, and inputting the feature sequence into a large language model, supervising the large language model through the real structural feature, and generating the object category and implicit structural feature of the reference object;

[0102] The object category and the implicit structural features are predicted and processed to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

[0103] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the generative reference segmentation method based on implicit structural features provided by the above methods. The method includes: obtaining an image to be segmented and a question text, wherein the question text is used to describe the reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object; encoding the question text to obtain text features of the question text, calling a pre-trained visual model to encode the image to be segmented Encoding processing is performed to obtain image features of the image to be segmented, and the text features are merged with the image features to obtain a feature sequence; a pre-trained structured feature extraction network is called to extract the real structural features of the real contour coordinate label, and the feature sequence is input into a large language model, and the large language model is supervised by the real structural features to generate the object category and implicit structural features of the referenced object; the object category and the implicit structural features are predicted to obtain the circumscribed polygon contour coordinates of the referenced object in the image to be segmented, wherein the circumscribed polygon contour coordinates are used to segment the referenced object in the image to be segmented.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the generative reference segmentation method based on implicit structural features provided by the above-mentioned methods, the method comprising obtaining an image to be segmented and a question text, wherein the question text is used to describe the reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object; encoding the question text to obtain text features of the question text, calling a pre-trained visual model to encode the image to be segmented to obtain the image to be segmented. The method comprises the following steps: extracting the real structural features of the real contour coordinate label by calling a pre-trained structured feature extraction network, supervising the large language model through the real structural features, and generating the object category and implicit structural features of the referenced object; predicting the object category and the implicit structural features, and obtaining the circumscribed polygon contour coordinates of the referenced object in the image to be segmented, wherein the circumscribed polygon contour coordinates are used to segment the referenced object in the image to be segmented.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generative coreference segmentation method based on implicit structural features, characterized in that: The method comprises: Acquire an image to be segmented and a question text, wherein the question text is used to describe a reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object; Encoding the question text to obtain text features of the question text, calling a pre-trained visual model to encode the image to be segmented to obtain image features of the image to be segmented, and merging the text features with the image features to obtain a feature sequence; Calling a pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and inputting the feature sequence into a large language model, supervising the large language model through the real structural features, and generating the object category and implicit structural features of the object being referred to, wherein the pre-trained structured feature extraction network is obtained by training with a mask label, and the mask label is obtained by randomly masking the real contour coordinate label of the object being referred to, and the training process of the pre-trained structured feature extraction network includes: Constructing a point token sample according to the mask label, wherein the point token sample includes a visible point token and an occluded point token; Encoding each of the point token samples through the pre-trained structured feature extraction network to obtain an embedded position feature of the point token sample, and fusing the embedded position feature with the point token sample to obtain an encoded point token; Calling a lightweight decoder to decode each of the code point tokens to obtain a to-be-reconstructed point token of the code point token and the number of tokens of the to-be-reconstructed point token; Reconstructing the point tokens to be reconstructed according to the number of tokens to obtain a reconstructed point set, and determining a mean square error between the point token samples and the reconstructed point set; Using the mean square error as a training loss value of the pre-trained structured feature extraction network, and performing back propagation in the pre-trained structured feature extraction network through the training loss value to update parameters of the pre-trained structured feature extraction network; The object category and the implicit structural features are predicted and processed to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

2. The generative coreference segmentation method based on implicit structural features according to claim 1, characterized in that: The coordinates of the circumscribed polygon outline of the reference object in the image to be segmented are predicted by calling a reference segmentation model, wherein the reference segmentation model includes a pre-trained visual model and a large language model, and the training process of the reference segmentation model includes: Acquire an image sample and a question description text corresponding to the image sample, wherein the question description text is used to indicate a referred object in the image sample; Encoding the question description text to obtain question text features, and calling the pre-trained visual model to encode the image sample to obtain image sample features; Merging the image sample features with the question text features to generate a graphic feature sequence, and inputting the graphic feature sequence into the large language model to output a descriptive feature text of the referred object, wherein the descriptive feature text includes an object prediction category and a predicted structural feature of the referred object; Calling the large language model to perform position prediction on the predicted category of the object and the predicted structural features, and generating predicted circumscribed polygon coordinates of the referred object in the image sample; A final loss value of the representative segmentation model is constructed based on the predicted circumscribed polygon coordinates, and the final loss value is used to perform back propagation in the representative segmentation model to update the parameters of the representative segmentation model.

3. The generative coreference segmentation method based on implicit structural features according to claim 2, characterized in that: The image sample also includes a thought chain, which is an implicit training structural feature of the object being referred to. The implicit training structural feature is extracted from the image sample by calling a pre-trained structured feature extraction network. The calling of the large language model predicts the position of the object prediction category and the predicted structural feature to generate the predicted circumscribed polygon coordinates of the object being referred to in the image sample, including: Calling the large language model to perform reasoning processing on the implicit training structural features, the object prediction category and the predicted structural features to obtain each contour position coordinate point of the referred object; The predicted circumscribed polygon coordinates of the designated object in the image sample are determined according to the contour position coordinate points.

4. The generative coreference segmentation method based on implicit structural features according to claim 2, characterized in that: The descriptive feature text of the referred object carries a real text label, the image sample carries a real contour coordinate label of the referred object, and the final loss value of the reference segmentation model is constructed based on the predicted circumscribed polygon coordinates, including: Based on the predicted structural features and the true contour coordinate labels, constructing a structural feature loss value representing the segmentation model; Constructing a text cross entropy loss value of the reference segmentation model according to the descriptive feature text and the real text label; Based on the predicted circumscribed polygon coordinates and the true contour coordinate labels, constructing a coordinate cross entropy loss value of the reference segmentation model; The structural feature loss value is weighted by the structural feature weight to obtain a structural feature weighted loss value, the text cross entropy loss value is weighted by the text weight to obtain a text weighted cross entropy loss value, and the coordinate cross entropy loss value is weighted by the coordinate weight to obtain a coordinate weighted cross entropy loss value; The structural feature weighted loss value, the text weighted cross entropy loss value, and the coordinate weighted cross entropy loss value are summed to obtain a final loss value of the reference segmentation model.

5. The generative coreference segmentation method based on implicit structural features according to claim 4, characterized in that: The step of constructing the structural feature loss value representing the segmentation model based on the predicted structural features and the true contour coordinate label includes: Calling a pre-trained structured feature extraction network to extract structure features from the real contour coordinate label to obtain real structured features; The mean square error between the true structured feature and the predicted structured feature is determined, and the mean square error is used as the structured feature loss value of the reference segmentation model.

6. A generative coreference segmentation device based on implicit structural features, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be segmented and a question text, wherein the question text is used to describe a reference object in the image to be segmented, and the image to be segmented carries a real contour coordinate label of the reference object; An encoding module is used to encode the question text to obtain text features of the question text, call a pre-trained visual model to encode the image to be segmented to obtain image features of the image to be segmented, and merge the text features with the image features to obtain a feature sequence; A generation module is used to call a pre-trained structured feature extraction network to extract the real structural features of the real contour coordinate label, and input the feature sequence into a large language model, supervise the large language model through the real structural features, and generate the object category and implicit structural features of the object referred to, wherein the pre-trained structured feature extraction network is obtained by training with a mask label, and the mask label is obtained by randomly masking the real contour coordinate label of the object referred to, and the training process of the pre-trained structured feature extraction network includes: Constructing a point token sample according to the mask label, wherein the point token sample includes a visible point token and an occluded point token; Encoding each of the point token samples through the pre-trained structured feature extraction network to obtain an embedded position feature of the point token sample, and fusing the embedded position feature with the point token sample to obtain an encoded point token; Calling a lightweight decoder to decode each of the code point tokens to obtain a to-be-reconstructed point token of the code point token and the number of tokens of the to-be-reconstructed point token; Reconstructing the point tokens to be reconstructed according to the number of tokens to obtain a reconstructed point set, and determining a mean square error between the point token samples and the reconstructed point set; Using the mean square error as a training loss value of the pre-trained structured feature extraction network, and performing back propagation in the pre-trained structured feature extraction network through the training loss value to update parameters of the pre-trained structured feature extraction network; A prediction module is used to predict the object category and the implicit structural features to obtain the circumscribed polygon outline coordinates of the reference object in the image to be segmented, wherein the circumscribed polygon outline coordinates are used to segment the reference object in the image to be segmented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the generative reference segmentation method based on implicit structural features as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the generative reference segmentation method based on implicit structural features as claimed in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the generative reference segmentation method based on implicit structural features as claimed in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Two-stage image multi-task processing method based on function module

    CN117095231A

  • Conversational target positioning method and device based on arbitrary granularity text input

    CN118350464A