A target analysis method and system based on multi-modal information collaborative enhancement

Abstract

This invention discloses a multimodal information collaborative enhancement method and system for target parsing. The method first dynamically extracts local features from the original image from multiple perspectives through radial bias sampling, thereby constructing a rich and complementary set of visual features. Next, it calculates the affinity matrix between implicit text features and multi-view visual features to obtain the region probability distribution and calculate its information entropy. Then, a heatmap generated based on the information entropy dynamically highlights key regions. The framework generates candidate boxes based on anchor point features and combines the heatmap to optimize the final localization result, completing a closed loop from feature alignment to target localization. The entire model performs backpropagation and iterative optimization by accumulating the contrast alignment loss and the detection box ranking loss, continuously enhancing the model's localization performance until training converges. This end-to-end process ensures deep collaboration between visual and text signals at multiple levels and in multiple steps, ultimately achieving highly robust weakly supervised target localization.

Description

Technical Field

[0001] This invention belongs to the field of visual-language multimodal understanding, and specifically relates to a target parsing method and system for multimodal information collaborative enhancement. Background Technology

[0002] The Representational Expression Understanding (REC) task in visual-language multimodal understanding aims to accurately locate target objects in images based on natural language descriptions. Based on the strength of the supervision signal, existing REC methods can be divided into three main paradigms: fully supervised, weakly supervised, and unsupervised. Fully supervised methods rely on a large amount of precisely labeled bounding boxes and category labels for training. For example, models like TransVG++ directly regress the coordinates of the target bounding box under the guidance of a given text description by designing a complex cross-modal fusion module. While these methods achieve high accuracy, their model training heavily depends on a large amount of manually labeled bounding box data, which is time-consuming, labor-intensive, and costly, greatly limiting their generalization ability in data-scarce scenarios. To reduce the dependence on fine-grained annotations, weakly supervised REC methods have emerged, using only image-text pairs as supervision signals without any object-level bounding box annotations. This type of method can be further subdivided into two-stage and one-stage approaches.

[0003] Early two-stage weakly supervised methods typically first generate category-independent region proposals, then calculate the similarity between each proposed region and the text description, and select the region with the highest score as the prediction result. While reducing the need for bounding box annotations, these methods heavily rely on the quality of external region proposals and are prone to error accumulation due to low-quality candidate regions. In recent years, one-stage weakly supervised methods have borrowed the image-text alignment prior from CLIP, typically using a pre-trained visual-language base model as an encoder to map images and text to the same semantic space, and locating the target through global feature matching or simple cross-attention mechanisms. However, this simple extraction of global features from the entire image often includes a large amount of background information or only incomplete object features, lacking the ability to focus on target details, resulting in insufficient model responsiveness to fine-grained attribute descriptions. Some studies have introduced bottom-up attention mechanisms to attempt to extract more discriminative local region features from images. However, such improvements often rely on additional pre-trained detectors or complex multi-stage pipelines, and the quality of their region proposals directly limits the upper limit of the final performance. While the methods described above achieve better local visual features, their interaction with text features often relies on simple feature concatenation or global attention mechanisms, failing to achieve deep, bidirectional cross-modal semantic alignment. This results in the model's utilization of text features remaining superficial, failing to fully deconstruct the semantic structure of the description (such as subjects, attributes, and relationships) and to engage in precise, fine-grained interactions with corresponding regions in the image. Consequently, it performs poorly when handling complex contextual reasoning.

[0004] Therefore, how to construct an innovative model architecture and training mechanism to dynamically extract multi-view visual features and interact with diverse implicit text features, thereby fully mining and utilizing the rich semantic information contained in the referential expression, achieving fine-grained cross-modal semantic alignment, and ultimately achieving high-precision target localization, is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical solution of this invention aims to solve the core challenge in the weakly supervised scene understanding task of multimodal information interaction: that is, how to achieve fine-grained semantic parsing of natural language descriptions and accurate localization of target objects in images using only image-text pairs of data.

[0006] The technical solution provided by this invention is as follows:

[0007] Firstly, a target parsing method with multimodal information collaborative enhancement includes:

[0008] Step 1: Extracting multi-view features and global features from the original image;

[0009] Step 2: Implicit text expression generation;

[0010] Step 3: Calculate the affinity matrix between implicit text features and multi-view visual features to obtain the region probability distribution and calculate the information entropy;

[0011] Step 4: Generate a heatmap based on information entropy, and designate image regions with information entropy values ​​above a set threshold as key regions;

[0012] Step 5: Apply contrast alignment loss to the most relevant regions and implicit text features. The most relevant regions are image regions in the key regions that are identified by at least one implicit text with a preset high probability and whose information entropy is non-zero.

[0013] Step 6: Use the heatmap obtained in Step 4 to filter anchor point features, input the anchor point features into the detection head to generate candidate boxes, and then determine the positioning result from the candidate boxes;

[0014] Step 7: Accumulate the comparison alignment loss and the detection box sorting loss, backpropagate to update the end-to-end model parameters from Step 1 to Step 6, return to Step 2 to continue iterating until the set number of iterations is reached, training stops, and the final localization result is obtained.

[0015] The framework first dynamically extracts multi-view local features from the original image through radial bias sampling, thereby constructing a rich and complementary set of visual features. Next, it calculates the affinity matrix between implicit text features and multi-view visual features to obtain the region probability distribution and calculate its information entropy. Then, a heatmap generated based on the information entropy dynamically highlights key regions.

[0016] The framework generates candidate bounding boxes based on anchor point features and optimizes the final localization result by combining heatmaps, completing a closed loop from feature alignment to target localization. The entire model performs backpropagation and iterative optimization by accumulating the contrast alignment loss and the detection box ranking loss, continuously improving the model's localization performance until training converges. This end-to-end process ensures deep collaboration between visual and textual signals at multiple levels and steps, ultimately achieving highly robust weakly supervised target localization.

[0017] Furthermore, the implicit text expression generation refers to:

[0018] First, enter the reference expression. The original text features are obtained through a text encoder. ;

[0019] Next, by randomly masking the parts other than the main body obtained from syntactic analysis, a set of L implicit attribute tokens is initialized. As a property for generating implicit diversification, and Z 0 The optimized implicit attribute tokens are obtained by using a self-attention mechanism. , Z represents0 A D-dimensional vector with index l in the middle, R D Represent a D-dimensional vector;

[0020] The implicit attribute tokens initialized at this point lack the necessary semantic associations and structural constraints, so they are obtained through information exchange and integration using a self-attention mechanism. ;

[0021] Z 0 The input self-attention mechanism exchanges information by calculating the similarity between tokens, aggregating relevant semantics, and obtaining structured implicit attribute tokens. , ;

[0022] For example, the main part of the text "Xiaoming sits in the corner" is Xiaoming. Keeping the main part unchanged (based on dependency parsing, the core noun phrases of the sentence are automatically identified as the main part), the other parts of the sentence are masked, and then a series of operations are used to generate diverse expressions, but the same content is expressed.

[0023] Then, calculate the features of the original text. Global features of the original image Based on the similarity, text-related image features were initially selected. :

[0024] ;

[0025] Among them, threshold This is the average of all similarities in the current batch. Indicates the cosine similarity between variables; Represents global features of the original image One of the eigenvector components;

[0026] Then, the image features related to the text are... As keys and values, implicit property tokens optimized for self-attention. As the query value, the two undergo cross-attention interaction to enhance and update the implicit attribute, resulting in a semantically richer implicit attribute, namely the enhanced implicit attribute token. :

[0027] ;

[0028] CrossAttention represents the cross-attention mechanism;

[0029] Finally, the enhanced implicit attribute token Z is concatenated with the subject word features from the original text features to obtain L enhanced implicit text representations. , This represents the l-th enhanced implicit textual expression, achieving diversified enhancement of textual semantics.

[0030] Furthermore, the image multi-view feature extraction process involves extracting features from the original input image through radial offset sampling. A partial view , This represents a local view of image region i. By inputting all local views into the same VisionTransformer encoder, the corresponding multi-view visual feature set is obtained. , Represents the multi-view visual features of image region i. This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions.

[0031] Each It provides rich candidate features for dynamic alignment, enhancing the diversity of visual features.

[0032] Furthermore, the calculation process for the information entropy is as follows:

[0033] First, calculate the multi-view visual features of each implicit text representation and image region i. The cosine similarity between them forms a Affinity matrix A={A l,i}; L represents the total number of implicit text expressions, This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions;

[0034] Element A l,i Indicates the first The implicit text expression and the first Similarity between image regions;

[0035] When implicit text is expressed as enhanced implicit text, ;

[0036] Secondly, along the implicit textual expression dimension, namely the affinity matrix Normalize the image region i along the column direction:

[0037] ;

[0038] in, For a given image region , by the The probability of an implicit text expression, i.e., the probability distribution of a region; This is a temperature parameter used to control the smoothness of the distribution;

[0039] To evaluate "descriptive consistency", for each image region Calculate a probability distribution. This distribution represents the probability distribution for the region... It is expressed by every implicit text. The relative possibilities described.

[0040] Next, based on probability distribution Calculate the information entropy H of image region i. i This measures the degree of consistency among all implicit textual expressions in their description:

[0041] .

[0042] Information entropy value H i A higher information entropy value indicates that the vast majority of implicit text expressions describe the image region with high consistency, which means that the image region is a key region; a lower information entropy value indicates that the implicit text expressions describe the image region inconsistently, which means that the image region is a vague or ambiguous distractor.

[0043] Furthermore, the specific process of generating a heatmap based on information entropy is as follows:

[0044] First, the attention weights for image region i are obtained using information entropy. ;

[0045] Secondly, image regions with information entropy values ​​higher than the set information entropy threshold are given greater weight. :

[0046] ;

[0047] in, It is an indicator function. It is the set information entropy threshold, P :,i Let represent the probability distribution vector of the i-th image region across all L implicit text representations. This represents the maximum probability of describing image region i among all text expressions;

[0048] A maximum probability-based filtering mechanism is used to accurately determine whether a high-entropy region is the target region or background information; only when a region is represented by at least one implicit text with a high probability ( Only when it is recognized is its high-entropy characteristic deemed valid, thus gaining high weight.

[0049] Finally, the attention weights for each image region are... Based on their original spatial locations, they are reorganized to ultimately generate an attention weight map corresponding to the spatial dimensions of the feature map. This refers to a heat map.

[0050] Furthermore, the specific process of using the heatmap obtained in step four to filter anchor point features, inputting the anchor point features into the detection head to generate candidate boxes, and then determining the localization result from the candidate boxes is as follows:

[0051] The current heatmap is upsampled to make its size exactly the same as the original input image, thus obtaining a pixel-level semantic localization heatmap. ;

[0052] Simultaneously, image anchor feature maps are extracted from the original input image. Combined with semantic location heatmap calculate The similarity between each anchor feature and the implicit text expression is calculated, and the top K anchor features with the highest similarity scores are selected.

[0053] Next, the selected anchor point features are input into the detection head to generate a set of candidate target boxes. and the corresponding detection confidence score, for each candidate box B j Calculate its coverage area in the semantic localization heatmap Average thermal response ;

[0054] Finally, by weighted fusion of the confidence score and average response of each candidate box, the candidate box with the highest comprehensive score is selected as the candidate box localization result, thus achieving the final collaboration between the visual detector and the semantic heatmap in localization decision-making:

[0055]

[0056]

[0057] in, Indicates the balance coefficient; This represents the detection confidence score of the candidate bounding boxes output by the detection head. For the first One candidate box, Candidate boxes semantic matching score, Candidate boxes The total number of pixels contained within. This indicates the pixel position on the semantic localization heatmap. The response value at that location.

[0058] Furthermore, an information entropy-weighted contrast alignment loss is designed. Attention weights for image region i Adjusting the loss contribution of different regions:

[0059]

[0060] in, It represents the similarity of positive sample pairs, indicating the number of samples matched with the image. The similarity between implicit text and the i-th image region. is the similarity of negative sample pairs, representing the similarity between text and regions that do not match the image; B is the batch size.

[0061] The loss function is weighted. This approach enhances the alignment of key regions while mitigating the impact of ambiguous areas. This mechanism, which combines information entropy with contrastive learning, effectively drives the model to focus on key regions.

[0062] Furthermore, the contrast alignment loss and the detection box sorting loss constitute the total loss function:

[0063] ;

[0064] in, It is the balance contrast alignment loss L align And sorting loss L rank hyperparameters; L align It is the contrast alignment loss, L rank It is the ranking loss; L total Represents the total loss function; The total number of candidate boxes. For the first One candidate box, This represents the best candidate box selected at the end; This represents the overall score of candidate box B. These are the set marginal parameters used to control... The score is the smallest margin higher than the scores of the other boxes.

[0065] Encourage the final selected prediction box The overall score of the candidate box is higher than that of other candidate boxes. The core function of the ranking loss is to ensure that the bounding box predicted by the model is the one with the highest "quality" among the candidate boxes. It will penalize those whose overall score is lower than the final predicted box. Higher-resolution candidate boxes. By minimizing this loss, the model is "pushed" during training to optimize its parameters so that it selects higher-resolution candidate boxes. It achieved the highest overall score across all candidate boxes. This ensures the inherent consistency and reliability of the final predictions output by the model.

[0066] Secondly, a system based on the aforementioned multimodal information collaborative enhancement target parsing method includes:

[0067] Feature extraction module: Extraction of multi-view features and global features of the original image;

[0068] Implicit text generation module: implicit text expression generation;

[0069] Information entropy calculation module: Calculates the affinity matrix between implicit text features and multi-view visual features, obtains the region probability distribution, and calculates the information entropy;

[0070] Heatmap generation module: Generates heatmaps based on information entropy and identifies image regions with information entropy values ​​above a set threshold as key regions;

[0071] Key region alignment module: Apply contrast alignment loss to the most relevant region and implicit text features. The most relevant region is the image region in the key region that is identified by at least one implicit text with a preset high probability and has non-zero information entropy.

[0072] Candidate box localization module: It uses the heat map generated by the heat map generation module to filter anchor point features, inputs the anchor point features into the detection head to generate candidate boxes, and then determines the localization result from the candidate boxes;

[0073] Training module: Accumulate the contrast alignment loss and the detection box sorting loss, backpropagate to update the parameters of each module from end to end, and sequentially call the implicit text generation module, information entropy calculation module, heatmap generation module, key region selection module and candidate box localization module to continue iterating until the set number of iterations is reached, at which point training stops and the final localization result is obtained.

[0074] Thirdly, a computer device comprising:

[0075] One or more processors;

[0076] And a memory that stores one or more computer programs;

[0077] The processor invokes a computer program to achieve the following:

[0078] The steps of the above-described multimodal information collaborative enhancement target parsing method.

[0079] Technical effect

[0080] The technical solution of this invention designs a dual-path collaborative enhancement architecture, which enhances visual and textual features in both the visual and textual paths, respectively. Furthermore, visual information assists in the generation of the enhanced textual expression in the textual path. Then, during the multi-view feature selection process in the visual branch, information entropy helps select the image viewpoint feature that is most consistent with all implicit text, i.e., collaborative enhancement. Specifically, as follows:

[0081] In the visual path, features from multiple local views are extracted from the original image through radial bias sampling to form a set of visual features that can represent different focuses of attention, thus solving the problem of coarse visual representation granularity and inability to focus on target details in traditional methods. In the text path, a diverse set of implicit text expressions is generated by decoupling the semantic units of the input description and utilizing image context information to overcome the limitation of semantic sparsity of a single text input.

[0082] Furthermore, this method innovatively introduces an alignment mechanism based on information entropy. By calculating the regional probability distribution of implicit text features and multi-view visual features, it uses information entropy to determine the image region most relevant to the text, weakening the learning of blurry or irrelevant regions, thus achieving accurate capture of visual details and deep semantic understanding. Finally, by generating candidate boxes to locate the target through anchor point features in conjunction with semantic information, it significantly improves the model's localization accuracy and robustness in complex scenes.

[0083] The entire model undergoes backpropagation and iterative optimization by accumulating contrast alignment loss and detection box ranking loss, continuously enhancing the model's localization performance until training converges. This end-to-end process ensures deep collaboration between visual and textual signals across multiple levels and steps, ultimately achieving highly robust weakly supervised target localization. Attached Figure Description

[0084] Figure 1 This is a flowchart of an embodiment of the technical solution of the present invention. Detailed Implementation

[0085] The present invention will now be further described in conjunction with the accompanying drawings and embodiments.

[0086] The multimodal information collaborative enhancement target parsing method and system of this invention can be widely applied to various weakly supervised target localization scenarios involving visual and language interaction. For example, in intelligent retrieval scenarios using massive image libraries, when an image with a complex background is acquired and natural language text descriptions such as "find the pedestrian wearing a red coat on the left side of the image" or "the kitten in the shoe" are received, the technical solution of this invention can automatically and accurately locate and select the target object by utilizing the collaborative enhancement of text and multi-view visual features without the need for manual bounding box annotation. As another example, in human-computer interaction scenarios for intelligent robots, when the robot acquires a visual image from its current perspective and receives a user instruction such as "grab the blue mug in the corner of the table," the technical solution of this invention can quickly resolve the accurate spatial position of the mug in the image, thereby guiding the robotic arm to complete the precise grasping task.

[0087] like Figure 1 As shown, a target parsing method with multimodal information collaborative enhancement includes:

[0088] Step 1: Extracting multi-view features and global features from the original image;

[0089] The multi-view feature extraction process involves extracting features from the original input image through radial offset sampling. A partial view Sampling Center It follows a mean of radial variance with the center of the original image as the standard. A two-dimensional Gaussian distribution with discreteness: ,in, Center of the original image set Control the sampling radius, The sampling radius is... This represents a local view of the original image region i. By inputting all local views into the same VisionTransformer encoder, the corresponding multi-view visual feature set is obtained. , Represents the multi-view visual features of the original image region i. This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions.

[0090] Specifically, the input image is one containing a complex background or multiple interfering objects. For example, in an image taken indoors, containing a sofa, a carpet, and a white kitten hiding in a blue sneaker, step one uses radial offset sampling. Instead of simply extracting global features from the entire image, it dynamically extracts multiple sets of local features from different local perspectives of the image (such as the edge area of ​​the sneaker, the shadow area of ​​the shoe opening, and the area of ​​the kitten's exposed fur). These multi-view features can capture fine-grained visual information such as "fur texture" and "shoe material," providing complementary visual evidence for accurate alignment with the semantics of "kitten" and "inside the shoe" in the subsequent text description.

[0091] Step 2: Implicit text expression generation;

[0092] The implicit text expression generation refers to:

[0093] First, enter the reference expression. (For example Figure 1 The phrase "The cat inside the shoe" was processed by a text encoder to obtain the original text features. ;

[0094] Next, by randomly masking the parts other than the main body obtained from syntactic analysis, a set of L implicit attribute tokens is initialized. As a property for generating implicit diversification, and Z 0 The optimized implicit attribute tokens are obtained by using a self-attention mechanism. , Z represents 0 A D-dimensional vector with index l in the middle, R D Represent a D-dimensional vector;

[0095] The implicit attribute tokens initialized at this point lack the necessary semantic associations and structural constraints, so they are obtained through information exchange and integration using a self-attention mechanism. ;

[0096] Z 0 The input self-attention mechanism exchanges information by calculating the similarity between tokens, aggregating relevant semantics, and obtaining structured implicit attribute tokens. , ;

[0097] for example Figure 1 The main part of the text "the kitten in the shoe" is the kitten. The main part remains unchanged (based on dependency parsing, the core noun phrase of the sentence is automatically identified as the main part). The other parts of the sentence (such as "in the shoe") are masked. Then, a series of operations are performed to generate diverse expressions, but the same content is expressed.

[0098] Then, calculate the features of the original text. Global features of the original image Based on the similarity, text-related image features were initially selected. :

[0099] ;

[0100] Among them, threshold This is the average of all similarities in the current batch. Indicates the cosine similarity between variables; Represents global features of the original image One of the eigenvector components;

[0101] Then, the image features related to the text are... As keys and values, implicit property tokens optimized for self-attention. As the query value, the two undergo cross-attention interaction to enhance and update the implicit attribute, resulting in a semantically richer implicit attribute, namely the enhanced implicit attribute token. :

[0102] ;

[0103] CrossAttention represents the cross-attention mechanism;

[0104] Finally, the enhanced implicit attribute token Z is concatenated with the subject word features from the original text features to obtain L enhanced implicit text representations. , This represents the l-th enhanced implicit textual expression, which achieves diversified enhancement of textual semantics. In this example, L is 8.

[0105] Step 3: Calculate the affinity matrix between implicit text features and multi-view visual features to obtain the region probability distribution and calculate the information entropy;

[0106] The calculation process for the information entropy is as follows:

[0107] First, calculate the multi-view visual features of each implicit text representation and image region i. The cosine similarity between them forms a Affinity matrix A={A l,i}; L represents the total number of implicit text expressions, This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions;

[0108] Element A l,i Indicates the first The implicit text expression and the first Similarity between image regions;

[0109] When implicit text is expressed as enhanced implicit text, ;

[0110] Secondly, along the implicit textual expression dimension, namely the affinity matrix Normalize the image region i along the column direction:

[0111] ;

[0112] in, For a given image region , by the The probability of an implicit text expression, i.e., the probability distribution of a region; This is a temperature parameter used to control the smoothness of the distribution;

[0113] To evaluate "descriptive consistency", for each image region Calculate a probability distribution. This distribution represents the probability distribution for the region... It is expressed by every implicit text. The relative possibilities described.

[0114] Next, based on probability distribution Calculate the information entropy H of image region i. iThis measures the degree of consistency among all implicit textual expressions in their description:

[0115] .

[0116] Information entropy value H i A higher information entropy value indicates that the vast majority of implicit text expressions describe the image region with high consistency, which means that the image region is a key region; a lower information entropy value indicates that the implicit text expressions describe the image region inconsistently, which means that the image region is a vague or ambiguous distractor.

[0117] Step 4: Generate a heatmap based on information entropy;

[0118] The specific process of generating a heatmap based on information entropy is as follows:

[0119] First, the attention weights for image region i are obtained using information entropy. ;

[0120] Secondly, image regions with information entropy values ​​higher than the set information entropy threshold are given greater weight. :

[0121] ;

[0122] in, It is an indicator function. This is the set information entropy threshold, which is 0.6 in this example. P :,i Let represent the probability distribution vector of the i-th image region over all L implicit text representations, therefore This represents the maximum probability of describing region i among all text expressions;

[0123] A maximum probability-based filtering mechanism is used to accurately determine whether a high-entropy region is the target region or background information; only when a region is represented by at least one implicit text with a high probability ( Only when it is recognized is its high-entropy characteristic deemed valid, thus gaining high weight.

[0124] Finally, the attention weights for each image region are... Based on their original spatial locations, they are reorganized to ultimately generate an attention weight map corresponding to the spatial dimensions of the feature map. This refers to a heat map.

[0125] Step 5: Apply contrast alignment loss to the most relevant regions and implicit text features. The most relevant regions are image regions in the key regions that are identified by at least one implicit text with a preset high probability and whose information entropy is non-zero.

[0126] Design an information entropy-weighted contrast alignment loss. Attention weights for image region i Adjusting the loss contribution of different regions:

[0127]

[0128] in, It represents the similarity of positive sample pairs, indicating the number of samples matched with the image. The similarity between implicit text and the i-th image region. is the similarity of negative sample pairs, representing the similarity between text and regions that do not match the image; B is the batch size.

[0129] Step 6: Use the heatmap obtained in Step 4 to filter anchor point features, input the anchor point features into the detection head to generate candidate boxes, and then determine the positioning result from the candidate boxes;

[0130] The specific process is as follows:

[0131] The current heatmap is upsampled to make its size exactly the same as the original input image, thus obtaining a pixel-level semantic localization heatmap. ;

[0132] Simultaneously, image anchor feature maps are extracted from the original input image. Combined with semantic location heatmap calculate The similarity between each anchor feature and the implicit text expression is calculated, and the top K anchor features with the highest similarity scores are selected.

[0133] Next, the selected anchor point features are input into the detection head to generate a set of candidate target boxes. and the corresponding detection confidence score, for each candidate box B j Calculate its coverage area in the semantic localization heatmap Average thermal response ;

[0134] Finally, by weighted fusion of the confidence score and average response of each candidate box, the candidate box with the highest comprehensive score is selected as the candidate box localization result, thus achieving the final collaboration between the visual detector and the semantic heatmap in localization decision-making:

[0135]

[0136]

[0137] in, This is the balance coefficient, which is initialized to 0.5 in this example; This represents the detection confidence score of the candidate bounding boxes output by the detection head. For the first One candidate box, Candidate boxes semantic matching score, Candidate boxes The total number of pixels contained within. This indicates the pixel position on the semantic localization heatmap. The response value at that location.

[0138] Step 7: Accumulate the comparison alignment loss and the detection box sorting loss, backpropagate to update the end-to-end model parameters from Step 1 to Step 6, return to Step 2 to continue iterating until the set number of iterations is reached, training stops, and the final localization result is obtained.

[0139] The contrast alignment loss and the detection box sorting loss constitute the total loss function:

[0140] ;

[0141] in, It is the balance contrast alignment loss L align And sorting loss L rank hyperparameters; L align It is the contrast alignment loss, L rank It is the ranking loss; L total Represents the total loss function; The total number of candidate boxes. For the first One candidate box, This represents the best candidate box selected at the end; This represents the overall score of candidate box B. These are the set marginal parameters used to control... The score is the smallest margin higher than the scores of the other boxes.

[0142] Encourage the final selected prediction box The overall score of the candidate box is higher than that of other candidate boxes. The core function of the ranking loss is to ensure that the bounding box predicted by the model is the one with the highest "quality" among the candidate boxes. It will penalize those whose overall score is lower than the final predicted box. Higher-resolution candidate boxes. By minimizing this loss, the model is "pushed" during training to optimize its parameters so that it selects higher-resolution candidate boxes. It achieved the highest overall score across all candidate boxes. This ensures the inherent consistency and reliability of the final predictions output by the model.

[0143] Through steps one through seven above, the final localization result obtained is an optimal candidate target bounding box in the original image coordinate system (as shown in the attached figure). Figure 1 (The target bounding box is shown in the lower right corner). Taking the input image and text "The cat in the shoe" as an example, the parsing result accurately provides the pixel coordinate range of the cat in the image, achieving a precise mapping from abstract linguistic description to specific spatial geometric location under weak supervision with only image-text pair annotation. This result not only eliminates interference from irrelevant objects in the background, but also ensures complete coverage of the target cat by the bounding box and accurate response to fine-grained attributes through the synergistic effect of semantic heatmap.

[0144] Example 2

[0145] A system based on the above-mentioned multimodal information collaborative enhancement target parsing method includes:

[0146] Feature extraction module: Extraction of multi-view features and global features of the original image;

[0147] Implicit text generation module: implicit text expression generation;

[0148] Information entropy calculation module: Calculates the affinity matrix between implicit text features and multi-view visual features, obtains the region probability distribution, and calculates the information entropy;

[0149] Heatmap generation module: Generates heatmaps based on information entropy and identifies image regions with information entropy values ​​above a set threshold as key regions;

[0150] Key region alignment module: Apply contrast alignment loss to the most relevant region and implicit text features. The most relevant region is the image region in the key region that is identified by at least one implicit text with a preset high probability and has non-zero information entropy.

[0151] Candidate box localization module: It uses the heat map generated by the heat map generation module to filter anchor point features, inputs the anchor point features into the detection head to generate candidate boxes, and then determines the localization result from the candidate boxes;

[0152] Training module: Accumulate the contrast alignment loss and the detection box sorting loss, backpropagate to update the parameters of each module from end to end, and sequentially call the implicit text generation module, information entropy calculation module, heatmap generation module, key region selection module and candidate box localization module to continue iterating until the set number of iterations is reached, at which point training stops and the final localization result is obtained.

[0153] It should also be understood that the specific implementation process of each module is described in the above method. This invention will not repeat it here. The above division of functional modules is only for illustrative purposes. In some embodiments, some functional modules can be combined and some functional modules can be separated. Each functional module can be implemented in software, hardware, or a combination of software and hardware. The software and hardware devices include, but are not limited to, general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0154] Example 3

[0155] A computer device, comprising:

[0156] One or more processors;

[0157] And a memory that stores one or more computer programs;

[0158] The processor invokes a computer program to achieve the following:

[0159] The steps of the above-described multimodal information collaborative enhancement target parsing method.

[0160] Please refer to the explanation of the method above for the specific implementation process of each step.

[0161] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0162] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application refers to flowchart illustrations and / or instructions executed by a processor of a method, apparatus (system), and computer program product according to embodiments of this application to create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0164] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target parsing method with multimodal information collaborative enhancement, characterized in that, include: Step 1: Extracting multi-view features and global features from the original image; Step 2: Implicit text expression generation; Step 3: Calculate the affinity matrix between implicit text features and multi-view visual features to obtain the region probability distribution and calculate the information entropy; Step 4: Generate a heatmap based on information entropy, and designate image regions with information entropy values ​​above a set threshold as key regions; Step 5: Apply contrast alignment loss to the most relevant regions and implicit text features. The most relevant regions are image regions in the key regions that are identified by at least one implicit text with a preset high probability and whose information entropy is non-zero. Step 6: Use the heatmap obtained in Step 4 to filter anchor point features, input the anchor point features into the detection head to generate candidate boxes, and then determine the positioning result from the candidate boxes; Step 7: Accumulate the comparison alignment loss and the detection box sorting loss, backpropagate to update the end-to-end model parameters from Step 1 to Step 6, return to Step 2 to continue iterating until the set number of iterations is reached, training stops, and the final localization result is obtained.

2. The method according to claim 1, characterized in that, The implicit text expression generation refers to: First, enter the reference expression. The original text features are obtained through a text encoder. ; Next, by randomly masking the parts other than the main body obtained from syntactic analysis, a set of L implicit attribute tokens is initialized. As a property for generating implicit diversification, and Z 0 The optimized implicit attribute tokens are obtained by using a self-attention mechanism. , Z represents 0 A D-dimensional vector with index l in the middle, R D Represent a D-dimensional vector; Then, calculate the features of the original text. Global features of the original image Based on the similarity, text-related image features were initially selected. : ； Among them, threshold This is the average of all similarities in the current batch. Indicates the cosine similarity between variables; Represents global features of the original image One of the eigenvector components; Then, the image features related to the text are... As keys and values, implicit property tokens optimized for self-attention. As the query value, the two undergo cross-attention interaction to enhance and update the implicit attribute, resulting in a semantically richer implicit attribute, namely the enhanced implicit attribute token. : ； CrossAttention represents the cross-attention mechanism; Finally, the enhanced implicit attribute token Z is concatenated with the subject word features from the original text features to obtain L enhanced implicit text representations. , This represents the l-th enhanced implicit text expression.

3. The method according to claim 1, characterized in that, The multi-view feature extraction process involves extracting features from the original input image through radial offset sampling. A partial view , This represents a local view of image region i. By inputting all local views into the same Vision Transformer encoder, the corresponding multi-view visual feature set is obtained. , Represents the multi-view visual features of image region i. This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions.

4. The method according to claim 1, characterized in that, The calculation process for the information entropy is as follows: First, calculate the multi-view visual features of each implicit text representation and image region i. The cosine similarity between them forms a Affinity matrix A={A l,i }; L represents the total number of implicit text expressions, This indicates the number of visual features from multiple perspectives, i.e., the total number of image regions; Element A l,i Indicates the first The implicit text expression and the first Similarity of image regions; Secondly, along the implicit textual expression dimension, namely the affinity matrix Normalize the image region i along the column direction: ； in, For a given image region , by the The probability of an implicit text expression, i.e., the probability distribution of a region; This is a temperature parameter used to control the smoothness of the distribution; Next, based on probability distribution Calculate the information entropy H of image region i. i This measures the degree of consistency among all implicit textual expressions in their description: 。 5. The method according to claim 1, characterized in that, The specific process of generating a heatmap based on information entropy is as follows: First, the attention weights for image region i are obtained using information entropy. ; Secondly, image regions with information entropy values ​​higher than the set information entropy threshold are given greater weight. : ； in, It is an indicator function. It is the set information entropy threshold, P :,i Let represent the probability distribution vector of the i-th image region across all L implicit text representations. This represents the maximum probability of describing image region i among all text expressions; Finally, the attention weights for each image region are... Based on their original spatial locations, they are reorganized to ultimately generate an attention weight map corresponding to the spatial dimensions of the feature map. This refers to a heat map.

6. The method according to claim 1, characterized in that, The specific process of using the heatmap obtained in step four to filter anchor point features, inputting the anchor point features into the detection head to generate candidate boxes, and then determining the localization result from the candidate boxes is as follows: The current heatmap is upsampled to make its size exactly the same as the original input image, thus obtaining a pixel-level semantic localization heatmap. ; Extract image anchor point feature maps from the original input image. Combined with semantic location heatmap calculate The similarity between each anchor feature and the implicit text expression is calculated, and the top K anchor features with the highest similarity scores are selected. Next, the selected anchor point features are input into the detection head to generate a set of candidate target boxes. and the corresponding detection confidence score, for each candidate box B j Calculate its coverage area in the semantic localization heatmap Average thermal response ; Finally, by weighted fusion of the confidence score and average response of each candidate box, the candidate box with the highest comprehensive score is selected as the candidate box localization result, thus achieving the final collaboration between the visual detector and the semantic heatmap in localization decision-making: ； ； in, Indicates the balance coefficient; This represents the detection confidence score of the candidate bounding boxes output by the detection head. For the first One candidate box, Candidate boxes The semantic matching score, Candidate boxes The total number of pixels contained within. This indicates the pixel position on the semantic location heatmap. The response value at that location.

7. The method according to claim 1, characterized in that, Design an information entropy-weighted contrast alignment loss. Attention weights for image region i Adjusting the loss contribution of different regions: ； in, It represents the similarity of positive sample pairs, indicating the number of samples matched with the image. The similarity between implicit text and the i-th image region. is the similarity of negative sample pairs, representing the similarity between text and regions that do not match the image; B is the batch size.

8. The method according to claim 7, characterized in that, The contrast alignment loss and the detection box sorting loss constitute the total loss function: ； in, It is the balance contrast alignment loss L align And sorting loss L rank hyperparameters; L align It is the contrast alignment loss, L rank It is the ranking loss; L total Represents the total loss function; This represents the total number of candidate boxes. For the first One candidate box, This represents the best candidate box selected at the end; This represents the overall score of candidate box B. These are the set marginal parameters used to control... The score is the smallest margin higher than the scores of the other boxes.

9. A target parsing system based on the method of any one of claims 1-8, characterized in that, include: Feature extraction module: Extraction of multi-view features and global features of the original image; Implicit text generation module: Generates implicit text expressions; Information entropy calculation module: Calculates the affinity matrix between implicit text features and multi-view visual features, obtains the region probability distribution, and calculates the information entropy; Heatmap generation module: Generates heatmaps based on information entropy and identifies image regions with information entropy values ​​above a set threshold as key regions; Key region alignment module: Apply contrast alignment loss to the most relevant region and implicit text features. The most relevant region is the image region in the key region that is identified by at least one implicit text with a preset high probability and has non-zero information entropy. Candidate box localization module: It uses the heat map generated by the heat map generation module to filter anchor point features, inputs the anchor point features into the detection head to generate candidate boxes, and then determines the localization result from the candidate boxes; Training module: Accumulate the contrast alignment loss and the detection box sorting loss, backpropagate to update the parameters of each module from end to end, and sequentially call the implicit text generation module, information entropy calculation module, heatmap generation module, key region selection module and candidate box localization module to continue iterating until the set number of iterations is reached, at which point training stops and the final localization result is obtained.

10. A computer device, characterized in that: include: One or more processors; And a memory that stores one or more computer programs; The processor invokes a computer program to achieve the following: The steps of the method according to any one of claims 1-8.

Images