Hybrid prompt-based recognition method, electronic device, and readable storage medium

Abstract

This application discloses a recognition method, electronic device, and readable storage medium based on hybrid prompts. The method includes: acquiring a pair to be recognized; the pair to be recognized includes text to be recognized and an image to be recognized; determining text-assisted prompts matching the text to be recognized, image-assisted prompts matching the image to be recognized, and fusion-assisted prompts matching the pair to be recognized; acquiring text fusion features obtained by fusing the text features of the text to be recognized with the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts; using the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, performing multiple rounds of fusion of the text fusion features and the image features of the image to be recognized to obtain target fusion features; and determining the target recognition result matching the pair to be recognized based on the target fusion features. Through the above methods, this application can improve the accuracy of target recognition.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a recognition method, electronic device and readable storage medium based on hybrid cues. Background Technology

[0002] With the development of target recognition technology, schemes based on multimodal data for target recognition have been more widely used. Current technologies typically extract features from both image and text data separately and then match them to determine the recognition result. However, the accuracy of the recognition result still depends on the quality of the data, resulting in relatively low accuracy in some scenarios. Therefore, improving the accuracy of target recognition has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a recognition method, electronic device, and readable storage medium based on hybrid prompts, which can improve the accuracy of target recognition.

[0004] To address the aforementioned technical problems, a first aspect of this application provides a recognition method based on hybrid prompts, comprising: acquiring a pair to be recognized; the pair to be recognized includes text to be recognized and an image to be recognized; determining text-assisted prompts matching the text to be recognized, image-assisted prompts matching the image to be recognized, and fusion-assisted prompts matching the pair to be recognized; acquiring text fusion features obtained by fusing the text features of the text to be recognized with the text-assisted prompts, the image-assisted prompts, and the fusion-assisted prompts; using the text-assisted prompts, the image-assisted prompts, and the fusion-assisted prompts, performing multiple rounds of fusion of the text fusion features and the image features of the image to be recognized to obtain target fusion features; and determining the target recognition result matching the pair to be recognized based on the target fusion features.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The above scheme acquires a pair of objects to be identified, including the text and the image to be identified, thus obtaining multimodal data. Based on the text to be identified, text-based auxiliary prompts are determined; based on the image to be identified, image-based auxiliary prompts are determined; and based on the pair of objects to be identified, fusion auxiliary prompts are determined, relating to both text and image. The text features of the text to be identified are then fused with the text-based, image-based, and fusion auxiliary prompts to obtain text fusion features, thus enabling preliminary feature interaction and fusion between the image and text. Using the text-based, image-based, and fusion auxiliary prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds. This uses the three prompts as a medium to fuse image features with text fusion features in multiple rounds, resulting in target fusion features after deep feature fusion of the two modalities. Based on the target fusion features, the target in the pair of objects to be identified is identified. Based on the more accurate image-text matching results fed back by the target fusion features, the target recognition result is obtained, reducing dependence on data quality and improving the accuracy of the target recognition result. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0009] Figure 1 This is a flowchart illustrating one implementation of the recognition method based on hybrid prompts in this application;

[0010] Figure 2 This is a flowchart illustrating another embodiment of the recognition method based on hybrid prompts in this application;

[0011] Figure 3 This is a schematic diagram illustrating an application scenario of one implementation method for obtaining text fusion features in this application;

[0012] Figure 4 This is a schematic diagram illustrating an application scenario of one implementation method for obtaining target fusion features in this application;

[0013] Figure 5 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0014] Figure 6 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0017] The hybrid prompt-based recognition method provided in this application is used for target recognition, and its corresponding execution subject is a processing unit capable of data processing.

[0018] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the hybrid prompt-based recognition method of this application, which includes:

[0019] S101: Obtain the pair to be recognized; the pair to be recognized includes the text to be recognized and the image to be recognized.

[0020] Specifically, the process involves acquiring a pair of objects to be identified, including the text to be identified and the image to be identified, thereby obtaining multimodal data.

[0021] It is understandable that the text to be recognized and the image to be recognized in the recognition pair are matched with each other, wherein the image to be recognized is collected from the target scene and the text to be recognized matches the target scene, or the image to be recognized is obtained from a preset data source and the text to be recognized matches the preset data source.

[0022] Optionally, the text to be identified includes at least descriptive text describing the image to be identified. In some implementation scenarios, the text to be identified also includes preset text, which is used to instruct the identification of targets in the image.

[0023] S102: Determine the text auxiliary prompts for matching the text to be identified, the image auxiliary prompts for matching the image to be identified, and the fusion auxiliary prompts for matching the image to be identified, and obtain the text features of the text to be identified and the text fusion features obtained by fusing the text auxiliary prompts, image auxiliary prompts and fusion auxiliary prompts.

[0024] Specifically, based on the text to be identified, text-assisted prompts associated with the text are determined; based on the image to be identified, image-assisted prompts associated with the image are determined; based on the image to be identified, fusion-assisted prompts associated with the text and image are determined; and the text features of the text to be identified are obtained by fusing the text features of the text to be identified with the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts to obtain the text fusion features, thereby enabling the image and text to generate preliminary feature interaction fusion in the text fusion features.

[0025] It should be noted that there are three types of prompts: text-assisted prompts, image-assisted prompts, and fusion-assisted prompts. Among them, text-assisted prompts are strongly correlated with the text features corresponding to the text to be identified, image-assisted prompts are strongly correlated with the image features corresponding to the image to be identified, and fusion-assisted prompts are strongly correlated with the fusion features corresponding to the text features and the image features. The text features are the text fusion features obtained by fusing the text features with the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts.

[0026] In some implementation scenarios, features are extracted from the text and image to be identified in the pair to be identified, respectively, to obtain text features corresponding to the text to be identified and image features corresponding to the image to be identified. Based on the text features, global characters in the text to be identified are determined and text auxiliary prompts associated with the global characters are generated. The global characters include at least conjunctions and prepositions. Based on the image features, the image to be identified is divided into multiple image regions and image auxiliary prompts associated with at least some of the image regions are generated. The text features and image features are fused to obtain image-text fusion features. Based on the image-text fusion features, image-text matching information is determined and fusion auxiliary prompts associated with the image-text matching information are generated. The text features are fused with text auxiliary prompts, image auxiliary prompts, and fusion auxiliary prompts to obtain text fusion features.

[0027] In some implementation scenarios, features are extracted from both the text to be identified and the image to be identified in the pair to be identified, resulting in text features corresponding to the text to be identified and image features corresponding to the image to be identified. Based on the text features, the target text used to describe the target and the non-target text outside the target text are identified, and text auxiliary prompts matching the target text and non-target text are generated. Based on the image features, the image to be identified is divided into background and foreground regions, and image auxiliary prompts matching the background and foreground regions are generated. The text features and image features are fused to obtain image-text fusion features. Based on the image-text fusion features, the associated characters related to the image information are identified and fusion auxiliary prompts corresponding to the associated characters are generated. The text features are fused with the text auxiliary prompts, image auxiliary prompts, and fusion auxiliary prompts to obtain text fusion features.

[0028] Optionally, the current text fusion features are updated to text features, and multiple rounds of construction are performed on text-assisted prompts, image-assisted prompts, and fusion-assisted prompts to obtain the final round of text fusion features, text-assisted prompts, image-assisted prompts, and fusion-assisted prompts. This improves the accuracy of each type of prompt and ensures the accuracy of the initial feature interaction fusion between images and text in the text fusion features.

[0029] S103: Using text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds to obtain the target fusion features.

[0030] Specifically, by using text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds. Thus, using the three prompts as a medium, the image features and text fusion features are fused in multiple rounds to obtain the target fusion features after deep fusion of the features corresponding to the two modalities.

[0031] In some implementation scenarios, the text fusion features of the current round are fused with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts to obtain the text fusion features of the next round. The image features of the current round are concatenated with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, and then fused. The fused features are decomposed into text-assisted prompts, image-assisted prompts, fusion-assisted prompts, and image features for the next round. After multiple iterations, the text fusion features of the final round are fused with the image features to obtain the target fusion features. Specifically, the image features are concatenated with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, and then subjected to local and global cross-fusion, with the fusion dimension remaining unchanged. The features are then decomposed into text-assisted prompts, image-assisted prompts, fusion-assisted prompts, and image features for the next round according to the concatenation order.

[0032] In some implementation scenarios, the text-based, image-based, and fusion-based cues from the current round are merged and concatenated to obtain a concatenated cue feature. This concatenated cue feature is then fused with the text features from the current round, and the fused feature is adjusted to the same dimension as the text fusion feature to obtain the text fusion feature for the next round. The image features from the current round are then fused with the concatenated cue feature and decomposed into image features, text-based, image-based, and fusion-based cues for the next round. After multiple rounds of iteration, the text fusion feature from the final round is used as the target fusion feature. Specifically, the concatenated cue feature is adjusted to the same position as the image feature and concatenated with it before feature fusion. The fused feature is then decomposed by channel into text-based, image-based, and fusion-based cues for the next round, as well as the image feature.

[0033] S104: Based on the target fusion features, determine the target recognition result of the matching pair to be identified.

[0034] Specifically, based on target fusion features, targets in the target pair are identified, thereby obtaining target recognition results by relying on more accurate image-text matching results fed back by the target fusion features, reducing dependence on data quality.

[0035] It is understandable that target fusion features are obtained by deep fusion of the features matched by the text and the image respectively. Based on the target fusion features, the targets included in the target pair can be identified to obtain the target recognition result.

[0036] In some implementation scenarios, based on target fusion features, the target category and target coordinates of the targets included in the pair to be identified are determined, and the target category and target coordinates are used as the target identification result.

[0037] In some implementation scenarios, based on target fusion features, the bounding boxes in the images to be identified corresponding to the targets included in the pair to be identified are determined. Target labels matching the target category are then assigned to the targets, and the bounding boxes and matching target labels in the images to be identified are used as the target recognition results.

[0038] Optionally, the above method is based on a target recognition model, which includes an image encoder, a text encoder, an early fusion module, a multimodal fusion module, and a prediction module. The image encoder extracts image features, the text encoder extracts text features, the early fusion module generates text fusion features, text-based auxiliary prompts, image-based auxiliary prompts, and fusion auxiliary prompts, the multimodal fusion module generates target fusion features, and the prediction module outputs the target recognition result. Furthermore, the image encoder and text encoder are pre-trained with fixed parameters, while the early fusion module, multimodal fusion module, and prediction module undergo supervised training.

[0039] Specifically, the parameters of the pre-trained image encoder and text encoder remain unchanged. During supervised training, the parameters of the early fusion module, multimodal fusion module, and prediction module are adjusted. The early fusion module enables the initial feature interaction fusion between the image and text. The multimodal fusion module uses three types of cues as a medium to achieve deep fusion of image features and text features. Finally, the prediction module outputs the target recognition result, realizing end-to-end output from the target pair to the target recognition result, thereby improving the efficiency and accuracy of recognition.

[0040] The above scheme acquires a pair of objects to be identified, including the text and the image to be identified, thus obtaining multimodal data. Based on the text to be identified, text-based auxiliary prompts are determined; based on the image to be identified, image-based auxiliary prompts are determined; and based on the pair of objects to be identified, fusion auxiliary prompts are determined, relating to both text and image. The text features of the text to be identified are then fused with the text-based, image-based, and fusion auxiliary prompts to obtain text fusion features, thus enabling preliminary feature interaction and fusion between the image and text. Using the text-based, image-based, and fusion auxiliary prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds. This uses the three prompts as a medium to fuse image features with text fusion features in multiple rounds, resulting in target fusion features after deep feature fusion of the two modalities. Based on the target fusion features, the target in the pair of objects to be identified is identified. Based on the more accurate image-text matching results fed back by the target fusion features, the target recognition result is obtained, reducing dependence on data quality and improving the accuracy of the target recognition result.

[0041] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the recognition method based on hybrid prompts in this application. The method includes:

[0042] S201: Obtain the pair to be recognized; the pair to be recognized includes the text to be recognized and the image to be recognized.

[0043] Specifically, the identification pairs consisting of mutually matching text and images are obtained to generate multimodal data.

[0044] S202: Obtain the text features of the text to be identified and the image features of the image to be identified.

[0045] Specifically, feature recognition is performed on the text to be recognized to obtain text features, and feature recognition is performed on the image to be recognized to obtain image features.

[0046] Optionally, the text to be recognized includes descriptive text and preset text. The text string corresponding to the descriptive text is first converted into a token sequence based on the ID corresponding to the token. Then, the token is vectorized, outputting a text embedding vector sequence, which is then concatenated with the target token vector corresponding to the preset text. The concatenated text embedding vector sequence is fed into the text encoder to obtain the text features corresponding to the text to be recognized. The image to be recognized is divided into multiple image patches, which are then converted into image embedding vectors through a linear layer. These image embedding vectors are fed into the image encoder to obtain the image features corresponding to the image to be recognized. The text encoder can use the BERT (Bidirectional Encoder Representations from Transformers) model or other pre-trained models, and the image encoder can use the ViT (Visual Transformer) model or other pre-trained models.

[0047] S203: Based on text features, obtain text-based auxiliary prompts; based on image features, obtain image-based auxiliary prompts; based on both text and image features, obtain fused auxiliary prompts.

[0048] Specifically, based on text features, the information contained in the text is analyzed to obtain text-based auxiliary prompts; based on image features, the information contained in the image is analyzed to obtain image-based auxiliary prompts.

[0049] Furthermore, text features are fused with image features, and the image and text information is analyzed based on the fused features to obtain fused auxiliary prompts, enabling text and image information to achieve preliminary interaction.

[0050] It should be noted that the fusion assistance prompts based on text features and image features include: fusing text features and image features to obtain image-text fusion features; determining the image-text correlation between characters in the text to be recognized and image regions in the image to be recognized based on the image-text fusion features; setting character weights for each character in the text to be recognized based on the image-text correlations; and obtaining fusion assistance prompts based on the characters in the text to be recognized and their corresponding character weights.

[0051] Specifically, text features and image features are fused to obtain image-text fusion features. Based on the image-text fusion features, the image-text correlation between characters in the text to be identified and image regions in the image to be identified is determined. Then, based on the features obtained after image-text interaction, and with the help of information obtained from the image to be identified, words with a high probability of change are obtained from the text to be identified, that is, words with a high correlation with the target in the image are obtained, thereby determining the image-text correlation between characters and image regions.

[0052] Furthermore, based on the correlation between text and images, a character weight is assigned to each character in the text to be identified, so that characters with a strong correlation with the image are assigned higher weights. Based on the characters in the text to be identified and their corresponding character weights, a fusion auxiliary prompt is obtained, thereby locating words with a strong correlation with the target in the image.

[0053] S204: Fuse the text features with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, and adjust them to the same dimension as the text features to obtain the text fusion features.

[0054] Specifically, text features are fused with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts to achieve initial interaction between text and image features. The dimensions of the fused features are then adjusted to obtain text-fused features while ensuring that the dimensions of the text-fused features are consistent with those of the text features.

[0055] Optionally, after fusing text features with text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, and adjusting them to the same dimension as the text features to obtain text-fused features, the method further includes: updating the current text-fused features to text features; returning to the steps of obtaining text-assisted prompts based on text features, obtaining image-assisted prompts based on image features, and obtaining fusion-assisted prompts based on text features and image features, until preset conditions are met to obtain the final text-fused features, text-assisted prompts, image-assisted prompts, and fusion-assisted prompts.

[0056] Specifically, the text fusion features of the current round are used as the text features of the next round, thereby extracting and generating text fusion features, text auxiliary prompts, image auxiliary prompts and fusion auxiliary prompts in multiple rounds, improving the accuracy of text fusion features and the three types of prompts.

[0057] In some implementation scenarios, text fusion features, text-based cues, image-based cues, and fusion-based cues are generated by the earlier fusion module. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of an application scenario for obtaining text fusion features according to this application. The early fusion module includes a linear layer, a splicing layer and a probability layer, and the early fusion module also includes a route embedding vector and dynamic prompts as parameters to be adjusted.

[0058] Specifically, text-based prompts l is the set length of the prompt, d t This refers to the text feature dimension. Text-based prompts primarily provide global prompts shared across all sentences, not specific prompt features from a single sentence. Image-based prompts are derived from image embedding features processed through a linear layer, outputting P. vImage-assisted cues integrate feature information from images to aid prompt learning, serving as a supplement to information-based learning.

[0059] Furthermore, the embedding features of the text and image after encoding are each passed through two linear layers, and then concatenated along the embedding dimension. The concatenated result is represented as follows: Where, d v This is the feature dimension of the image embedding vector. During the training phase, the routing embedding vector is represented as... The dot product of f and r is passed through a probability layer to output the weights of the characters in the text to be recognized, and finally combined with learnable dynamic cues. Multiplication yields fusion aid hint P f Fusion cues learn dynamic cues within the prompt, using information gleaned from image content to learn words with high probability of change and identify words most closely related to the target in the image.

[0060] S205: Using text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds to obtain the target fusion features.

[0061] Specifically, by using text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds. Thus, using the three prompts as a medium, the image features and text fusion features are fused in multiple rounds to obtain the target fusion features after deep fusion of the features corresponding to the two modalities.

[0062] It should be noted that, by utilizing text-assisted prompts, image-assisted prompts, and fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds to obtain the target fusion feature. This includes: concatenating the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts from the current round to obtain the concatenated prompt feature; fusing the text fusion feature from the current round with the concatenated prompt feature and adjusting it to the same dimension as the text fusion feature to obtain the text fusion feature for the next round; fusing the image features from the current round with the concatenated prompt feature and decomposing it into the image features, text-assisted prompts, image-assisted prompts, and fusion-assisted prompts for the next round; returning to the step of concatenating the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts from the current round to obtain the concatenated prompt feature, until a preset number of rounds are iterated, and the text fusion feature from the final round is used as the target fusion feature.

[0063] Specifically, the text-based, image-based, and fusion-based cues of the current round are concatenated to obtain concatenated cues features, thereby integrating multiple types of cues.

[0064] Furthermore, the text fusion features of the current round are fused with the splicing prompt features and adjusted to the same dimension as the text fusion features to obtain the text fusion features of the next round. This ensures that the dimensionality of the text fusion features is consistent in each round, and that the text fusion features can be continuously fused with the splicing prompt features in each round. The image features of the current round are fused with the splicing prompt features, and the fused features are decomposed to obtain the image features, text auxiliary prompts, image auxiliary prompts, and fusion auxiliary prompts of the next round. This integrates image features into multiple types of prompts. Using multiple types of prompts as a medium, a preset number of rounds are iterated to achieve multi-round fusion of features corresponding to the data of the two modalities of text and images until the final round. The text fusion features of the final round are used as the target fusion features to improve the accuracy of the target fusion features.

[0065] It should be noted that fusing the image features of the current round with the splicing prompt features and decomposing them into image features, text-assisted prompts, image-assisted prompts, and fused prompts for the next round includes: performing feature fusion on the splicing prompt features and adjusting them to the same dimension as the image features to obtain fused prompt features; splicing the fused prompt features with the image features of the current round and then performing feature fusion, decomposing them into image features, text-assisted prompts, image-assisted prompts, and fused prompts for the next round.

[0066] Specifically, the features to be spliced ​​are fused and their dimensions adjusted to obtain fused cue features with the same dimensions as the image features, facilitating fusion with the image features. The fused cue features are then evaluated with the image features of the current round before feature fusion, thus clarifying the dimensions corresponding to the fused features. This allows the fused features to be decomposed according to dimensions, yielding the image features, text-based cueing, image-based cueing, and fused cueing for the next round, ensuring the fusion effect of image features with multiple types of cueing and the accuracy of decomposition.

[0067] In some implementation scenarios, target fusion features are generated by a multimodal fusion module. Please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario of one implementation method for obtaining target fusion features according to this application. The multimodal fusion module includes multiple Transformer layers and linear layers, wherein... Figure 4 The feature transformation process for only one round is provided adaptively.

[0068] Specifically, the text fusion features T0 and text auxiliary cues P output by the early fusion module are obtained. s Fusion Auxiliary Tips P f and image-assisted cues P vThe text fusion feature T0 is concatenated with the concatenated cue features corresponding to the three types of cue in the sequence length dimension, and then fed into a transformer layer for fusion. After adjusting the dimensions, the text fusion feature T1 for the next round is obtained. The concatenated cue features are then fused through a transformer layer to obtain the fused cue feature. The fused cue feature is then adjusted by a linear layer to obtain a new text auxiliary cue P. s ′ Fusion Auxiliary Tips P f ′ and image-assisted cues P v ′ The output dimension of the linear layer is the image feature dimension.

[0069] Furthermore, the new prompt P s ′ P f ′ and P v ′ The image features V0 are concatenated with the image features V0 along the sequence dimension and then fed into a transformer layer to decompose and obtain the image features V1 for the next round, as well as the text aids P for the next round after linear layer adjustment. s1 Fusion Auxiliary Tips P f1 and image-assisted cues P v1 Similarly, the newly generated text fusion feature T1 and the concatenated prompt features corresponding to the three types of prompts are concatenated along the sequence length dimension and fed into the next layer, iterating for a preset number of rounds.

[0070] S206: Based on the target fusion features, determine the target recognition result of the matching pair to be identified.

[0071] Specifically, based on target fusion features, targets in the target pair are identified, and the target identification result is obtained based on the more accurate image-text matching result fed back by the target fusion features.

[0072] In some implementation scenarios, based on target fusion features, the target recognition result, classification and region of the target to be recognized are determined, including: based on target fusion features, determining the target category and target location of the target included in the target to be recognized; generating target bounding boxes in the image to be recognized based on the target location, and obtaining the target recognition result including the target category and target bounding boxes.

[0073] Specifically, based on target fusion features, the categories and locations of targets included in the pair to be identified are analyzed to obtain the target category and target location matched by the target. Based on the target location, the corresponding target bounding box is generated in the image to be identified, resulting in a target recognition result that includes the target category and the target bounding box, making the target recognition result more comprehensive and intuitive.

[0074] Optionally, the target recognition result is output by the prediction module. After the target fusion features and the image to be recognized are sent to the prediction module, the prediction module outputs the predicted target class head and the target location Bbox head on the image to be recognized.

[0075] It is understandable that the above method is based on a target recognition model, which includes an image encoder, a text encoder, an early fusion module, a multimodal fusion module, and a prediction module. The image encoder is used to extract image features, the text encoder is used to extract text features, the early fusion module is used to generate text fusion features, text auxiliary prompts, image auxiliary prompts, and fusion auxiliary prompts, the multimodal fusion module is used to generate target fusion features, and the prediction module is used to output the target recognition result. Furthermore, the image encoder and text encoder are pre-trained and have their parameters fixed, while the early fusion module, multimodal fusion module, and prediction module undergo supervised training.

[0076] Optionally, the entire model is trained end-to-end using the RefCOCO, RefCOCO+, and RefCOCOg datasets, without requiring special data preprocessing. The input image size can be set to 640×640, and the text sequence length can be set to 40. If the length is less than the set sequence length, empty tokens are added after the last token to maintain consistent input length for each batch. The padding tokens are represented by a mask. In different specific implementation scenarios, both the image size and text sequence length can be customized; this application does not impose specific restrictions on them.

[0077] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure), and the processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0078] Please see Figure 6 , Figure 6 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0079] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A recognition method based on hybrid prompts, characterized in that, The method includes: Obtain the pair to be identified; the pair to be identified includes the text to be identified and the image to be identified. Determine the text assistance prompts matching the text to be identified, the image assistance prompts matching the image to be identified, and the fusion assistance prompts matching the pair to be identified; obtain the text fusion features obtained by fusing the text features of the text to be identified with the text assistance prompts, the image assistance prompts, and the fusion assistance prompts. By utilizing the text-assisted prompts, the image-assisted prompts, and the fusion-assisted prompts, the text fusion features and the image features of the image to be identified are fused in multiple rounds to obtain the target fusion features; Based on the target fusion features, the target identification result of the matching pair to be identified is determined; The target fusion feature is obtained based on the following steps: concatenating the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts of the current round to obtain a concatenated prompt feature; fusing the text-fused feature of the current round with the concatenated prompt feature and adjusting it to the same dimension as the text-fused feature to obtain the text-fused feature of the next round; fusing the image feature of the current round with the concatenated prompt feature and decomposing it into the image feature, text-assisted prompt, image-assisted prompt, and fusion-assisted prompt of the next round; returning to the step of concatenating the text-assisted prompts, image-assisted prompts, and fusion-assisted prompts of the current round to obtain the concatenated prompt feature, until a preset number of rounds are iterated, and the text-fused feature of the final round is taken as the target fusion feature.

2. The recognition method based on hybrid prompts according to claim 1, characterized in that, The step of determining the text assistance prompts matching the text to be identified, the image assistance prompts matching the image to be identified, and the fusion assistance prompts matching the image to be identified, and obtaining the text fusion features obtained by fusing the text features of the text to be identified with the text assistance prompts, the image assistance prompts, and the fusion assistance prompts, includes: Obtain the text features of the text to be identified and the image features of the image to be identified; Based on the text features, the text-assisted prompt is obtained; based on the image features, the image-assisted prompt is obtained; and based on the text features and the image features, the fused-in-place prompt is obtained. The text features are fused with the text auxiliary prompts, the image auxiliary prompts, and the fusion auxiliary prompts, and adjusted to the same dimension as the text features to obtain the text fusion features.

3. The recognition method based on hybrid prompts according to claim 2, characterized in that, The process of obtaining the fused auxiliary prompt based on the text features and the image features includes: The text features and the image features are fused to obtain image-text fusion features. Based on the image-text fusion features, the image-text correlation between characters in the text to be identified and image regions in the image to be identified is determined. Based on the image-text correlation, a character weight is set for each character in the text to be recognized, and the fusion auxiliary prompt is obtained based on the characters in the text to be recognized and their corresponding character weights.

4. The recognition method based on hybrid prompts according to claim 2, characterized in that, After fusing the text features with the text auxiliary prompts, the image auxiliary prompts, and the fusion auxiliary prompts, and adjusting them to the same dimension as the text features to obtain the text fusion features, the method further includes: Update the current text fusion features to the text features; Returning to the steps of obtaining the text assistance prompt based on the text features, obtaining the image assistance prompt based on the image features, and obtaining the fusion assistance prompt based on the text features and the image features, the process continues until preset conditions are met, resulting in the final text fusion feature, the text assistance prompt, the image assistance prompt, and the fusion assistance prompt.

5. The recognition method based on hybrid prompts according to claim 1, characterized in that, The step of fusing the image features of the current round with the splicing prompt features and decomposing them into the image features of the next round, the text-assisted prompts, the image-assisted prompts, and the fused-in-place prompts includes: The spliced ​​prompt features are fused and adjusted to the same dimension as the image features to obtain the fused prompt features; The fused prompt features are concatenated with the image features of the current round, and then feature fusion is performed. The result is decomposed into the image features, text-assisted prompts, image-assisted prompts, and fused prompts of the next round.

6. The recognition method based on hybrid prompts according to claim 1, characterized in that, The step of determining the target identification result of the target to be identified based on the target fusion features includes: Based on the target fusion features, determine the target category and target location matched by the targets included in the pair to be identified; Based on the target location, a target bounding box is generated in the image to be identified, and the target recognition result including the target category and the target bounding box is obtained.

7. The recognition method based on hybrid prompts according to any one of claims 1-6, characterized in that, The method is based on a target recognition model, which includes an image encoder, a text encoder, an early fusion module, a multimodal fusion module, and a prediction module. The image encoder extracts image features, the text encoder extracts text features, the early fusion module generates text fusion features, text-based auxiliary prompts, image-based auxiliary prompts, and fusion-based auxiliary prompts, the multimodal fusion module generates target fusion features, and the prediction module outputs the target recognition result. Furthermore, the image encoder and text encoder are pre-trained with fixed parameters, while the early fusion module, multimodal fusion module, and prediction module undergo supervised training.

8. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, the method as described in any one of claims 1-7 is implemented.

Images