General visual identification model based on cue word interaction technology

Abstract

The invention provides a general visual identification model based on a cue word interaction technology. The general visual identification model comprises an image encoder, a text cue word encoder, a visual cue word encoder, a multi-mode interaction module and a target decoding module, the multi-modal interaction module is a bidirectional interaction module and comprises a text and image interaction module, an image and text interaction module and an image and cue word interaction module. Key technologies such as the bidirectional multi-modal interaction module, the cue word detection mechanism and the interested area interaction mechanism are introduced; image features and text prompts can be subjected to deep bidirectional interaction, robust, stable and fine-grained feature support is provided for downstream tasks, and it is ensured that the model has higher accuracy and generalization performance; according to the method, a'detection cue word mechanism 'is used for positioning the target position, meanwhile, text-image modal matching is used for target semantic recognition, and the key effect is achieved in a plurality of visual understanding tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of calculation, estimation or counting, and in particular to a general visual recognition model based on prompt word interaction technology. BACKGROUND

[0002] With the rapid development of large language model technology, the field of natural language understanding has ushered in an unprecedented revolution. From deep semantic analysis to precise processing of reference resolution, to accurate recognition of named entity recognition, these traditional complex and tedious problems can now be efficiently and accurately solved in a single generation model. This revolutionary progress not only greatly improves the efficiency of natural language processing, but also greatly promotes the rapid development of general artificial intelligence, making it more close to the all-round simulation of human intelligence.

[0003] However, in the key field of computer vision, although there are rich application scenarios and high research value, due to the uniqueness of task form and expression form, so far there is no unified model that can comprehensively cover target detection, image question answering, semantic segmentation, instance segmentation, target recognition and other basic and crucial fields. This scattered research status not only limits the further development of computer vision technology, but also hinders the perfection of the overall framework of general artificial intelligence.

[0004] The implementation scheme of the existing model is: widely collect data, build a training database, and accurately label the data to complete the task requirement; usually consider target detection, semantic segmentation, instance segmentation, target tracking and other tasks as isolated subfields, each field uses an independent technical solution to build a model, and there is no direct correlation between tasks; only image data is used as the only input, and the model output is given according to the task and the training data labeling result. The model output result is uncontrollable.

[0005] The existing model is often limited to limited data sources and cannot widely collect diversified data to build a comprehensive and rich training database. The generalization ability and adaptability of the model in complex and variable environments are limited; the separation strategy between tasks ignores the internal relationship and mutual assistance potential between different tasks, limiting the overall performance of the model; mainly rely on image data as input, lack of effective interaction with users. Users cannot directly tell the model the real intention, which may not meet the actual needs of the user. Such a model lacking understanding of user intent is inadequate when dealing with complex or personalized tasks.

[0006] Therefore, a general visual model is needed. SUMMARY

[0007] The application is to solve the problem of single task processing of the existing model, and provides a general visual recognition model based on prompt word interaction technology, which introduces key technologies such as a bidirectional multi-modal interaction module, a detection prompt word mechanism and an interesting region interaction mechanism. These modules enable deep bidirectional interaction between image features and text prompts, providing robust, stable and fine-grained feature support for downstream tasks, ensuring higher accuracy and generalization performance of the model; the application uses the detection prompt word mechanism to locate the target position, and uses text-image modal matching to identify the target semantics, which plays a key role in multiple visual understanding tasks. When performing text-image modal matching, the application uses a bidirectional contrast loss function and a super large negative sample library to further improve the recognition ability of the model; the application adopts a new technical route and constructs a comprehensive multi-modal large model semantic recognition architecture, solves the limitations of traditional methods, provides a wider space for future intelligent applications, and promotes the further development and application of general artificial intelligence.

[0008] The application provides a general visual recognition model based on prompt word interaction technology, which includes an image encoder, a text prompt word encoder and a visual prompt word encoder arranged side by side, a multi-modal interaction module connected to the output ends of the image encoder, the text prompt word encoder and the visual prompt word encoder, and a target decoding module connected to the output end of the multi-modal interaction module.

[0009] The multi-modal interaction module is a bidirectional interaction module, which includes a text-image interaction module, an image-text interaction module and an image-visual prompt word interaction module.

[0010] When using text prompt words for target detection, semantic positioning and target tracking, the image encoder encodes the image to be detected to obtain encoded image features, the text prompt word encoder encodes the user's input semantic recognition requirements to obtain encoded text prompt word features, the multi-modal interaction module interacts and fuses the encoded image features and the encoded text prompt word features to obtain new image features fused with other modal features and new text prompt word features fused with other modal features, and the target decoding module interacts the position coordinate prompt word vector and the new image features fused with other modal features to obtain target position prediction features, and then decodes the target position prediction features to obtain a detection frame and a corresponding confidence. Through confidence and similarity calculation, the target position and category described by the text prompt word are obtained, and the target tracking can also be performed.

[0011] When the target detection, instance segmentation and target tracking are performed by using the visual prompt words, an image encoder encodes a to-be-detected image to obtain an encoded image feature, a visual prompt word encoder encodes an input manually labeled region of interest image to obtain an encoded visual prompt word feature, a multi-modal interaction module interacts and fuses the encoded image feature and the encoded visual prompt word feature to obtain a new image feature fused with other modal features and a new visual prompt word feature fused with other modal features, a target decoding module interacts the visual special position coordinate prompt word vector and the new image feature fused with other modal features to obtain a visual target position prediction feature, and then decodes the visual target position prediction feature to obtain the probability that each image feature point belongs to the visual prompt word target object, so that the pixel point target position and the category are obtained through screening, and the target tracking or the pixel segmentation can be performed to convert the target detection result.

[0012] As a preferred mode, the image encoder is a multi-level image encoder, and the image encoder obtains encoded image features EI i of different levels.

[0013] The text prompt word encoder is a multi-level text prompt word encoder, and the text prompt word encoder obtains encoded text prompt word features EL j of different levels.

[0014] The visual prompt word encoder is a multi-level visual prompt word encoder, and the visual prompt word encoder obtains encoded visual prompt word features EP k of different levels.

[0015] The fusion method of the multi-modal interaction module is as follows:

[0016] EI in ,EL jn =B-Attention(EI i ,EL j );

[0017] EI in ,EP kn =B-Attention(EI i ,EP k );

[0018] wherein, EI in , EL jn and EP kn are new features fused after mutual interaction of different modalities, EI in is a new image feature fused with other modal features, and ELjn For the new text prompt word feature fused with other modal features, EP kn For the new visual prompt word feature fused with other modal features.

[0019] The universal visual recognition model based on the prompt word interaction technology, as a preferred mode, the method of B-Attention is:

[0020] Q A = W QA *X A ;

[0021] K B = W KB *X B ;

[0022]

[0023] Q B = W QB *X B ;

[0024] K A = W KA *X A ;

[0025]

[0026] Wherein, Q A is the query vector of input X A , K B is the keyword vector of input X B , E A is the new A feature fused with X B feature, Q B is the query vector of input X B , K A is the keyword vector of input X A , E B is the new B feature fused with X A feature; W QA , W KB , W QB , W KA are mapping vectors respectively, C is a normalization constant, X A , X B are EI i , EL j or EL j , EP k .

[0027] In a preferred embodiment of the general visual recognition model based on prompt word interaction technology described in this invention, the location coordinate prompt word vector is a learnable vector that includes the target location region that may exist in the image.

[0028] The method for interacting location coordinate cue word vectors with new image features fused with other modalities is as follows:

[0029] Q loc =W loc *X loc

[0030] K pic =W pic *X pic

[0031]

[0032] X loc (new) = X loc +X loc (old)

[0033] Among them, W loc The position coordinates provide hints for the word vector mapping parameters, X loc W provides word vector features to indicate location coordinates. pic X is the image feature mapping parameter. pic The final layer of features for the new image features, which integrates text cue word features, is C1, which is the normalization constant for the position coordinate cue word vector, and X... loc (old) represents the location coordinates indicating the features before the word vector update, X loc (new) indicates the updated features of the word vectors, suggesting the location coordinates.

[0034] The location coordinate cue word vectors are used to predict the coordinate position of the corresponding foreground target coordinate box and confidence level;

[0035] Filter out bounding boxes with confidence scores below a threshold, and then match the feature regions corresponding to the remaining bounding boxes with the new text prompt word features fused with other modal features. The matching method is as follows:

[0036]

[0037] Where, p t Features of the t-th coordinate frame Summary of new text cue word features that integrate other modal features X tp Given the probability of the same object, select the coordinate box corresponding to the highest probability to obtain the target location and category described by the text prompt.

[0038] The universal visual recognition model based on the prompt word interaction technology, as a preferred mode, the visual special position coordinate prompt word vector is a learnable vector, the visual special position coordinate prompt word vector and the new image feature fused with other modal features interact to extract the specific features of the target visual prompt word:

[0039] Q loc0 = W loc0 *X loc0

[0040] K crop = W crop *X crop

[0041]

[0042] X loc0 (new) = X loc0 + X loc0 (old)

[0043] Wherein, W loc0 is a visual special position coordinate prompt word vector mapping parameter, X loc0 is a visual special position coordinate prompt word vector feature, W crop is a visual special image feature mapping parameter, X crop is the last layer image feature of the new image feature fused with the visual prompt word feature, C0 is a visual special normalization constant, X loc0 (old) is the feature before updating the visual special position coordinate prompt word vector, X loc0 (new) is the updated feature.

[0044] The visual special position coordinate prompt word vector after interaction includes visual prompt word key features, and the visual special position coordinate prompt word vector is cross-interacted with the visual prompt word feature to obtain the probability that each image feature point belongs to the visual prompt word target object:

[0045] p u = sigmoid (X loc0 T *X crop )

[0046] Wherein, sigmoid is an activation function, and p u is the probability that the u-th pixel point belongs to the visual prompt word target.

[0047] The universal visual recognition model based on the prompt word interaction technology, as a preferred mode, when p u ≥0.5, the u-th pixel point belongs to the target category, otherwise it is regarded as a background category.

[0048] The general visual recognition model based on the prompt word interaction technology, as a preferred mode, when a tracking task is completed using a visual prompt word, first, an image feature of an encoded first frame image of a to-be-tracked sequence is obtained by inputting the first frame image into an image encoder; a tracking target of the first frame image is marked by using an interactive tool, and then an image content surrounded is cropped and input into a visual prompt word encoder to obtain an encoded visual prompt word feature; a multimodal interaction module interacts the encoded image feature of each frame of the video sequence with the encoded visual prompt word to obtain a new image feature fused with other modal features and a new visual prompt word feature fused with other modalities; when a target decoding module decodes, each frame of image interacts a visual special position coordinate prompt word vector with the visual prompt word feature to obtain a target position prediction feature, and the target decoding module predicts a plurality of prediction results by using the target position prediction feature, and a final detection result after threshold screening is obtained, and a video sequence tracking task is completed.

[0049] The general visual recognition model based on the prompt word interaction technology, as a preferred mode, the construction and training method of the general visual recognition model comprises the following steps:

[0050] S1, an original database containing a target detection field, a semantic segmentation field, an instance segmentation field, a target tracking field and a multimodal interaction field is constructed; the original database includes medical data and natural data, the medical data is class-labeled by doctors using a labeling tool, and for the natural data, the data is labeled by a related field expert or an outsourcing service team using LabelMe;

[0051] S2, the data in the original database is labeled, cleaned and audited, and then is included in a training database; the labeled data is divided into a training set, a validation set and a test set according to a proportion;

[0052] S3, the training set and the test set are used to simultaneously optimize a plurality of tasks, the validation set is used to evaluate the general visual recognition model, and the model with the best performance is saved;

[0053] In the training stage, the target categories of each task are encoded into text descriptions, and a single category is a text sentence; the visual prompt word and the text prompt word are input at the same time;

[0054] The multimodal interaction module interacts and fuses the encoded image feature with the encoded text prompt word feature, or interacts and fuses the encoded image feature with the encoded visual prompt word feature and then outputs to the target decoding module;

[0055] The target decoding module aligns the new text prompt word feature fused with other modal features and the new image feature fused with other modal features, so that each text prompt word and the described image part are one-to-one corresponding:

[0056]

[0057] Wherein, OI s+ is the s-th target object in the image, represents the text prompt word matched with the s-th target object, is obtained by averaging all characters of the text prompt word, is obtained by corresponding to the target frame labeled in the training set in the image;

[0058] S4, using the model with the best performance for inference application.

[0059] The universal visual recognition model based on the prompt word interaction technology, as a preferred mode, the annotation of medical data in step S1 includes: type, quantity, shape, state and color.

[0060] The universal visual recognition model based on the prompt word interaction technology, as a preferred mode, in step S3, natural language is used as an interactive means, and multiple semantic recognition tasks can be cooperated; during training, the image data corresponding to each task is input into the image encoder to obtain encoded image features EI of different levels i , the text form expression corresponding to the image is input into the text prompt word encoder to obtain encoded text prompt word features EL of different levels j , and the visual prompt word corresponding to the image is input into the visual prompt word encoder to obtain encoded visual prompt word features EP of different levels k The visual prompt word is the image in the labeled frame.

[0061] The application provides a universal visual recognition model based on a prompt word interactive technology. The model introduces the prompt word as a bridge for user interaction with the model, realizes unified processing of multiple visual fields, and greatly improves the performance and efficiency of visual processing through unified tasks and interactive operations. The model provides the possibility for further unification of other visual tasks, and injects new vitality into the comprehensive development of general artificial intelligence.

[0062] Specifically, the model uses "prompt words" as input to guide the model to process specific visual tasks. Through interaction between the user and the model, the model can more accurately understand the user's intention and more accurately complete various visual tasks. At the same time, due to the universality of the model, it can easily adapt to different visual application scenarios and provide efficient and accurate solutions for various visual tasks.

[0063] The universal visual recognition model based on the interactive technology of "prompt words" provided by the present application not only provides a new idea and method for the development of the field of computer vision, but also lays a solid foundation for the further development of general artificial intelligence.

[0064] The present application has the following advantages:

[0065] (1) The present application realizes unified processing of multiple tasks through "text guidance" and "visual guidance", that is, using natural language as the main interaction means while supporting the collaborative work of multiple semantic recognition tasks. The "bidirectional multi-modal interaction module" is introduced, which enables image features and text prompts to have deep bidirectional interaction, providing robust, stable and fine-grained feature support for downstream tasks, ensuring higher accuracy and generalization performance of the model.

[0066] (2) The present application constructs a unified multi-modal large model semantic recognition architecture, aiming to realize multiple complex semantic recognition tasks through a single model, including but not limited to object detection, semantic segmentation, instance segmentation and object tracking. The present application is not limited to classification problems, but covers a wider range of visual understanding tasks. Users can describe their requirements in natural language or manually specify the region of interest (ROI) to accurately locate, recognize, segment and track image targets.

[0067] (3) The present application uses the "detection prompt word mechanism" to locate the target position, and uses text-image modal matching to recognize the target semantics, playing a key role in multiple visual understanding tasks. The present application uses a bidirectional contrast loss function and a super large negative sample library when performing text-image modal matching, further improving the recognition ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 It is a general visual recognition model architecture based on prompt word interaction technology;

[0069] Figure 2 It is a flowchart of a general visual recognition model based on prompt word interaction technology;

[0070] Figure 3 It is a data labeling diagram of a general visual recognition model based on prompt word interaction technology Figure 1 ;

[0071] Figure 4 This is a data annotation diagram of a general visual recognition model based on prompt word interaction technology. Figure 2 ;

[0072] Figure 5 A schematic diagram of the original database for a general visual recognition model based on prompt word interaction technology. Figure 1 ;

[0073] Figure 6 A schematic diagram of the original database for a general visual recognition model based on prompt word interaction technology. Figure 2 ;

[0074] Figure 7 This is a schematic diagram of the training database for a general visual recognition model based on prompt word interaction technology. Detailed Implementation

[0075] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0076] Example 1

[0077] like Figure 1 , 2 As shown, a general visual recognition model based on prompt word interaction technology mainly consists of four parts.

[0078] 1. Build the initial database

[0079] Different types of data require different acquisition methods. In the acquisition of medical image data, this invention utilizes specialized equipment such as electron microscopes, phase-contrast microscopes, and optical microscopes to obtain images and employs a microscopic camera for data acquisition. For the acquisition of natural image data, this invention can use ordinary cameras for on-site photography or utilize web crawling technology to retrieve relevant image data from the internet. After data acquisition, this invention cleans and processes the images to ensure their quality and accuracy, and ultimately stores them in a database for subsequent use. Simultaneously, utilizing large-scale public databases that have been labeled, reviewed, cleaned, and made publicly available can effectively supplement the training database.

[0080] 2. Data labeling

[0081] Based on the "initial database," the data already stored is labeled. This task involves areas such as object detection, semantic segmentation, instance segmentation, object tracking, and multimodal interaction. For medical data, professional doctors can use specific labeling tools to label information such as categories. For natural data, experts in relevant fields or outsourced service teams can use labeling tools such as LabelMe to label the data. After the labeled data reaches a certain scale, it will be divided into training, validation, and test sets according to a certain ratio to prepare data for model building.

[0082] When labeling databases, such as Figure 3 As shown, the cell diagram is labeled as follows: This is a neutrophil. This cell has three nuclei. The nuclei are roughly round. The cytoplasm of this cell is relatively fine. The granules of this cell are relatively plump and are red.

[0083] This means labeling the cell type, number of nuclei, shape of the nuclei, state of the cytoplasm, state of the granules, and color of the granules.

[0084] like Figure 4 As shown, for natural images, regions of interest are manually labeled, such as heads, backpacks, arms, letters, etc.

[0085] 3. Model Building and Optimization

[0086] Based on the requirements of tasks such as object detection, semantic segmentation, instance segmentation, object tracking, and multimodal interaction, this invention proposes a model architecture with an image encoder, a text cue word encoder, a visual cue word encoder, and an object decoder. Based on this single model architecture, the model can simultaneously, quickly, and accurately complete all the above tasks and output corresponding results. Using the labeled training database described in section 2, multiple tasks can be optimized simultaneously, and the model can be evaluated using a validation set, with the best-performing model being saved.

[0087] 4. Reasoning Application

[0088] Based on the optimal model obtained in step 3, accurate reasoning can be performed for each requirement. Inputting textual prompts (such as "the fourth boy from left to right in the image"), the model can provide corresponding coordinates and bounding boxes based on the user's specific needs. Inputting visual prompts (such as "mark the fourth boy in the image with a bounding box" or "mark a point"), the model can provide a mask for the corresponding target based on the user's specific needs, segmenting the target. These tasks can be performed simultaneously, outputting various targets of interest to the user in different formats.

[0089] To train the general visual recognition model based on the prompt word technology, first, the data needs to be preliminarily collected to build an "original database". Then, based on the "original database", the data is labeled to build a "training database". The model with "text prompt word encoder", "image encoder", "visual prompt word encoder", "target decoder" is trained and optimized, and the optimal model parameters are saved. The optimal model can be used to complete various task requirements and output target results with a single model, and the specific process is as follows.

[0090] (1) Building an original database

[0091] Firstly, the present application is committed to the detailed collection of target data, striving to include all available data resources and tasks, and ensuring that all types of samples that may occur in actual application scenarios are covered. On this basis, an "original database" is built, which covers five fields such as target detection, semantic segmentation, instance segmentation, target tracking, and multi-model identification, as shown in Figure 5 , 6 .

[0092] (2) Data labeling

[0093] As shown in Figure 7 , based on the built "original database", the data in the database is labeled. This task involves target detection, semantic segmentation, instance segmentation, target tracking, and multi-modal interaction. For medical data, professional doctors can use professional labeling tools to label class, position coordinates, and other information. For natural data, relevant field experts or outsourcing service teams can use LabelMe and other labeling tools to label data. At the same time, public data sets that have been cleaned and audited are included in the training database. After the labeling amount reaches a certain scale, the labeled data is divided into training set, validation set and test set according to a certain proportion to prepare data for building the model.

[0094] (3) Model building and optimization

[0095] The present application proposes a general visual recognition model based on image encoder, text prompt word encoder, visual prompt word encoder, text-image interaction module, image-text interaction module, and image and prompt word interaction module (bidirectional interaction module). This model can simultaneously solve the task requirements of target detection, semantic positioning (fine detection task), semantic segmentation, instance segmentation, target tracking, and multi-modal interaction. The present application abstracts the above tasks as semantic positioning tasks. In the training stage, in addition to the semantic positioning task and the multi-modal interaction task, the target classes of each task (such as cat, dog, duck) are encoded as text descriptions, and a single class is a text sentence (such as cat. It is a sentence). For the semantic positioning task and the multi-modal interaction task, the text description is its text label.

[0096] After determining the standard representation form of the text, the model can be trained. First, the image data corresponding to each task is input into the multi-level image encoder to obtain image features E1 i , where i = 1, 2, 3, 4, 5, 6. represents the image features obtained by different levels of encoding. Then, the text form of the expression corresponding to the image is input into the different level text prompt word encoder to obtain the different level encoded text prompt word features EL j , where j = 1, 2, 3, 4, 5, 6. At the same time, the visual prompt word corresponding to the image (i.e. the image in the annotation box) is input into the different level visual prompt word encoder to obtain the different level encoded visual prompt word features EP k , where k = 1, 2, 3, 4, 5, 6. Then, the image features, text prompt word features, and image prompt word features of different levels are respectively sent into the bidirectional interaction module to fuse the features of different modalities and prompt words, and increase the feature expression ability of a single modality. The specific fusion rule is as follows:

[0097] EI in , EL jn = B-Attention(EI i , EL j );

[0098] EI in , EP kn = B-Attention(EI i , EP k );

[0099] wherein, EI in , EL jn is the new feature fused after the interaction of the text prompt word features and the image features. EI in , EP kn is the new feature fused after the interaction of the visual prompt word features and the image features. The features of different levels are all subjected to the above interaction to obtain features rich in semantic information fused with other modalities.

[0100] wherein, the specific operation of B-Attention is:

[0101] Q A = W QA *X A ;

[0102] K B = W KB *X B ;

[0103]

[0104] QB = W QB * X B ;

[0105] K A = W KA * X A ;

[0106]

[0107] wherein Q A represents a query vector of input X A , K B represents a keyword vector of input X B , E A represents a new A feature fused with the characteristics of X B , Q B represents a query vector of input X B , K A represents a keyword vector of input X A , E B represents a new B feature fused with the characteristics of X A . W QA , W KB , W QB , W KA are mapping vectors respectively, and C is a normalization constant.

[0108] After the above operations, the new features fused with other modalities are obtained respectively, and then the above new features can be used to complete downstream tasks such as target detection, semantic positioning, semantic segmentation, instance segmentation, target tracking, multi-modal interaction. The above features are all sent into a “target decoder” for decoding. First, the text prompt feature and the image feature are aligned, so that each text prompt and the image part it describes are one-to-one corresponding. The specific process is as follows:

[0109]

[0110] wherein OI m+ represents the mth target object (image) in the image, represents the text prompt matched with the mth target object, wherein is obtained by averaging all characters described by the text, is obtained by corresponding to the target box labeled in the training set in the image. The present application hopes that the distance between the target object and the text of the same matching pair is smaller, and the distance between different matching pairs is larger, and the model can find the corresponding target object in the image according to the text description by optimizing this goal.

[0111] In the target decoder, a preset learnable position coordinate prompt vector is interacted with an image feature to output a coordinate position of a target object in the image, and meanwhile, a feature region corresponding to the coordinate box is matched with a text prompt feature to anchor a text description corresponding to the target object. That is, the task of positioning the corresponding target object in the image according to the text prompt can be completed. The position coordinate prompt vector is interacted with a visual prompt feature to perform pixel-level segmentation on a target image represented by the image prompt, and a segmentation result is output. The segmentation result and the positioning result are continuously optimized to the target, and correct detection coordinates and segmented pixels are obtained.

[0112] In the target decoder, a preset learnable position coordinate prompt vector is interacted with an image feature to output a coordinate position of a target object in the image, and meanwhile, a feature region corresponding to the coordinate box is matched with a text prompt feature to anchor a text description corresponding to the target object. That is, the task of positioning the corresponding target object in the image according to the text prompt can be completed. The position coordinate prompt vector is interacted with a visual prompt feature to perform pixel-level segmentation on a target image represented by the image prompt, and a segmentation result is output. The segmentation result and the positioning result are continuously optimized to the target, and correct detection coordinates and segmented pixels are obtained.

[0113] The position coordinate prompt vector is cross-interacted with the image feature, and the specific form is as follows:

[0114] Q loc =W loc *X loc

[0115] K pic =W pic *X pic

[0116]

[0117] X loc (new)=X loc +X loc (old)

[0118] Wherein, W loc is a mapping parameter of the position coordinate prompt vector, X loc is a feature of the position coordinate prompt vector, W pic is an image feature mapping parameter, X pic is an image feature of the last layer of a new image feature fused with a text prompt feature, C1 is a normalization constant of the position coordinate prompt vector, X loc (old) is a feature of the position coordinate prompt vector before update, and X loc(new) is the updated feature. The updated "position coordinate prompt word vector" contains the features of the corresponding target object, so that the new "position coordinate prompt word vector" can be used for position coordinate decoding and category prediction. The "position coordinate prompt word vector" is input into a 2-layer nonlinear mapping layer for coordinate position prediction, and the foreground target coordinate frame and confidence size corresponding to the "position coordinate prompt word vector" are obtained. All "position coordinate prompt word vectors" can output the coordinate frame and confidence of the corresponding foreground. The coordinate frame with a confidence less than a threshold is filtered, and the feature region corresponding to the remaining coordinate frame is matched with the "text prompt word" feature after interaction. The specific matching strategy is as follows:

[0119]

[0120] wherein p t represents the feature of the tth coordinate frame is the same as the summary X of the new text prompt word feature fused with other modal features tp is the probability of the same object. The coordinate frame corresponding to the maximum probability is selected, and the specific target corresponding to the "text prompt word" and the category of the target are determined. The task of positioning the corresponding target object in the image according to the text prompt word is completed.

[0121] For the image segmentation task, the visual special "position coordinate prompt word vector" is cross-interacted with the "image prompt word" feature fused with other modal features to extract the specific key features of the target "image prompt word". The specific interaction mode is as follows:

[0122] Q loc0 = W loc0 *X loc0

[0123] K crop = W crop *X crop

[0124]

[0125] X loc0 (new) = X loc0 + X loc0 (old)

[0126] wherein W loc0 is the mapping parameter of the visual special position coordinate prompt word vector (1), X loc0 is the visual special position coordinate prompt word vector feature, W crop is the image feature mapping parameter, X crop is the visual prompt word feature fused with the visual prompt word labeled target, and is EI in , EP knthe last layer of image features, C0 is a visual-specific normalization constant, X loc0 (old) is the feature before the update of the visual-specific location coordinate cue vector, X loc0 (new) is the updated feature. At this time, the visual-specific location coordinate cue vector contains the key features of the "visual cue", and the probability that each image feature point belongs to the "visual cue" target object is obtained by cross interaction between the visual-specific location coordinate cue vector and the "image feature". The specific interaction mode is as follows:

[0127] p u = sigmoid(X loc0 T *X crop )

[0128] where sigmoid represents the activation function, p u represents the probability that the u-th pixel belongs to the "visual cue" target, when p u ≥ 0.5, it can be considered that the pixel belongs to the target category, otherwise it is considered as the background category. The target image represented by the image cue can be segmented at the pixel level, and the segmentation result can be output. The segmentation result and the positioning result are continuously optimized to the target, and the correct detection coordinates and segmented pixels can be obtained.

[0129] (4) Model application

[0130] When the model is trained according to the steps described in (3), the model can be applied at any time. The inference process is as follows:

[0131] For detection and tracking tasks, first, the target object to be detected is expressed in the form of text, such as "a male eagle soaring", and then the text is input into the text cue encoder to obtain the encoded text cue feature. At the same time, the image to be detected is input into the image encoder to obtain the encoded image feature. The image feature and the text cue feature interact with each other to obtain features that enhance and fuse each other's modal information. Then, the position detection cue vector (100) is interacted with the image feature to obtain the target position prediction feature (100). The 100 target position prediction features are decoded to obtain 100 detection boxes and corresponding confidence, and all detection boxes with confidence lower than the predetermined threshold are filtered out. The image feature corresponding to the remaining detection boxes is calculated with the text cue feature to find the detection box with the smallest distance (maximum probability), and the target position and category described by the text cue are obtained. In the tracking task, since the "text cue" is limited, the target found in each frame of image is the same object, and the tracking task can be completed.

[0132] For the tracking task using the "visual cue word", first input the first frame image of the to-be-tracked sequence into the image encoder to obtain the encoded image features. At the same time, mark the tracking target of the first frame image using an interactive tool, such as wanting to track the small boy wearing a blue sweater and wearing a black hat in the video sequence, the small boy can be circled with a frame using the interactive tool, and then the model cuts down the circled image content and inputs it into the visual cue word encoder for encoding to obtain the encoded visual cue word features. Cross-interact the visual special position coordinate cue word vector with the "image cue word" features fused with other modal features, extract the specific key features of the target "image cue word", and cross-interact the visual special position coordinate cue word vector with the image features of each frame of the video sequence to obtain the probability that each image feature point belongs to the "visual cue word" target object. When the probability is greater than or equal to 0.5, it is considered that the pixel point belongs to the target category, otherwise it is considered as a background category. The target image represented by the image cue word can be segmented at the pixel level, and the segmentation result is output. Since the visual cue word is unique in the video sequence, only one small boy to be tracked will be detected in each frame, and the detection results of all frames can constitute a complete video "target tracking sequence". Through the above operation, the video sequence tracking task can be completed.

[0133] For the segmentation task, first input the to-be-segmented image into the image encoder to obtain the encoded image features. At the same time, mark the to-be-segmented target using an interactive tool, such as wanting to segment a cat in the image, the cat can be circled with a frame using the interactive tool, and then the model cuts down the circled image content and inputs it into the visual cue word encoder for encoding to obtain the encoded visual cue word features. Cross-interact the image features and the visual cue word features to obtain features that enhance and fuse each other's modal information. Then, cross-interact the visual special position coordinate cue word vector (1) with the visual cue word features to obtain a new visual special position coordinate cue word vector (1). Cross-interact the visual special position coordinate cue word vector (1) with the image feature points to obtain the probability that each feature point belongs to the "visual cue word". Points with a probability value lower than 0.5 are considered as background points, and other points are considered as target points, so that the segmentation result based on the "visual cue word" is obtained. Through the above operation, a single model can complete the tasks of detection, positioning, segmentation, tracking, and multi-modal interaction.

[0134] In the self-attention network architecture, the position detection prompt word vector mainly undertakes the key task of detecting potential foreground objects in the image. It is a learnable high-dimensional vector, and each position detection prompt word vector is closely related to whether there is a foreground of interest at each position of the image. If there is a foreground at the position corresponding to a certain position detection prompt word vector, the feature generated by interacting with the image through it can decode the object coordinates that may belong to the target of interest with high confidence. The position detection prompt word vector will carefully search for the foreground target in the area it is responsible for, striving to cover all interested objects comprehensively to ensure that no interested object is missed, thereby effectively improving the detection effect of the foreground objects in the image.

[0135] The technical innovation of the present application starts from building a unified multi-modal large model semantic recognition architecture, aiming to realize multiple semantic recognition tasks through a single model, including but not limited to object detection, semantic segmentation, instance segmentation, and target tracking. The design concept of this architecture conforms to the development direction of general artificial intelligence, significantly reducing the user's interaction cost. Users only need to input specific semantic recognition requirements through natural language, such as "please help me find the little girl wearing a red hat and a black sweater", and the model can accurately recognize and segment / detect / tracking the target object in the image.

[0136] In terms of technical implementation, the present application introduces a learnable "detection prompt word" mechanism and develops a "bidirectional multi-modal feature interaction module". This not only enables deep bidirectional interaction between image features and text prompt word features, but also ensures the robustness and stability of the features, thereby providing fine-grained and high-quality feature support for downstream tasks. Compared with the single-directional multi-modal interaction method, the present application has obvious advantages in feature learning effect, and can more effectively capture and understand complex visual scenes.

[0137] The true core competitiveness of the present application lies in the unified processing of multiple tasks through "text guidance" and "visual guidance", i.e. using natural language as the main interaction means while supporting the collaborative work of multiple semantic recognition tasks. This method not only improves user experience, but also promotes the pace towards general artificial intelligence. Through this comprehensive multi-modal processing capability, the present application provides a wider application space and development potential for future intelligent applications.

[0138] The technical innovation of the present application starts from building a unified multi-modal large model semantic recognition architecture, aiming to realize multiple complex semantic recognition tasks through a single model, including but not limited to target detection, semantic segmentation, instance segmentation and target tracking. The present application is not limited to classification problems, but covers a wider range of visual understanding tasks. Users can accurately locate, identify, segment and track image targets through natural language description requirements or manual specification of regions of interest (ROI).

[0139] In specific implementation, the visual prompt module of the present application is based on "hand-labeled regions of interest", while the text prompt module is guided by "textually described requirements". This design enables the model to flexibly cope with diversified application scenarios and provide more personalized and accurate recognition results.

[0140] The model of the present application can handle a wider range of image and text requirements, support multiple semantic recognition tasks, emphasize hand-labeled visual prompts and natural language text prompts, and provide more intuitive and flexible interaction methods.

[0141] In summary, the present application adopts a completely new technical route and builds a comprehensive multi-modal large model semantic recognition architecture, which not only improves user experience, but also provides broad development space for future intelligent applications. Through this highly integrated and multi-functional architecture, the application and further development of general artificial intelligence are promoted.

[0142] In specific implementation, the present application introduces "bidirectional multi-modal interaction module", "detection prompt word mechanism" and "region of interest interaction mechanism" and other key technologies. These modules enable deep bidirectional interaction between image features and text prompts, providing robust, stable and fine-grained feature support for downstream tasks, ensuring higher accuracy and generalization performance of the model.

[0143] The present application uses "detection prompt word mechanism" to locate the target position, and uses text-image modal matching to recognize the target semantic, playing a key role in multiple visual understanding tasks. When performing text-image modal matching, the present application adopts a bidirectional contrast loss function and a super large negative sample library, further improving the recognition ability of the model.

[0144] The present application adopts a completely new technical route and builds a comprehensive multi-modal large model semantic recognition architecture, solving the limitations of traditional methods and providing more broad space for future intelligent applications, promoting the further development and application of general artificial intelligence.

[0145] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A general visual recognition model based on prompt word interaction technology, characterized in that: The image encoder, the text prompt word encoder, and the visual prompt word encoder are arranged in parallel, a multi-modal interaction module is connected to the output ends of the image encoder, the text prompt word encoder, and the visual prompt word encoder, and a target decoding module is connected to the output end of the multi-modal interaction module; The multi-modal interaction module is a bidirectional interaction module, and includes a text-image interaction module, an image-text interaction module, and an image-visual prompt word interaction module; When target detection, semantic positioning, semantic segmentation, and target tracking are performed by using a text prompt word, the image encoder encodes a to-be-detected image to obtain an encoded image feature, the text prompt word encoder encodes a semantic recognition requirement input by a user to obtain an encoded text prompt word feature, the multi-modal interaction module interactively fuses the encoded image feature and the encoded text prompt word feature to obtain a new image feature fused with other modal features and a new text prompt word feature fused with other modal features, and the target decoding module interacts a position coordinate prompt word vector with the new image feature fused with other modal features to obtain a target position prediction feature, and then decodes the target position prediction feature to obtain a detection frame and a corresponding confidence, and the target position and the category described by the text prompt word are obtained by screening through the confidence and similarity calculation, and the target tracking can also be performed; When target detection, instance segmentation, and target tracking are performed by using a visual prompt word, the image encoder encodes a to-be-detected image to obtain an encoded image feature, the visual prompt word encoder encodes an input hand-labeled region of interest image to obtain an encoded visual prompt word feature, the multi-modal interaction module interactively fuses the encoded image feature and the encoded visual prompt word feature to obtain a new image feature fused with other modal features and a new visual prompt word feature fused with other modal features, and the target decoding module interacts a visual-specific position coordinate prompt word vector with the new image feature fused with other modal features to obtain a visual target position prediction feature, and then decodes the visual target position prediction feature to obtain a probability that each image feature point belongs to a visual prompt word target object, and the pixel point target position and the category are obtained by screening, and the target tracking can also be performed, or the target detection result is obtained after pixel segmentation.

2. The general visual recognition model based on prompt word interaction technology according to claim 1, characterized in that: The image encoder is a multi-stage image encoder, and the image encoder obtains encoded image features E1 i of different levels, i is the number of levels of the image encoder. The text prompt encoder is a multi-level text prompt encoder, and the text prompt encoder obtains encoded text prompt features EL of different levels j , and j is the number of levels of the text prompt encoder. The visual prompt word encoder is a multi-level visual prompt word encoder, and the visual prompt word encoder obtains encoded visual prompt word features EP of different levels k , k is the number of levels of the visual prompt word encoder; The fusion method of the multi-modal interaction module is: EI in ,EL jn = B-Attention(EI i ,EL j ); EI in ,EP kn = B-Attention(EI i ,EP k ); wherein, EI in , EL jn , EP kn are new features after different modalities interact with each other, EI in is a new image feature after fusing other modalities, EL jn is a new text prompt word feature after fusing other modalities, EP kn is a new visual prompt word feature after fusing other modalities.

3. The general visual recognition model based on prompt word interaction technology according to claim 2, characterized in that: The method of B-Attention is: Q A = W QA X A ; K B = W KB * X B ; Q B = W QB X B ; K A = W KA * X A ; wherein Q A is a query vector of input X A , K B is a keyword vector of input X B , E A is a new A feature fused with the feature of X B , Q B is a query vector of input X B , K A is a keyword vector of input X A , E B is a new B feature fused with the feature of X A ; W QA , W KB , W QB , W KA are mapping vectors respectively, C is a normalization constant, X A , X B are EI i , EI j or EL j , EP k .

4. The general visual recognition model based on prompt word interaction technology according to claim 2, characterized in that: The position coordinate prompt word vector is a learnable vector including a target position region possibly existing in an image; The method of interacting the position coordinate prompt word vector with the new image feature fused with other modal features is: Q loc = W loc X loc K pic = W pic * X pic X loc (new) = X loc + X loc (old) wherein, W loc is a position coordinate prompt word vector mapping parameter, X loc is a position coordinate prompt word vector feature, W pic is an image feature mapping parameter, X pic is the last layer feature of the new image feature fused with the text prompt word feature, C1 is a normalization constant of the position coordinate prompt word vector, X loc (old) is the feature before updating of the position coordinate prompt word vector, X loc (new) is the feature after updating of the position coordinate prompt word vector; The position coordinate prompt word vector is used to predict a corresponding foreground target coordinate frame and a confidence; Coordinate frames with a confidence less than a threshold value are filtered, and the feature regions corresponding to the remaining coordinate frames are matched with the new text prompt word feature fused with other modal features, and the matching method is: where p t is the feature of the t-th coordinate frame X is the summary of the new text cue features fused with other modality features tp is the probability of the same object, and the coordinate frame corresponding to the maximum probability is selected to obtain the target position and category described by the text cue.

5. The general visual recognition model based on prompt word interaction technology according to claim 2, characterized in that: The visual special position coordinate prompt word vector is a learnable vector, and the visual special position coordinate prompt word vector interacts with the new image feature fused with other modal features to extract specific features of the target visual prompt word: Q loc0 = W loc0 X loc0 K crop = W crop * X crop X loc0 (new) = X loc0 + X loc0 (old) wherein W loc0 is a visual-specific location coordinate cue vector mapping parameter, X loc0 is a visual-specific location coordinate cue vector feature, W crop is a visual-specific image feature mapping parameter, X crop is a last layer image feature of a new image feature fused with visual cue features, C0is a visual-specific normalization constant, X loc0 (old) is a feature before updating of the visual-specific location coordinate cue vector, X loc0 (new) is an updated feature; After the interaction, the visual special position coordinate prompt word vector includes visual prompt word key features, and the visual special position coordinate prompt word vector is compared with X crop Cross interaction is performed to obtain the probability that each image feature point belongs to the visual prompt word target object: p u = sigmoid(X loc0 T *X crop ) where sigmoid is an activation function, p u is the probability that the u-th pixel belongs to the visual cue target.

6. The general visual recognition model based on prompt word interaction technology according to claim 5, characterized in that: When p u u-th pixel belongs to the target category, otherwise it is considered as the background category.

7. The general visual recognition model based on prompt word interaction technology according to claim 1, characterized in that: When the tracking task is completed using the visual prompt word, first, the first frame image of the to-be-tracked sequence is input into the image encoder to obtain the encoded image feature; The tracking target of the first frame image is marked using an interactive tool, and then the image content surrounded is cropped and input into the visual prompt word encoder for encoding to obtain the encoded visual prompt word feature; the multi-modal interaction module interacts the encoded image feature and the encoded visual prompt word of each frame of the video sequence to obtain the new image feature fused with other modal features and the new visual prompt word feature fused with other modal features, and the target decoding module interacts the target position prediction feature obtained by the visual special position coordinate prompt word vector and the visual prompt word feature when decoding, and the target decoding module predicts a plurality of prediction results by using the target position prediction feature, and the final detection result after threshold screening is obtained to complete the video sequence tracking task.

8. The general visual recognition model based on prompt word interaction technology according to claim 2, characterized in that: The construction and training method of the general visual recognition model comprises the following steps: S1, constructing an original database comprising a target detection field, a semantic segmentation field, an instance segmentation field, a target tracking field and a multi-modal interaction field; the original database comprises medical data and natural data, the medical data is class-labeled by a doctor using a labeling tool, and the natural data is labeled by a related field expert or an outsourcing service team using LabelMe; S2, labeling, cleaning and auditing the data in the original database, and then incorporating the data into a training database; the labeled data is divided into a training set, a validation set and a test set according to a proportion; S3, simultaneously optimizing a plurality of tasks by using the training set and the test set, evaluating the general visual recognition model by using the validation set, and saving the model with the best performance; In the training stage, the target classes of each task are encoded as text descriptions, and a single class is a text sentence; the visual prompt word and the text prompt word are input simultaneously; The multi-modal interaction module interacts and fuses the encoded image feature and the encoded text prompt word feature, and outputs the fused encoded image feature and encoded visual prompt word feature to the target decoding module; The target decoding module performs alignment operation on the new text prompt word feature fused with other modal features and the new image feature fused with other modal features, so that each text prompt word and the described image part are one-to-one corresponding: wherein OI s+ is the s-th target object in the image, represents a text prompt word matched with the s-th target object, is obtained by averaging all characters of the text prompt word, is obtained by corresponding to the target frame labeled in the training set in the image; S4, using the model with the best performance for inference application.

9. The general visual recognition model based on prompt word interaction technology according to claim 8, characterized in that: In step S1, the medical data labeling comprises: category, quantity, shape, state and color.

10. The general visual recognition model based on prompt word interaction technology according to claim 8, characterized in that: In step S3, the natural language is used as the interactive means, and the multiple semantic recognition tasks can be cooperated; during training, the image data corresponding to each task is input into the image encoder to obtain the encoded image features EI of different levels i The text form of the expression corresponding to the image is input into the text prompt word encoder to obtain the encoded text prompt word features EL of different levels j Meanwhile, the visual prompt word corresponding to the image is input into the visual prompt word encoder to obtain the encoded visual prompt word features EP of different levels k The visual prompt word is the image in the annotation box.

Images