Model training method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a model training method and device, equipment, a storage medium and a program product. The method comprises the following steps: inputting a sample image into an image coding module to obtain image features; inputting the sample text into a text coding module to obtain text features; obtaining a fusion prompt vector corresponding to the sample text from a pre-constructed fusion prompt vector dictionary, and inputting the fusion prompt vector and the text feature into a vector replacement module to obtain a mixed text feature; and inputting the image features and the mixed text features into a detector module, and training a visual cue open set detection model. Through the above mode, the condition of overfitting or evaluation distortion of the model can be avoided, the generalization performance of the visual prompt open set detection model is improved, and the model training overhead is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, device, equipment, storage medium and program product. Background Art

[0002] As computing power and data size continue to grow, traditional small closed-set object detection models are gradually being replaced by large open-set object detection models. The large general open-set object detection model has a large number of parameters, which enables it to derive a certain degree of zero-shot or few-shot reasoning capabilities. Even without training for specific tasks, the large general open-set object detection model can perform well in many new downstream tasks. However, when faced with situations that are very different from the training dataset scenario, the performance of the large general open-set object detection model will also be affected. In addition, fine-tuning the large open-set object detection model requires a lot of computing resources and time.

[0003] Based on this, in some related technologies, it is chosen to introduce an additional cue encoder module into the open set object detection large model to form a visual cue open set detection model.

[0004] See also Figure 1 , Figure 1 It is a structural diagram of a visual cue open set detection model provided by the prior art.

[0005] like Figure 1 As shown in Figure 1, the visual cue open set detection model includes an open set detection module and a visual cue module. The open set detection module uses image-text alignment to detect the target category area in the image through text definition; the visual cue module uses images to better guide text information and fine-tune the model based on the open set detection module. Figure 1 The hint encoding module in processes the information in the image field, and the category hint vector it generates can carry more information related to the detection target compared to the plain text hint vector after plain text feature processing.

[0006] Specifically, Figure 1As shown in FIG, the training process of the existing visual cue open set detection model is as follows: input the sample image into the image encoding module to generate image features; input the sample text into the text encoding module to generate text features; in the training set or the validation set, extract an image from the images corresponding to each text category, and input the image and the image annotation information (denoted as BBox) into the cue encoding module, and after feature extraction, generate a visual cue vector; locally replace part of the original text features with the visual cue vector according to the position of the corresponding category to generate mixed text features, which carry a certain degree of image semantic information; input the image features and the mixed text features into the detector module to complete the training and reasoning process of the visual cue open set detection model.

[0007] Compared with the image-text alignment training model, the visual cue open-set detection model introduces some image semantic information into the text features for replacement processing, which can optimize the effect of model training.

[0008] However, in the above training method, the input data of the cue encoding module needs to be extracted from the training set or the validation set. If the images in the training set and the corresponding annotation information of the images are extracted, it may cause the model to overfit; if the images in the validation set and the corresponding annotation information of the images are extracted, it may cause the evaluation of the model to be distorted. There is a situation where the validation set index is high but the test set effect is not good. The above two situations will lead to poor generalization performance of the visual cue open set detection model.

[0009] In addition, when training the model, for samples in different batches, the model needs to extract image information from the training set or validation set corresponding to each text category, and then calculate and generate the visual cue vector. This process will increase the training cost of the model.

[0010] Therefore, existing model training methods easily lead to poor generalization performance of visual cue open-set detection models and high model training overhead. Summary of the invention

[0011] The embodiments of the present application provide a model training method, apparatus, device, storage medium and program product to solve the technical problems that the existing model training method easily leads to poor generalization performance of the visual cue open set detection model and high model training overhead.

[0012] In a first aspect, an embodiment of the present application provides a model training method, including: obtaining a sample image and a sample text; inputting the sample image into an image encoding module of a visual cue open set detection model to obtain image features output by the image encoding module; inputting the sample text into a text encoding module of the visual cue open set detection model to obtain text features output by the text encoding module; obtaining a fused cue vector corresponding to the sample text from a pre-built fused cue vector dictionary, and inputting the fused cue vector and the text features into a vector replacement module of the visual cue open set detection model to obtain mixed text features output by the vector replacement module; inputting the image features and the mixed text features into a detector module of the visual cue open set detection model to train the visual cue open set detection model.

[0013] In one embodiment, before obtaining the sample image and the sample text, the method further includes: obtaining the sample text and a plurality of sample images; inputting the plurality of sample images into the image encoding module of the visual cue open set detection model to obtain the image features of each sample image output by the image encoding module; inputting the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module; performing zero-sample inference on the text features and the image features of each sample image to generate a plurality of first images; performing image sampling fusion on the plurality of first images to generate a target image; generating a fused cue vector corresponding to the sample text based on the target image and the text features; and constructing a fused cue vector dictionary based on the fused cue vector.

[0014] In one embodiment, zero-sample inference is performed on text features and image features of each sample image to generate multiple first images, including: inputting the text features and the image features of each sample image into a contrast supervision layer of a visual cue open set detection model to obtain candidate text features and multiple candidate image features output by the contrast supervision layer; inputting the candidate text features and multiple candidate image features into a detector module of the visual cue open set detection model to obtain multiple first images output by the detector module; wherein the feature similarity between the candidate image features and the candidate text features is greater than a preset threshold; the contrast supervision layer is used to calculate the feature similarity of each image feature and text feature respectively, and based on the feature similarity between each image feature and text feature, determine and output the candidate image features and candidate text features.

[0015] In one embodiment, image sampling and fusion are performed on multiple first images to generate a target image, including: inputting the multiple first images into an image sampling module of a visual cue open set detection model, performing importance sampling, and obtaining multiple second images output by the image sampling module; inputting the multiple second images into an image fusion module of the visual cue open set detection model, performing image fusion processing, and obtaining a target image output by the image fusion module.

[0016] In one embodiment, a fused prompt vector corresponding to a sample text is generated based on a target image and text features, including: inputting the target image into a prompt encoding module of a visual prompt open set detection model to obtain a visual prompt vector output by the prompt encoding module; inputting the visual prompt vector and text features into a prompt fusion module of a visual prompt open set detection model to obtain a fused prompt vector corresponding to the sample text output by the prompt fusion module.

[0017] In one embodiment, a prompt fusion module includes a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer and a feedforward neural network layer; a visual prompt vector and a text feature are input into the prompt fusion module of a visual prompt open set detection model to obtain a fused prompt vector corresponding to the sample text output by the prompt fusion module, including: inputting the text feature into the first self-attention layer of the prompt fusion module to obtain the first text feature vector output by the first self-attention layer; inputting the visual prompt vector into the second self-attention layer of the prompt fusion module to obtain the first image feature vector output by the second self-attention layer; inputting the first text feature vector and the first image feature vector into the first cross-attention layer of the prompt fusion module to obtain the second image feature vector output by the first cross-attention layer; inputting the first text feature vector and the first image feature vector into the second cross-attention layer of the prompt fusion module to obtain the second text feature vector output by the second cross-attention layer; inputting the second image feature vector and the second text feature vector into the feedforward neural network layer of the prompt fusion module to obtain the fused prompt vector corresponding to the sample text output by the feedforward neural network layer.

[0018] In a second aspect, an embodiment of the present application provides a model training device, including: an acquisition module for acquiring sample images and sample texts; an image feature extraction module for inputting the sample images into the image encoding module of the visual cue open set detection model to obtain image features output by the image encoding module; a text feature extraction module for inputting the sample text into the text encoding module of the visual cue open set detection model to obtain text features output by the text encoding module; a dictionary extraction module for obtaining a fused cue vector corresponding to the sample text from a pre-built fused cue vector dictionary, and inputting the fused cue vector and the text features into the vector replacement module of the visual cue open set detection model to obtain mixed text features output by the vector replacement module; and a training module for inputting the image features and the mixed text features into the detector module of the visual cue open set detection model to train the visual cue open set detection model.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned model training methods is implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned model training methods.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned model training methods when executed by a processor.

[0022] The model training method, apparatus, equipment, storage medium and program product provided in the embodiments of the present application no longer extract images and corresponding annotation information from the training set or the validation set during the model training process, and there is no need to calculate and generate visual cue vectors. Instead, the fused cue vectors corresponding to the sample texts are directly obtained from a pre-built fused cue vector dictionary, and mixed text features are generated based on the fused cue vectors and text features. Then, a visual cue open set detection model is trained based on the image features and the mixed text features, thereby avoiding overfitting of the model or evaluation distortion, improving the generalization performance of the visual cue open set detection model, and reducing the model training overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a structural diagram of a visual cue open set detection model provided by the prior art.

[0025] Figure 2 This is one of the flow charts of the model training method provided in the embodiment of the present application.

[0026] Figure 3 This is the second flow chart of the model training method provided in the embodiment of the present application.

[0027] Figure 4 It is a comparative schematic diagram of the model training method before and after improvement provided in the embodiment of the present application.

[0028] Figure 5 It is a schematic diagram of the construction process of the fusion hint vector dictionary provided in the embodiment of the present application.

[0029] Figure 6 It is a schematic diagram of the module self-reuse mechanism provided in the embodiment of the present application.

[0030] Figure 7It is a flowchart of image sampling and fusion provided in an embodiment of the present application.

[0031] Figure 8 It is a schematic diagram of the generation process of the fusion prompt vector provided in the embodiment of the present application.

[0032] Fig. 9 It is a structural diagram of the model training device provided in an embodiment of the present application.

[0033] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] See also Figure 2 and Figure 3 , Figure 2 This is one of the flow charts of the model training method provided in the embodiment of the present application. Figure 3 This is the second flow chart of the model training method provided in the embodiment of the present application. In the embodiment of the present application, the model training method includes steps S210 to S250, and each step is as follows: S210: Acquire sample images and sample texts.

[0036] S220: Input the sample image into the image encoding module of the visual cue open set detection model to obtain image features output by the image encoding module.

[0037] S230: Input the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module.

[0038] S240: Obtain a fused prompt vector corresponding to the sample text from a pre-built fused prompt vector dictionary, and input the fused prompt vector and text features into a vector replacement module of a visual prompt open set detection model to obtain a mixed text feature output by the vector replacement module.

[0039] like Figure 3 As shown, the core idea of ​​this application is: without changing the existing visual cue open set detection model backbone, a fused cue vector dictionary pre-built before model training is introduced, and the fused cue vector dictionary contains fused cue vectors corresponding to sample texts.

[0040] Specifically, during the model training process, images and corresponding annotation information are no longer extracted from the training set or validation set, and there is no need to calculate and generate visual cue vectors. Instead, the fused cue vectors corresponding to the sample texts are obtained from a pre-built fused cue vector dictionary, and the fused cue vectors and text features are input into the vector replacement module of the visual cue open set detection model to obtain the mixed text features output by the vector replacement module.

[0041] S250: Inputting the image features and the mixed text features into the detector module of the visual cue open set detection model to train the visual cue open set detection model.

[0042] See also Figure 4 , Figure 4 It is a comparative schematic diagram of the model training method before and after improvement provided in the embodiment of the present application.

[0043] like Figure 4 As shown in the figure, compared with the existing training scheme, this scheme can pre-generate the fusion prompt vector corresponding to the sample text before model training by pre-building the fusion prompt vector dictionary. During the entire model training and inference process, for samples of different batches, there is no need to recalculate the prompt vector each time, but directly obtain the pre-calculated fusion prompt vector from the fusion prompt vector dictionary.

[0044] Compared with the existing solutions, this solution avoids the process of recalculating the hint vector during each batch of sample training or inference, saving a lot of computing resources and time costs. At the same time, the method of preloading the fused hint vector from the fused hint vector dictionary also reduces the complexity of model training and inference, and improves the overall efficiency and performance of the model.

[0045] The model training method provided in the embodiment of the present application no longer extracts images and corresponding annotation information from the training set or the validation set during the model training process, and there is no need to calculate and generate visual cue vectors. Instead, the fused cue vectors corresponding to the sample texts are directly obtained from a pre-built fused cue vector dictionary, and mixed text features are generated based on the fused cue vectors and text features. The visual cue open set detection model is then trained based on the image features and the mixed text features, thereby avoiding overfitting of the model or evaluation distortion, improving the generalization performance of the visual cue open set detection model, and reducing the model training overhead.

[0046] In some embodiments, before obtaining the sample image and sample text, it also includes: obtaining sample text and multiple sample images; inputting the multiple sample images into the image encoding module of the visual cue open set detection model to obtain the image features of each sample image output by the image encoding module; inputting the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module; performing zero-sample inference on the text features and the image features of each sample image to generate multiple first images; performing image sampling fusion on the multiple first images to generate a target image; based on the target image and text features, generating a fused prompt vector corresponding to the sample text; and constructing a fused prompt vector dictionary based on the fused prompt vector.

[0047] The core idea of ​​this application is: without changing the existing visual cue open set detection model backbone, a fusion cue vector dictionary pre-built before model training is introduced, and the fusion cue vector dictionary contains fusion cue vectors corresponding to sample texts. Therefore, before model training and reasoning, it is necessary to pre-build a fusion cue vector dictionary. The purpose of pre-building a fusion cue vector dictionary is to pre-load the fusion vector representation of key information before model training, so as to improve the efficiency and accuracy of model training and reasoning. The fusion cue vector pre-encodes the text information into a vector form and constructs a corresponding dictionary, so that the model can directly obtain the vector representation from the dictionary during training and reasoning, avoiding the process of repeated calculation and processing of text information, and improving the efficiency and accuracy of the model.

[0048] See also Figure 5 , Figure 5 It is a schematic diagram of the construction process of the fusion hint vector dictionary provided in the embodiment of the present application.

[0049] like Figure 5 As shown, in this embodiment, the steps of constructing a fusion hint vector dictionary are as follows: Step 1: Data collection and preprocessing.

[0050] First, a dataset containing different text categories is collected and preprocessed.

[0051] Optionally, the preprocessing includes operations such as word segmentation and stop word removal.

[0052] Step 2: Text extraction.

[0053] For each text sample, the keywords are extracted as the index of the fused hint vector dictionary.

[0054] Optionally, methods such as TF-IDF and TextRank are used for keyword extraction.

[0055] Step 3: Add non-keywords.

[0056] Randomly select non-key nouns in the text as negative sample categories to improve the model's ability to distinguish rare category features.

[0057] It can be understood that after completing the step of adding non-keywords, a variety of sample texts can be obtained, and the sample texts are divided into positive samples and negative samples.

[0058] Among them, negative samples are non-key nouns, and positive samples are key nouns.

[0059] Step 4: Dictionary construction.

[0060] Each sample text and the fused hint vector corresponding to each sample text are stored in the fused hint vector dictionary, thus completing the construction of the fused hint vector dictionary.

[0061] Specifically, for each sample text, if a fusion hint vector corresponding to the sample text is to be generated, the sample text and a plurality of sample images need to be obtained first.

[0062] Furthermore, multiple sample images are input into the image encoding module of the visual cue open set detection model to obtain image features of each sample image output by the image encoding module; sample texts are input into the text encoding module of the visual cue open set detection model to obtain text features output by the text encoding module.

[0063] Furthermore, through the module self-reuse mechanism, the open set detection module of the existing visual cue open set detection model is used to perform zero-sample inference on the text features and the image features of each sample image, and multiple first images are inferred.

[0064] Furthermore, image sampling and fusion are performed on the multiple first images to generate a target image.

[0065] Specifically, image filtering and fusion are performed on the multiple first images obtained by inference to obtain enhanced image information and generate a target image.

[0066] Furthermore, based on the target image and text features, a fused prompt vector corresponding to the sample text is generated.

[0067] It can be understood that for each sample text, it is necessary to obtain the fused prompt vector corresponding to the sample text, and store the fused prompt vector corresponding to each sample text in the fused prompt vector dictionary, so as to construct the final fused prompt vector dictionary.

[0068] The model training method provided in the embodiment of the present application pre-constructs a fusion prompt vector dictionary before model training and reasoning. Compared with the prior art, the scheme of pre-constructing a fusion prompt vector dictionary has the following advantages: (1) Accelerating model training and reasoning. By pre-constructing a fusion prompt vector dictionary, the vector representation pre-loading of text key information is realized, avoiding the process of repeated calculation and processing of text information during training and reasoning, thereby improving the efficiency of the model. (2) Improving model accuracy. The vector representation of text key information (i.e., the fusion prompt vector) can better capture the semantic and contextual information of the text, which helps to improve the accuracy and generalization ability of the model in various text-related tasks. (3) Enhancing the model's ability to recognize rare categories. By randomly selecting non-key nouns in the text as negative sample categories, the model can be helped to better identify and distinguish rare categories, thereby improving the robustness and generalization ability of the model.

[0069] In some embodiments, zero-sample inference is performed on text features and image features of each sample image to generate multiple first images, including: inputting the text features and the image features of each sample image into a contrast supervision layer of a visual cue open set detection model to obtain candidate text features and multiple candidate image features output by the contrast supervision layer; inputting the candidate text features and multiple candidate image features into a detector module of the visual cue open set detection model to obtain multiple first images output by the detector module; wherein the feature similarity between the candidate image features and the candidate text features is greater than a preset threshold; the contrast supervision layer is used to calculate the feature similarity of each image feature and text feature respectively, and based on the feature similarity between each image feature and text feature, determine and output the candidate image features and candidate text features.

[0070] In this embodiment, for each sample text, after obtaining the text features of the sample text and the image features of each sample image, the open set detection module of the existing visual cue open set detection model can be reused through the module self-reuse mechanism to perform zero-sample inference on the text features and the image features of each sample image, and obtain multiple first images by inference.

[0071] See also Figure 6 , Figure 6 It is a schematic diagram of the module self-reuse mechanism provided in the embodiment of the present application.

[0072] like Figure 6 As shown in the figure, since the goal of this scheme is to introduce high-quality target images rather than improve the recall rate of detection during zero-shot reasoning, this scheme adds a comparative supervision layer to the visual cue open set detection model to calculate the similarity of image and text features.

[0073] Specifically, Figure 6As shown, for each sample text, the sample text and multiple sample images are first obtained.

[0074] Furthermore, a plurality of sample images are input into an image encoding module of a visual cue open set detection model to obtain image features of each sample image output by the image encoding module.

[0075] At the same time, the sample text is input into the text encoding module of the visual cue open set detection model to obtain the text features of the sample text output by the text encoding module.

[0076] After obtaining the text features of the sample text and the image features of each sample image, the text features and the image features of each sample image are input into the contrast supervision layer of the visual cue open set detection model to obtain candidate text features and multiple candidate image features output by the contrast supervision layer.

[0077] Specifically, the contrast supervision layer includes a feature extractor (the feature extractor can be a convolutional neural network), which can extract features from each image feature and text feature, calculate the feature similarity of each image feature and text feature through a contrast function, and determine and output candidate image features and candidate text features based on the feature similarity of each image feature and text feature.

[0078] If the feature similarity between an image feature and a text feature is greater than a preset threshold, the comparative supervision layer can select the image feature as a candidate image feature and the text feature as a candidate text feature, thereby realizing the screening of image features and text features.

[0079] Optionally, the comparison function of the comparison supervision layer is a triple comparison function, the triple comparison function The calculation formula is as follows: ; in, The feature vector representing the anchor sample, which is the text feature here; The feature vector representing the positive sample, which is the image feature whose feature similarity with the text feature is greater than the preset threshold; represents the feature vector of negative samples, which are image features whose feature similarity with text features is less than or equal to a preset threshold; f(·) represents a feature extractor (such as a convolutional neural network); represents the norm of a vector; represents the margin parameter, which is used to control the distance between negative samples and anchor samples; N is the total number of image features.

[0080] Further, input the candidate text features and multiple candidate image features into the detector module of the visual prompt open-set detection model to obtain multiple first images output by the detector module.

[0081] Specifically, after the input candidate text features and multiple candidate image features undergo self-attention calculation, deformable attention calculation, and cross-modal attention calculation in the detector module, multiple first images will be generated and output, completing the entire detection process.

[0082] In some embodiments, image sampling and fusion are performed on the multiple first images to generate a target image, including: inputting the multiple first images into the image sampling module of the visual prompt open-set detection model for importance sampling to obtain multiple second images output by the image sampling module; inputting the multiple second images into the image fusion module of the visual prompt open-set detection model for image fusion processing to obtain the target image output by the image fusion module.

[0083] Please refer to Figure 7 , Figure 7 which is a schematic flowchart of image sampling and fusion provided by an embodiment of the present application.

[0084] As Figure 7 shown, assuming that the detector module generates N first images (i.e., the image regions in Figure 7 ), then the N first images can be respectively passed through the image sampling module and the image fusion module to obtain a high-quality target image as the subsequent input of the model.

[0085] Specifically, input the multiple first images into the image sampling module of the visual prompt open-set detection model. The image sampling module can perform importance sampling on the multiple first images. Importance sampling refers to weighting the regions of different images according to image quality or other metrics and selecting regions with higher quality as the sampling regions to obtain M (M < N) second images.

[0086] Optionally, the steps of importance sampling are as follows: Step 1: Calculate the weight of each first image.

[0087] Assume that the quality metric of the first image is , and the value range is [0, 1]. Then, for the i-th first image, its weight is , and at this time, the weight of the i-th first image is calculated as follows: ; where N is the number of first images.

[0088] Step 2: Select M images with higher quality as sampling results according to the weights.

[0089] Specifically, since the larger the weight is, the higher the quality of the first image corresponding to the weight is, the top M first images in the weight ranking may be selected as sampling results to obtain M second images.

[0090] Furthermore, the M second images are input into the image fusion module of the visual cue open set detection model. The image fusion module can perform depth image fusion (Depth Fusion) on the M second images, take into account the depth information of multiple second images, and perform weighted fusion on the pixel values ​​to produce a more accurate depth perception result, and finally obtain a high-quality target image.

[0091] Optionally, the steps of image fusion are as follows: Step 1: Perform weighted fusion on each pixel.

[0092] Assume that there are M second images, and the pixel value of the i-th second image is recorded as , the depth information corresponding to the i-th second image is , the weight of the i-th second image is recorded as , then the pixel value obtained after image fusion of M second images is The calculation formula is as follows: ; Step 2: Perform weighted fusion on the depth information of each pixel.

[0093] Depth information can be used to control the weight of pixels, with pixels that are close to each other having higher weights and pixels that are far away having lower weights.

[0094] Assume the pixel distance is , then the pixel value Weight The calculation formula is as follows: .

[0095] In some embodiments, based on the target image and text features, a fused prompt vector corresponding to the sample text is generated, including: inputting the target image into a prompt encoding module of a visual prompt open set detection model to obtain a visual prompt vector output by the prompt encoding module; inputting the visual prompt vector and text features into a prompt fusion module of the visual prompt open set detection model to obtain a fused prompt vector corresponding to the sample text output by the prompt fusion module.

[0096] In the existing model training scheme, the visual cue vector is locally replaced with some of the original text features according to the position of the corresponding category, which can generate mixed text features. The mixed text features carry a certain degree of image semantic information. However, the replaced position only retains the corresponding image features, and lacks the original text features.

[0097] For example, the category of the input text is three words "cat. dog. pig", and the calculated prompt vector is the visual prompt vector corresponding to the word "pig". Then the final mixed text feature is "cat text feature, dog text feature, pig visual prompt vector". At this time, the mixed text feature lacks the text information of the word "pig".

[0098] Based on this, this embodiment adds a prompt fusion module to the visual prompt open set detection model, and fuses the image information and the text information of the corresponding category through the prompt fusion module to obtain a fused prompt vector, so that while generating visual prompts, the text information of the category is retained, so that the final generated mixed text features are "cat text vector, dog text vector, pig fused prompt vector".

[0099] See also Figure 8 , Figure 8 It is a schematic diagram of the generation process of the fusion prompt vector provided in the embodiment of the present application.

[0100] like Figure 8 As shown, after obtaining the target image and text features, the target image is input into the cue encoding module of the visual cue open set detection model to obtain the visual cue vector output by the cue encoding module.

[0101] Furthermore, the visual cue vector and the text feature are input into a cue fusion module of a visual cue open set detection model to obtain a fused cue vector corresponding to the sample text output by the cue fusion module.

[0102] In some embodiments, a prompt fusion module includes a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer and a feedforward neural network layer; a visual prompt vector and a text feature are input into the prompt fusion module of a visual prompt open set detection model to obtain a fused prompt vector corresponding to the sample text output by the prompt fusion module, including: inputting the text feature into the first self-attention layer of the prompt fusion module to obtain the first text feature vector output by the first self-attention layer; inputting the visual prompt vector into the second self-attention layer of the prompt fusion module to obtain the first image feature vector output by the second self-attention layer; inputting the first text feature vector and the first image feature vector into the first cross-attention layer of the prompt fusion module to obtain the second image feature vector output by the first cross-attention layer; inputting the first text feature vector and the first image feature vector into the second cross-attention layer of the prompt fusion module to obtain the second text feature vector output by the second cross-attention layer; inputting the second image feature vector and the second text feature vector into the feedforward neural network layer of the prompt fusion module to obtain the fused prompt vector corresponding to the sample text output by the feedforward neural network layer.

[0103] Please continue reading Figure 8 ,like Figure 8 As shown, the prompt fusion module includes a first self-attention layer, a second self-attention layer, a first cross-attention layer (i.e., text-image cross-attention layer), a second cross-attention layer (i.e., image-text cross-attention layer), and a feed-forward neural network layer (i.e., FFN layer).

[0104] After obtaining the visual cue vector and text features, the text features are input into the first self-attention layer of the cue fusion module, which can perform weighted processing on the text features to generate and output a weighted first text feature vector T; the visual cue vector is input into the second self-attention layer of the cue fusion module, which can perform weighted processing on the visual cue vector to generate and output a weighted first image feature vector I.

[0105] Furthermore, the first text feature vector T is used as the query vector (query), and the first image feature vector I is used as the key vector (key) and the value vector (value), which are input into the text-image cross attention layer of the prompt fusion module, and the text attention score is calculated to generate and output the second image feature vector I weighted by the text attention. text_att .

[0106] Furthermore, the first image feature vector I is used as the query vector (query), and the first text feature vector T is used as the key vector (key) and the value vector (value), which are input into the image-text cross attention layer of the prompt fusion module, the image attention score is calculated, and the second text feature vector T weighted by the image attention is generated and output. image_att .

[0107] Further, the second image feature vector I text_att and the second text feature vector T image_att The feedforward neural network layer is input to the prompt fusion module, and the feedforward neural network layer can be used to transform the second image feature vector I text_att and the second text feature vector T image_att Perform concatenation or attention weighted addition to generate a fused prompt vector corresponding to the sample text.

[0108] Optionally, in the second image feature vector I text_att and the second text feature vector T image_att In the process of weighted addition of attention, the softmax function can be used to normalize the attention score, and the normalized attention score is multiplied by the corresponding feature vector, and finally the weighted feature vector is added. The specific formula of this process is as follows: ; ; ; ; Wherein, TQ represents the query matrix of the first text feature vector T; Represents the image vector I i The key matrix of Represents the image vector I i IQ represents the query matrix of the first image feature vector I; Represents the text vector T i The key matrix of Represents the text vector T i The value matrix of ; N represents the number of eigenvectors; Represents the second text feature vector T image_att Attention score; Represents the second image feature vector I text_att Attention score; Represents the second text feature vector T image_att The feature weights of Represents the second image feature vector I text_att The feature weights of .

[0109] In summary, compared with the prior art, the model training method provided in the embodiment of the present application has at least the following advantages: (1) In the existing scheme, the input data of the prompt encoding module needs to be extracted from the training set or the validation set. If the images in the training set and the corresponding annotation information of the images are extracted, it may cause the model to overfit; if the images in the validation set and the corresponding annotation information of the images are extracted, it may cause the evaluation of the model to be distorted, and there is a situation where the validation set index is high but the test set effect is not good. Both of the above situations will lead to poor generalization performance of the visual prompt open set detection model. However, the embodiment of the present application proposes to use the module self-reuse mechanism and the image sampling fusion method to self-generate high-quality target images, which can avoid overfitting in model training. At the same time, due to the high quality of the target image, the generalization of the model can be further improved during reasoning.

[0110] (2) In the existing scheme, the vector replacement module can generate mixed text features that carry a certain degree of image semantic information, but the replaced position can only retain the corresponding image features, lacking the original text features, resulting in reduced robustness of the model's zero-shot reasoning. The embodiment of the present application proposes a method of using a prompt fusion module and a prompt fusion module architecture, which improves the richness of the corresponding information of the replacement category through deep fusion of prompt vectors and text vectors, so that the final generated mixed features will not lose text information while obtaining visual information, thereby improving the robustness of the model's zero-shot reasoning.

[0111] (3) In the existing scheme, when training the model, for samples of different batches, the model needs to extract image information from the training set or validation set corresponding to each text category, and then calculate and generate the visual cue vector. This process will increase the training cost of the model. The embodiment of the present application proposes a training strategy for preloading cue vectors using a fused cue vector dictionary. After constructing the fused cue vector dictionary, the model can directly obtain vector representations from the fused cue vector dictionary during training and reasoning, avoiding the process of repeated calculation and processing of text information, which can improve the efficiency and accuracy of the model and reduce the training cost of the model.

[0112] The present application also provides a model training device. Fig. 9 , Fig. 9 9 is a schematic diagram of the structure of the model training device provided in the embodiment of the present application. In the embodiment of the present application, the model training device includes an acquisition module 910, an image feature extraction module 920, a text feature extraction module 930, a dictionary extraction module 940 and a training module 950.

[0113] The acquisition module 910 is used to acquire sample images and sample texts.

[0114] The image feature extraction module 920 is used to input the sample image into the image encoding module of the visual cue open set detection model to obtain the image features output by the image encoding module.

[0115] The text feature extraction module 930 is used to input the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module.

[0116] The dictionary extraction module 940 is used to obtain the fused prompt vector corresponding to the sample text from the pre-built fused prompt vector dictionary, and input the fused prompt vector and text features into the vector replacement module of the visual prompt open set detection model to obtain the mixed text features output by the vector replacement module.

[0117] The training module 950 is used to input the image features and the mixed text features into the detector module of the visual cue open set detection model to train the visual cue open set detection model.

[0118] In some embodiments, the dictionary extraction module 940 is used to obtain sample text and multiple sample images; input the multiple sample images into the image encoding module of the visual cue open set detection model to obtain the image features of each sample image output by the image encoding module; input the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module; perform zero-sample inference on the text features and the image features of each sample image to generate multiple first images; perform image sampling fusion on the multiple first images to generate a target image; based on the target image and the text features, generate a fused prompt vector corresponding to the sample text; based on the fused prompt vector, construct a fused prompt vector dictionary.

[0119] In some embodiments, the dictionary extraction module 940 is used to input the text features and the image features of each sample image into the contrast supervision layer of the visual cue open set detection model to obtain candidate text features and multiple candidate image features output by the contrast supervision layer; input the candidate text features and multiple candidate image features into the detector module of the visual cue open set detection model to obtain multiple first images output by the detector module; wherein the feature similarity between the candidate image features and the candidate text features is greater than a preset threshold; the contrast supervision layer is used to calculate the feature similarity of each image feature and text feature respectively, and based on the feature similarity between each image feature and text feature, determine and output the candidate image features and candidate text features.

[0120] In some embodiments, the dictionary extraction module 940 is used to input multiple first images into the image sampling module of the visual cue open set detection model, perform importance sampling, and obtain multiple second images output by the image sampling module; input the multiple second images into the image fusion module of the visual cue open set detection model, perform image fusion processing, and obtain the target image output by the image fusion module.

[0121] In some embodiments, the dictionary extraction module 940 is used to input the target image into the prompt encoding module of the visual prompt open set detection model to obtain the visual prompt vector output by the prompt encoding module; input the visual prompt vector and text features into the prompt fusion module of the visual prompt open set detection model to obtain the fused prompt vector corresponding to the sample text output by the prompt fusion module.

[0122] In some embodiments, the hint fusion module includes a first self-attention layer, a second self-attention layer, a first criss-cross attention layer, a second criss-cross attention layer, and a feed-forward neural network layer.

[0123] The dictionary extraction module 940 is used to input the text feature into the first self-attention layer of the prompt fusion module to obtain the first text feature vector output by the first self-attention layer; input the visual prompt vector into the second self-attention layer of the prompt fusion module to obtain the first image feature vector output by the second self-attention layer; input the first text feature vector and the first image feature vector into the first cross-attention layer of the prompt fusion module to obtain the second image feature vector output by the first cross-attention layer; input the first text feature vector and the first image feature vector into the second cross-attention layer of the prompt fusion module to obtain the second text feature vector output by the second cross-attention layer; input the second image feature vector and the second text feature vector into the feedforward neural network layer of the prompt fusion module to obtain the fusion prompt vector corresponding to the sample text output by the feedforward neural network layer.

[0124] The present application also provides an electronic device, Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the model training method.

[0125] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0126] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the model training method provided by the above-mentioned methods.

[0127] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model training methods provided by the above methods.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that: include: Get sample images and sample text; Inputting the sample image into an image encoding module of a visual cue open set detection model to obtain image features output by the image encoding module; Inputting the sample text into the text encoding module of the visual cue open set detection model to obtain text features output by the text encoding module; Obtaining a fused prompt vector corresponding to the sample text from a pre-built fused prompt vector dictionary, and inputting the fused prompt vector and the text feature into a vector replacement module of the visual prompt open set detection model to obtain a mixed text feature output by the vector replacement module; The image features and the mixed text features are input into a detector module of the visual cue open set detection model to train the visual cue open set detection model.

2. The model training method according to claim 1, characterized in that: Before obtaining the sample image and the sample text, the method further includes: Acquire the sample text and a plurality of the sample images; Inputting the plurality of sample images into an image encoding module of a visual cue open set detection model, and obtaining image features of each of the sample images output by the image encoding module; Inputting the sample text into the text encoding module of the visual cue open set detection model to obtain text features output by the text encoding module; Performing zero-sample inference on the text feature and the image feature of each of the sample images to generate a plurality of first images; Performing image sampling and fusion on a plurality of the first images to generate a target image; Based on the target image and the text features, generating a fusion prompt vector corresponding to the sample text; Based on the fused hint vector, the fused hint vector dictionary is constructed.

3. The model training method according to claim 2, characterized in that: The performing zero-sample inference on the text feature and the image feature of each of the sample images to generate a plurality of first images comprises: Inputting the text features and the image features of each of the sample images into the contrast supervision layer of the visual cue open set detection model to obtain candidate text features and multiple candidate image features output by the contrast supervision layer; Inputting the candidate text feature and the plurality of candidate image features into a detector module of the visual cue open set detection model to obtain a plurality of the first images output by the detector module; Wherein, the feature similarity between the candidate image feature and the candidate text feature is greater than a preset threshold; The comparison supervision layer is used to calculate the feature similarity of each of the image features and the text features respectively, and based on the feature similarity of each of the image features and the text features, determine and output the candidate image features and the candidate text features.

4. The model training method according to claim 2, characterized in that: The performing image sampling and fusion on the plurality of the first images to generate a target image comprises: Inputting the plurality of first images into the image sampling module of the visual cue open set detection model, performing importance sampling, and obtaining a plurality of second images output by the image sampling module; The plurality of second images are input into the image fusion module of the visual cue open set detection model to perform image fusion processing, so as to obtain the target image output by the image fusion module.

5. The model training method according to claim 2, characterized in that: The step of generating a fusion hint vector corresponding to the sample text based on the target image and the text feature includes: Inputting the target image into a cue encoding module of the visual cue open set detection model to obtain a visual cue vector output by the cue encoding module; The visual cue vector and the text feature are input into a cue fusion module of the visual cue open set detection model to obtain a fused cue vector corresponding to the sample text output by the cue fusion module.

6. The model training method according to claim 5, characterized in that: The prompt fusion module includes a first self-attention layer, a second self-attention layer, a first cross-attention layer, a second cross-attention layer and a feedforward neural network layer; The step of inputting the visual cue vector and the text feature into a cue fusion module of the visual cue open set detection model to obtain a fused cue vector corresponding to the sample text output by the cue fusion module comprises: Inputting the text feature into the first self-attention layer of the prompt fusion module to obtain a first text feature vector output by the first self-attention layer; Inputting the visual cue vector into the second self-attention layer of the cue fusion module to obtain a first image feature vector output by the second self-attention layer; Inputting the first text feature vector and the first image feature vector into a first cross attention layer of the prompt fusion module to obtain a second image feature vector output by the first cross attention layer; Inputting the first text feature vector and the first image feature vector into a second cross attention layer of the prompt fusion module to obtain a second text feature vector output by the second cross attention layer; The second image feature vector and the second text feature vector are input into the feedforward neural network layer of the prompt fusion module to obtain a fusion prompt vector corresponding to the sample text output by the feedforward neural network layer.

7. A model training device, characterized in that: include: An acquisition module, used to acquire sample images and sample texts; An image feature extraction module, used for inputting the sample image into an image encoding module of a visual cue open set detection model to obtain image features output by the image encoding module; A text feature extraction module, used for inputting the sample text into the text encoding module of the visual cue open set detection model to obtain the text features output by the text encoding module; A dictionary extraction module, used for obtaining the fused prompt vector corresponding to the sample text from a pre-built fused prompt vector dictionary, and inputting the fused prompt vector and the text feature into the vector replacement module of the visual prompt open set detection model to obtain the mixed text feature output by the vector replacement module; A training module is used to input the image features and the mixed text features into a detector module of the visual cue open set detection model to train the visual cue open set detection model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model training method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6.

Images