Multimodal detection model training method and device based on small sample prompt fine-tuning

Abstract

The embodiments of the present disclosure relate to the field of target detection, and specifically to a multimodal detection model training method and device based on small sample prompt fine-tuning. One implementation of the method includes: obtaining a training sample set and an initial multimodal detection model; executing training steps: obtaining a text semantic embedding vector sequence, splicing the first-layer text prompt vector sequence with the text semantic embedding vector sequence to obtain an initial text embedding vector sequence; obtaining an image block embedding vector sequence group, splicing the first-layer image prompt vector sequence with the image block embedding vector sequence to obtain an initial image embedding vector sequence group; obtaining an image-text similarity matrix group; fusing each image-text similarity matrix to obtain an image-text alignment score matrix; generating a classified annotated image; determining a multimodal detection loss value; and determining a multimodal detection model. This implementation can reduce the complexity of the multimodal detection model and reduce the occupation of computing resources.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of target detection, and more particularly to a method and apparatus for training a multimodal detection model based on small sample cue fine-tuning. Background Art

[0002] Multimodal detection models can effectively identify unknown categories without relying on a large number of labeled samples. However, when the detection object is not representative or has significant differences in the original training set, the multimodal detection model will still face the problem of conceptual inconsistency. In order to solve the problem of conceptual inconsistency faced by the above-mentioned multimodal detection model and enable the multimodal detection model to quickly adapt to downstream detection tasks, researchers have begun to explore small sample fine-tuning strategies. At present, when training multimodal detection models through small sample fine-tuning, the usual approach is to keep the pre-trained model parameters unchanged, combine the learnable text prompt vector and the image features learned by the meta-network, or dynamically generate category-aware visual cues through class-aware visual cue tuning to fuse with the original image features.

[0003] However, in practice, it is found that when the above method is used to train a multimodal detection model, the following technical problems often occur:

[0004] If a meta-network is introduced, the model complexity will be higher and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image needs to be processed to generate visual cues related to its content, which further increases the complexity of the model calculation and consumes more computing resources.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a multimodal detection model training method and device based on small sample prompt fine-tuning to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a multimodal detection model training method based on small sample prompt fine-tuning, the method comprising: obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification annotated images, the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the language image pre-training model includes a language encoder, a visual encoder and a classifier, the language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator; selecting training samples from the training sample set, and performing the following training steps based on the initial multimodal detection model: inputting the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and concatenating a preset first-layer text prompt vector sequence with the text semantic embedding vector sequence to obtain an initial text embedding vector sequence; inputting the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector. A quantity sequence group is prepared, and a first-layer image hint vector sequence is concatenated with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image hint vector sequence is obtained by dimensionally transforming the above-mentioned first-layer text hint vector sequence through a projection layer included in the initial multimodal detection model; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; each image-text similarity matrix in the above-mentioned image-text similarity matrix group is fused to obtain an image-text alignment score matrix; a classified annotated image is generated based on the above-mentioned image-text alignment score matrix; a multimodal detection loss value is determined based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as a multimodal detection model.

[0009] In a second aspect, some embodiments of the present disclosure provide an image detection method, the method comprising: acquiring an object image and an object description text corresponding to the object image; inputting the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by the method described in any implementation method of the first aspect.

[0010] In a third aspect, some embodiments of the present disclosure provide a multimodal detection model training device based on small sample prompt fine-tuning, the device comprising: an acquisition unit, configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification annotated images, the above-mentioned initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the above-mentioned language image pre-training model includes a language encoder, a visual encoder and a classifier, the above-mentioned language encoder includes a language encoding layer sequence, and the above-mentioned visual encoder includes an image encoding layer sequence and an anchor generator; a selection and execution unit, configured to select training samples from the training sample set, and based on the initial multimodal detection model, perform the following training steps: input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and concatenate the preset first-layer text prompt vector sequence with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; input the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model The invention relates to a method for detecting a plurality of image blocks, obtaining an image block embedding vector sequence group, and concatenating a first-layer image hint vector sequence with each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the first-layer image hint vector sequence is obtained by dimensionally transforming the first-layer text hint vector sequence through a projection layer included in an initial multimodal detection model; inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in the initial multimodal detection model to obtain an image text similarity matrix group, wherein the language image pre-training model freezes parameters during the training process of the initial multimodal detection model; fusing each image text similarity matrix in the image text similarity matrix group to obtain an image text alignment score matrix; generating a classified annotated image based on the image text alignment score matrix; determining a multimodal detection loss value based on the image text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; and determining the trained initial multimodal detection model as a multimodal detection model in response to determining that the multimodal detection loss value is less than a preset loss threshold.

[0011] In a fourth aspect, some embodiments of the present disclosure provide an image detection device, comprising: an acquisition unit, configured to acquire an object image and an object description text corresponding to the object image; an input unit, configured to input the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by the method described in any implementation method of the first aspect.

[0012] In a fifth aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect above.

[0013] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation of the first aspect or the second aspect is implemented.

[0014] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure, the complexity of the model can be reduced, and the occupation of memory and computing resources can be reduced. Specifically, the reasons for the high complexity of the model and the occupation of more computing resources are: if a meta-network is introduced, the model complexity will be higher, and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image needs to be processed to generate visual cues related to its content, which further makes the model calculation more complex, thereby resulting in the occupation of more computing resources. Based on this, the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure first obtains a training sample set and an initial multimodal detection model. Among them, the training samples in the training sample set include sample text, sample images and sample classification annotated images. The initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model. The language image pre-training model includes a language encoder, a visual encoder and a classifier. The language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator. Here, the initial multimodal detection model and sample data to be trained can be obtained for subsequent model training.Then, training samples are selected from the training sample set, and the following training steps are performed based on the initial multimodal detection model: the sample text included in the selected training samples is input into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and the preset first-layer text prompt vector sequence is spliced ​​with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; the sample image included in the selected training samples is input into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and the first-layer image prompt vector sequence is spliced ​​with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image prompt vector sequence is obtained through the projection layer included in the initial multimodal detection model, and the above-mentioned first-layer image prompt vector sequence is spliced ​​with the above-mentioned first-layer image prompt vector sequence. A layer of text prompt vector sequence is obtained by dimension transformation; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; each image-text similarity matrix in the above-mentioned image-text similarity matrix group is fused to obtain an image-text alignment score matrix; based on the above-mentioned image-text alignment score matrix, a classified annotated image is generated; based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample and the classified annotated image, a multimodal detection loss value is determined; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as a multimodal detection model. Therefore, in some embodiments of the present disclosure, the multimodal detection model training method based on small sample prompt fine-tuning, on the basis of freezing the parameters of the language image pre-training model, splices the prompt vector with the image embedding representation and the text embedding representation respectively, and inputs the splicing result into the visual encoder and the language encoder included in the language image pre-training model for fine-tuning training, thereby achieving the injection of downstream image detection task domain knowledge while maintaining the integrity of the original knowledge of the language image pre-training model. In addition, because the multimodal detection model of the present disclosure also includes a projection layer, the corresponding visual prompts can be coupled with the language prompts to strengthen the synergy between the visual and language branches, promote the concept alignment between the two modalities, and further, there is no need to introduce other neural networks and dynamically generate the visual prompts corresponding to each image. Thereby, the complexity of the multimodal detection model can be reduced, and the occupation of memory and computing resources can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is a flowchart of some embodiments of a multimodal detection model training method based on small sample cue fine-tuning according to the present disclosure;

[0017] Figure 2 is a comparison chart of experimental data of the multimodal detection model disclosed in the present invention and other detection methods when identifying objects;

[0018] Figure 3 is a flow chart of some embodiments of the image detection method according to the present disclosure;

[0019] Figure 4 It is a structural schematic diagram of some embodiments of a multimodal detection model training device based on small sample prompt fine-tuning according to the present disclosure;

[0020] Figure 5 is a schematic structural diagram of some embodiments of the image detection device according to the present disclosure;

[0021] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1 The process of some embodiments of the multimodal detection model training method based on small sample prompt fine-tuning according to the present disclosure is shown. The multimodal detection model training method based on small sample prompt fine-tuning 100 includes the following steps:

[0029] Step 101: Obtain a training sample set and an initial multimodal detection model.

[0030] In some embodiments, the execution subject (e.g., computing device) of the multimodal detection model training method based on small sample prompt fine-tuning can obtain a training sample set and an initial multimodal detection model from a database through a wired connection or a wireless connection. The training samples in the training sample set may include sample text, sample image, and sample classification and annotation image. The sample text may be a text describing the sample image. The sample image may be an image to be classified and annotated. The sample classification and annotation image may be a real result after the sample image is classified and annotated. The initial multimodal detection model may be composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model. The language image pre-training model may include a language encoder, a visual encoder, and a classifier. The text embedding module may be used to embed the input text. The image embedding module may be used to embed the input image. The projection layer may be used to project the language embedding dimension to the visual embedding dimension. The language encoder may be used to extract text features. The visual encoder may be used to extract image features. The classifier may be used to align the output results of the language encoder and the visual encoder, and output the classification and annotation results of the image.

[0031] In addition, the above-mentioned language encoder may include a language encoding layer sequence. The above-mentioned language encoding layer sequence may be an ordered set of language encoding layers arranged in an execution order. The language encoding layers in the above-mentioned language encoding layer sequence may be used to feature encode text to capture contextual information and temporal relationships in the text. The above-mentioned visual encoder may include an image encoding layer sequence and an anchor generator. The above-mentioned image encoding layer sequence may be an ordered set of image encoding layers arranged in an execution order. The image encoding layers in the above-mentioned image encoding layer sequence may be used to feature encode images to capture contextual information and spatial relationships in images. The above-mentioned anchor generator may be used to generate anchor boxes on image features and extract regional features corresponding to regions of interest. It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other wireless connection methods currently known or to be developed in the future.

[0032] As an example, the language-image pre-training model may be a GLIP (Grounded Language-Image Pre-training) model. The language encoder may be a BERT-based module. The visual encoder may be a Swin Transformer model.

[0033] Optionally, each language coding layer in the above-mentioned language coding layer sequence may correspond one-to-one to the image coding layer corresponding to the same sequence number in the above-mentioned image coding layer sequence. Wherein, each language coding layer in the above-mentioned language coding layer sequence that meets the preset sequence number condition may correspond one-to-one to the preset text prompt vector sequence. The above-mentioned preset sequence number condition may be that the layer sequence number corresponding to the coding layer is less than or equal to the preset sequence number. The above-mentioned preset sequence number may be a pre-set layer sequence number. The layer sequence number may be the sequence number of the coding layer in the coding layer sequence. For example, the above-mentioned preset sequence number may be 9. The text prompt vector sequence may be a vector sequence that is adjusted and optimized through a training process for providing additional context or task information. Each image coding layer in the above-mentioned image coding layer sequence that meets the above-mentioned preset sequence number condition may correspond one-to-one to the preset image prompt vector sequence. Wherein, the image prompt vector sequence may be obtained by dimensionally transforming the text prompt vector sequence corresponding to the target language coding layer through the projection layer included in the initial multimodal detection model. The target language coding layer may be a language coding layer in the above-mentioned language coding layer sequence whose corresponding sequence number is the same as the sequence number of the image coding layer corresponding to the image prompt vector sequence.

[0034] It should be noted that each text hint vector sequence corresponding to each language coding layer and each image hint vector sequence corresponding to each image coding layer have the same sequence length. Each text hint vector in the text hint vector sequence has the same vector length. The image hint vector and the text hint vector have the same vector length. The above sequence length may be the number of vectors in the sequence. The above vector length may be the number of dimensions of the vector. Except for the text hint vector corresponding to the first language coding layer, the hint vectors of other language coding layers are initialized by random initialization. The text hint vector corresponding to the first language coding layer can be initialized by a pre-set text embedding. For example, the text hint vector corresponding to the first language coding layer can be initialized to the text embedding corresponding to the example text "a (a) photo (photo) of a".

[0035] In practice, the multimodal detection model disclosed in the present invention facilitates the concept alignment between the two modalities by coupling the corresponding image prompts with the text prompts layer by layer and realizing the cross-modal interaction layer by layer through the projection layer, thereby strengthening the collaboration between the visual and language branches, so that the multimodal detection model can learn the relationship between the two modal features while gradually extracting image and text features. Thus, the classification performance of the multimodal detection model can be improved.

[0036] Step 102, selecting training samples from the training sample set, and performing the following training steps based on the initial multimodal detection model:

[0037] Step 1021, input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and concatenate the preset first-layer text prompt vector sequence and the text semantic embedding vector sequence to obtain an initial text embedding vector sequence.

[0038] In some embodiments, the execution subject may input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splice the preset first-layer text prompt vector sequence with the text semantic embedding vector sequence to obtain an initial text embedding vector sequence. Among them, the text semantic embedding vector in the text semantic embedding vector sequence may be an embedding representation corresponding to text word segmentation. Text word segmentation may be obtained by segmenting the sample text included in the training sample. The first-layer text prompt vector sequence may be a text prompt vector sequence corresponding to the language coding layer whose corresponding layer number is 1 in the language coding layer sequence. The initial text embedding vector sequence may be a sequence composed of each first-layer text prompt vector and each text semantic embedding vector. The first-layer text prompt vector sequence and the text semantic embedding vector sequence may be spliced ​​by horizontal splicing to obtain an initial text embedding vector sequence.

[0039] As an example, if the first-layer text prompt vector sequence is {text prompt vector 1, text prompt vector 2}, and the text semantic embedding vector sequence is {text semantic embedding vector 1, text semantic embedding vector 2, text semantic embedding vector 3}, then the initial text embedding vector sequence is {text prompt vector 1, text prompt vector 2, text semantic embedding vector 1, text semantic embedding vector 2, text semantic embedding vector 3}.

[0040] Step 1022, input the sample images included in the selected training samples into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and concatenate the first layer image prompt vector sequence with each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group.

[0041] In some embodiments, the execution subject may input the sample images included in the selected training samples into the image embedding module included in the initial multimodal detection model, obtain an image block embedding vector sequence group through various methods, and perform splicing processing on the first-layer image prompt vector sequence and each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group. The first-layer image prompt vector sequence may be obtained by dimensionally transforming the first-layer text prompt vector sequence through the projection layer included in the initial multimodal detection model. The first-layer image prompt vector sequence may correspond to the image coding layer with the layer number 1 in the image coding layer sequence. The initial image embedding vector sequence in the initial image embedding vector sequence group corresponds one-to-one to the image block embedding vector sequence in the image block embedding vector sequence group. The initial image embedding vector sequence in the initial image embedding vector sequence group may be obtained by splicing the corresponding image block embedding vector sequence with the first-layer image prompt vector sequence.

[0042] As an example, the execution subject may first output an image block embedding vector sequence group according to the sample images included in the selected training samples through the image embedding module included in the initial multimodal detection model. Then, for each image block embedding vector sequence in the image block embedding vector sequence group, the first layer image hint vector sequence is spliced ​​with the image block embedding vector sequence in a horizontal splicing manner to obtain an initial image embedding vector sequence.

[0043] In some optional implementations of some embodiments, the execution subject may output an image block embedding vector sequence group according to the sample images included in the selected training samples through an image embedding module included in the initial multimodal detection model:

[0044] In the first step, a multi-scale transformation is performed on the sample images included in the selected training samples to obtain a reduced-scale sample image and an enlarged-scale sample image. The reduced-scale sample image may be a sample image with a lower resolution after downsampling. The enlarged-scale sample image may be a sample image with a higher resolution after upsampling. The sample images included in the selected training samples may be multi-scale transformed by an image pyramid method to obtain a reduced-scale sample image and an enlarged-scale sample image.

[0045] The second step is to embed the above-mentioned scaled-down sample image to obtain a first image block embedding vector sequence. Among them, the first image block embedding vector in the above-mentioned first image block embedding vector sequence can be an embedded representation of the image block after the above-mentioned scaled-down sample image is divided into blocks. The above-mentioned scaled-down sample image can be embedded using a preset image embedding method to obtain a first image block embedding vector sequence. The above-mentioned image embedding method may include image segmentation and image block embedding processing steps. It should be noted that the image segmentation and image block embedding processing steps belong to the well-known technologies that are currently widely studied and applied, and will not be repeated here.

[0046] The third step is to embed the sample images included in the selected training samples to obtain a second image block embedding vector sequence. The second image block embedding vector in the second image block embedding vector sequence may be an embedding representation of the image block after the sample images included in the training samples are divided into blocks. The sample images included in the selected training samples may be embedded using the above image embedding method to obtain a second image block embedding vector sequence.

[0047] The fourth step is to embed the enlarged scale sample image to obtain a third image block embedding vector sequence. The third image block embedding vector in the third image block embedding vector sequence may be an embedding representation of an image block after the enlarged scale sample image is divided into blocks. The enlarged scale sample image may be embedded by the above-mentioned image embedding method to obtain a third image block embedding vector sequence.

[0048] The fifth step is to generate an image block embedding vector sequence group based on the first image block embedding vector sequence, the second image block embedding vector sequence and the third image block embedding vector sequence. The first image block embedding vector sequence, the second image block embedding vector sequence and the third image block embedding vector sequence can be used as image block embedding vector sequences to obtain an image block embedding vector sequence group.

[0049] Optionally, the projection layer may transform the first-layer text hint vector sequence by a dimension transformation formula according to a preset learnable weight matrix to obtain a first-layer image hint vector sequence. The dimension transformation formula may be: .

[0050] in, Represents a projection layer. Represents the learnable weight matrix of the projection layer. Represents the sequence of text prompt vectors corresponding to the language encoding layer. represents the image hint vector sequence corresponding to the target image coding layer. The above-mentioned target image coding layer can be the image coding layer sequence corresponding to The corresponding language coding layer has the image coding layer with the same layer number.

[0051] Step 1023, inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group.

[0052] In some embodiments, the execution entity may input the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model in various ways to obtain an image-text similarity matrix group.

[0053] In some optional implementations of some embodiments, the execution subject may input the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model, and obtain an image-text similarity matrix group through the following steps:

[0054] In the first step, the language encoder in the language image pre-training model is used to perform feature extraction on the initial text embedding vector sequence to obtain a text feature vector sequence. The text feature vectors in the text feature vector sequence can represent the features of the text segmentation corresponding to the sample text.

[0055] In some optional implementations of some embodiments, the execution subject may perform feature extraction processing on the initial text embedding vector sequence through the language encoding layer sequence included in the language encoder in the initial multimodal detection model, and obtain a text feature vector sequence by using the following steps:

[0056] Step 1: determine the language coding layer with sequence number 1 in the above language coding layer sequence as the current language coding layer.

[0057] Step 2: Based on the above initial text embedding vector sequence and the current language encoding layer, perform the following text feature vector sequence generation steps:

[0058] Sub-step 1: In response to determining that the current language coding layer satisfies the preset first position condition, the previous output text embedding vector sequence is input into the current language coding layer to obtain the output text embedding vector sequence corresponding to the current language coding layer. The above-mentioned previous output text embedding vector sequence may be an output text embedding vector sequence output by the previous language coding layer. The previous language coding layer may be a language coding layer in the language coding layer sequence that is adjacent to the above-mentioned current language coding layer and has a layer sequence number that is smaller than the above-mentioned current language coding layer. The above-mentioned preset first position condition may be that the layer sequence number of the current language coding layer is greater than the above-mentioned preset sequence number.

[0059] Sub-step 2: in response to determining that the current language coding layer satisfies a preset end position condition, determining the output text embedding vector sequence as a text feature vector sequence. The preset end position condition may be that the current language coding layer is the last language coding layer in the language coding layer sequence.

[0060] Optionally, the execution subject may also determine the next language coding layer as the current language coding layer in response to determining that the current language coding layer does not meet the preset end position condition, and execute the text feature vector sequence generation step again. The next language coding layer may be a language coding layer in the language coding layer sequence that is adjacent to the current language coding layer and has a layer number greater than the current language coding layer.

[0061] Optionally, the above execution entity may further perform the following sub-steps:

[0062] Sub-step 1: In response to determining that the current language coding layer satisfies the preset second position condition, the initial text embedding vector sequence is input into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer. The preset second position condition may be that the current language coding layer is the first language coding layer in the language coding layer sequence.

[0063] Sub-step 2: determining the next language coding layer as the current language coding layer, and executing the above text feature vector sequence generation step again.

[0064] Optionally, the above execution entity may further perform the following sub-steps:

[0065] Sub-step one, in response to determining that the current language coding layer satisfies the preset third position condition, based on the text prompt vector sequence corresponding to the current language coding layer, the previous output text embedding vector sequence is updated to obtain the text embedding vector sequence to be input. Among them, the above-mentioned preset third position condition may be that the layer number corresponding to the current language coding layer is greater than 1 and less than or equal to the above-mentioned preset number. The above-mentioned text embedding vector sequence to be input may be the input data of the current language coding layer. The first preset value vectors of the previous output text embedding vector sequence may be discarded and replaced with the text prompt vector sequence corresponding to the current language coding layer to obtain the text embedding vector sequence to be input. The above-mentioned preset value may be a pre-set number of vectors. The above-mentioned preset value may be the sequence length of the text prompt vector sequence.

[0066] Sub-step 2: input the above-mentioned text embedding vector sequence to be input into the current language encoding layer to obtain the output text embedding vector sequence corresponding to the current language encoding layer.

[0067] Sub-step three: determining the next language coding layer as the current language coding layer, and executing the above-mentioned text feature vector sequence generation step again.

[0068] In the second step, for each initial image embedding vector sequence in the above initial image embedding vector sequence group, perform the following steps:

[0069] In the first sub-step, each initial image embedding vector in the initial image embedding vector sequence is encoded through the image coding layer sequence included in the visual encoder to obtain an image block feature vector sequence. The image block feature vectors in the image block feature vector sequence can correspond one-to-one to the initial image embedding vectors in the initial image embedding vector sequence. The image block feature vectors in the image block feature vector sequence can characterize the features of the image blocks. Specifically, the following sub-steps can be performed through the image coding layer sequence included in the visual encoder in the initial multimodal detection model:

[0070] Sub-step 1: determining the image coding layer with sequence number 1 in the above image coding layer sequence as the current image coding layer.

[0071] Sub-step 2: Based on the above initial image embedding vector sequence and the current image coding layer, perform the following steps to generate an image block feature vector sequence:

[0072] In the first step, in response to determining that the current image coding layer satisfies the preset first sequence position condition, the previous output image embedding vector sequence is input into the current image coding layer to obtain the output image embedding vector sequence corresponding to the current image coding layer. The above-mentioned previous output image embedding vector sequence can be the output image embedding vector sequence output by the previous image coding layer. The previous image coding layer can be an image coding layer in the image coding layer sequence that is adjacent to the above-mentioned current image coding layer and has a layer sequence number less than the above-mentioned current image coding layer. The above-mentioned preset first image coding layer position condition can be that the layer sequence number of the current image coding layer is greater than the above-mentioned preset sequence number.

[0073] In the second step, in response to determining that the current image coding layer satisfies a preset end sequence position condition, the output image embedding vector sequence is determined as an image block feature vector sequence. The preset end sequence position condition may be that the current image coding layer is the last image coding layer in the image coding layer sequence.

[0074] Optionally, the execution subject may also determine the next image coding layer as the current image coding layer in response to determining that the current image coding layer does not meet the preset end sequence position condition, and execute the image block feature vector sequence generation step again. The next image coding layer may be an image coding layer in the image coding layer sequence that is adjacent to the current image coding layer and has a layer number greater than the current image coding layer.

[0075] Optionally, the above execution entity may further perform the following steps:

[0076] In the first step, in response to determining that the current image coding layer satisfies a preset second sequence position condition, the initial image embedding vector sequence is input into the current image coding layer to obtain an output image embedding vector sequence corresponding to the current image coding layer. The preset second sequence position condition may be that the current image coding layer is the first image coding layer in the image coding layer sequence.

[0077] In the second step, the next image coding layer is determined as the current image coding layer, and the above-mentioned image block feature vector sequence generation step is performed again.

[0078] Optionally, the above execution entity may further perform the following steps:

[0079] In the first step, in response to determining that the current image coding layer satisfies the preset third sequence position condition, based on the image prompt vector sequence corresponding to the current image coding layer, the previous output image embedding vector sequence is updated to obtain the image embedding vector sequence to be input. Among them, the above-mentioned preset third sequence position condition can be that the layer sequence number corresponding to the current language coding layer is greater than 1 and less than or equal to the above-mentioned preset sequence number. The above-mentioned image embedding vector sequence to be input can be the input data of the current image coding layer. The first preset value vectors of the previous output image embedding vector sequence can be discarded and replaced with the image prompt vector sequence corresponding to the current image coding layer to obtain the image embedding vector sequence to be input.

[0080] In the second step, the above-mentioned image embedding vector sequence to be input is input into the current image coding layer to obtain the output image embedding vector sequence corresponding to the current image coding layer.

[0081] The third step is to determine the next image coding layer as the current image coding layer, and perform the above-mentioned image block feature vector sequence generation step again.

[0082] In the second sub-step, the anchor generator included in the visual encoder performs region of interest pooling processing on each image block feature vector in the image block feature vector sequence to obtain an image region feature vector sequence, wherein the image region feature vector in the image region feature vector sequence can represent the features of the region of interest in the sample image.

[0083] The third sub-step is to determine the image-text similarity matrix between the text feature vector sequence and the image region feature vector sequence through the classifier in the language image pre-training model. The image-text similarity matrix can be a matrix with each text feature vector as a row, each image region feature vector as a column, and the similarity between the text feature vector and the image region feature vector as an element. First, the text feature vector sequence and the image region feature vector sequence are mapped to the same embedding space through the fully connected layer included in the classifier to obtain a mapped text feature vector sequence and a mapped image region feature vector sequence. The mapped text feature vectors in the mapped text feature vector sequence correspond to the text feature vectors in the text feature vector sequence. The mapped image region feature vectors in the mapped image region feature vector sequence correspond to the image region feature vectors in the image region feature vector sequence. Then, the mapped text feature vector sequence and the mapped image region feature vector sequence are matched to obtain each image-text feature vector pair. Each image-text feature vector pair can be composed of a mapped text feature vector sequence and a mapped image region feature vector. Afterwards, the similarity between each pair of image-text feature vectors can be determined by the cosine similarity method. Finally, the image-text similarity matrix is ​​obtained by taking each text feature vector as a row, each image region feature vector as a column, and the corresponding similarity between the text feature vector and the image region feature vector as the element of the matrix.

[0084] Step 1024 , fusing the image-text similarity matrices in the image-text similarity matrix group to obtain an image-text alignment score matrix.

[0085] In some embodiments, the execution entity may fuse the individual image-text similarity matrices in the image-text similarity matrix group to obtain an image-text alignment score matrix. The image-text alignment score matrix may be the result of weighted averaging of the individual image-text similarity matrices. The image-text similarity matrices in the image-text similarity matrix group may be fused according to a preset image scale weight information group to obtain an image-text alignment score matrix. The image scale weight information may be information on the weights of images of various scales. It should be noted that, considering that the downstream tasks of the present disclosure usually need to pay attention to image details, a relatively large weight value may be set for low-scale reduced images.

[0086] It should be noted that the above-mentioned execution entity first generates multiple images of different scales based on the sample image. This facilitates the subsequent learning of the features of objects of different scales in the image. Then, for each image in the different scale images, feature extraction is performed by the visual encoder to obtain an image feature sequence, and each obtained image feature sequence is matched with the text feature sequence for similarity. In this way, each image-text similarity matrix corresponding to the images of different scales can be obtained. Finally, each image-text similarity matrix is ​​fused into an image-text alignment score matrix. Therefore, by fusing the matching results of images and texts of different scales, a more accurate image-text matching result can be obtained. Furthermore, the objects in the image can be classified and labeled more accurately.

[0087] Step 1025: Generate a classified annotated image based on the image-text alignment score matrix.

[0088] In some embodiments, the execution subject may generate a classified annotated image based on the image-text alignment score matrix. The classified annotated image may be a sample image that is predicted to be annotated with each anchor box and a corresponding classification label. The classified annotated image may be generated based on the image-text alignment score matrix by referring to the process of generating an image with a bounding box and a classification label by the GLIP model.

[0089] As an example, the execution subject may generate a classified annotated image based on the image-text alignment score matrix through the following steps:

[0090] The first step is to generate each bounding box information. Wherein, each bounding box information includes the coordinates of the center point of the image area, the width of the area and the height of the area. First, select the elements that meet the first preset score condition from the above-mentioned image text alignment score matrix as the target elements to obtain a target element group. Wherein, the above-mentioned first preset score condition can be that the elements in the above-mentioned image text alignment score matrix are greater than the preset score threshold. The above-mentioned preset score threshold can be a preset lower limit value of the image text alignment score. Then, for each target element in the above-mentioned target element group, the image area feature vector corresponding to the above-mentioned target element is used as the target image area feature vector, and the image area center point coordinates, area width and area height of the image area corresponding to the above-mentioned target image area feature vector are determined as bounding box information.

[0091] The second step is to generate various classification label information. The classification label information may be the classification label information of the corresponding image area. For each of the bounding box information, a text feature vector that meets the second preset score condition may be selected from the image text alignment score matrix as the target text feature vector based on the image area feature vector corresponding to the bounding box information, and the text segmentation corresponding to the target text feature vector may be determined as the classification label information corresponding to the bounding box information. The second preset score condition may be that the image text alignment score between the text feature vector and the image area feature vector is the maximum value among the image text alignment scores corresponding to the image area feature vectors.

[0092] The third step is to annotate the image according to the above-mentioned bounding box information and the above-mentioned classification label information to obtain a classified annotated image. First, each bounding box can be drawn in the above-mentioned sample image according to the above-mentioned bounding box information. Then, next to each bounding box, according to the classification label information corresponding to the bounding box, the classification label corresponding to the bounding box is added to the side of the bounding box in the form of text.

[0093] Step 1026, determining a multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotated image corresponding to the selected training sample, and the classification annotated image.

[0094] In some embodiments, the execution subject may determine a multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotation image and the classification annotation image corresponding to the selected training sample, wherein the multimodal detection loss value may represent the difference between the actual classification annotation result and the predicted classification annotation result.

[0095] As an example, the execution subject may first determine the classification loss value through a preset first loss function according to the sample classification annotated image and the classification annotated image corresponding to the selected training sample. The first loss function may be a bounding box regression loss function. Then, the image-text alignment loss value is determined based on the image-text alignment score matrix through a preset second loss function. The second loss function may be an image-text alignment loss function. The image-text alignment loss value may characterize the confidence between the most likely aligned word segmentation and image region. Finally, the sum of the classification loss value and the image-text alignment loss value is determined as the multimodal detection loss value. The image-text alignment loss function may be expressed by the following formula: .

[0096] in, Represents the image-text alignment loss function. Represents the image-text alignment score matrix. Represents the Softmax activation function. Represents the maximum value function. Represents a logarithmic function.

[0097] While solving the above-mentioned technical problem 1 mentioned in the background technology part, the present disclosure also takes into account that common multimodal detection models usually still have technical problem 2: "When there is inconsistency between visual representation and language representation in a shared space, or when the prompt vector learning is too dependent on a certain encoding layer, it is easy to make the matching results of the image and text have large errors, which further makes the classification accuracy of the multimodal detection model low, thus making it difficult to accurately identify and label objects in the image." Therefore, in the face of the above-mentioned technical problem 2, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decides to adopt the following solution.

[0098] In some optional implementations of some embodiments, the execution subject may determine the multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotated image corresponding to the selected training sample, and the classification annotated image through the following steps:

[0099] In the first step, a first loss value is determined according to the sample classification annotated image and the classification annotated image corresponding to the selected training sample through a preset classification loss function. The classification loss function may be a bounding box regression loss function, and the first loss value may be the classification loss value.

[0100] In the second step, a second loss value is determined according to the image-text alignment score matrix by using a preset image-text consistency loss function. The second loss value can represent the distance difference between the image feature and the text feature in the shared space. The image-text consistency loss function can be: .

[0101] in, Represents the second loss value. Indicates the number of image regions. Represents the number of word vectors. Indicates the sequence number of the image area. Indicates the ordinal number of the word segment. Indicates The image features of an image region. Indicates The text features corresponding to each word.

[0102] The third step is to obtain each current text hint vector sequence and each current image hint vector. Among them, each current text hint vector sequence can be each text hint vector sequence of the current iteration round obtained from each language encoder of the language encoder. Each current image hint vector sequence can be each image hint vector sequence of the current iteration round obtained from each visual encoder of the visual encoder.

[0103] In the fourth step, a third loss value is determined according to each current text hint vector sequence and each current image hint vector sequence by using a preset hint vector fine-tuning loss function. The third loss value can be used to constrain the size and stability of the hint vector. The hint vector fine-tuning loss function can be: .

[0104] in, Represents the third loss value. Represents the total number of encoding layers in the language encoder or visual encoder. Indicates the sequence number of the coding layer. Represents the current sequence of text prompt vectors corresponding to the language encoding layer. Represents the current image hint vector sequence corresponding to the image coding layer. Indicates The current text prompt vector sequence corresponding to the layer language encoding layer. Indicates The layer image encodes the sequence of current image hint vectors corresponding to the layer. represents the L2 norm.

[0105] Step 5: Perform weighted summation on the first loss value, the second loss value and the third loss value to obtain a multimodal detection loss value. The multimodal detection loss value can be obtained by a pre-set formula: .

[0106] in, Represents the multimodal detection loss value. Represents the first loss value. and represents the weight hyperparameter.

[0107] The above-mentioned multimodal detection loss value generation step and its related contents, as an inventive point of an embodiment of the present disclosure, solve the above-mentioned technical problem 2 "difficult to accurately identify and label objects in the image". The reason why it is difficult to accurately identify and label objects in the image is that when there is inconsistency between the visual representation and the language representation in the shared space, or when the hint vector is too dependent on a certain encoding layer when learning, it is easy to make the matching result of the image and the text have a large error, and further make the classification accuracy of the multimodal detection model low. If this solution solves the above-mentioned problem, the effect of accurately identifying objects in the image can be achieved. In order to achieve this effect, the present disclosure designs an image-text consistency loss function on the basis of the classification loss, which is used to make the representation of the image and the text as close as possible during the model fine-tuning process, thereby increasing the synergy between them. Furthermore, a hint vector fine-tuning loss function is also designed to constrain the size and stability of the hint vector, ensuring that the learning of the hint vector is balanced in the entire network and will not be too dependent on a certain layer. As a result, the error of the subsequent image-text matching results can be small, thereby improving the classification accuracy of the multimodal detection model. In addition, the objects in the image can be accurately identified and labeled.

[0108] Step 1027, in response to determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as the multimodal detection model.

[0109] In some embodiments, the execution subject may determine the trained initial multimodal detection model as the multimodal detection model in response to determining that the multimodal detection loss value is less than a preset loss threshold value in various ways. The preset loss threshold value may be a preset upper limit value of the multimodal detection loss value.

[0110] Optionally, the execution subject may also adjust the learnable network parameters in the initial multimodal detection model in response to determining that the multimodal detection loss value is greater than or equal to the preset loss threshold, and reselect training samples from the training sample set, and use the adjusted initial multimodal detection model as the initial multimodal detection model to continue to perform the training step, wherein the learnable network parameters may include various text prompt vectors, various image prompt vectors and projection layer parameters. The projection layer parameters may be the weight matrix of the projection layer.

[0111] As an example, the execution subject can adjust the learnable network parameters in the initial multimodal detection model through the gradient descent method and the AdamW optimizer. In addition, during the prompt fine-tuning process, the setting depth of the two branch prompts, that is, the preset sequence number, can be set to 9, and the length of the text prompt and the image prompt can be set to 3. Except for the prompt vector, all other model parameters in the GLIP model are frozen, and the learning rate of the unfrozen parameters can be set to 0.0001.

[0112] Continuing, as an example, Figure 2 The following is a comparison chart of experimental data of the multimodal detection model disclosed in the present invention and other detection methods when identifying objects. Figure 2 The data sets, comparative detection methods, experimental methods and detection accuracy data used in the comparative experiments are shown in . The data sets include data set 1 and data set 2. Data set 1 can be the ODinW13 data set, and data set 2 can be the ODinW35 data set. ODinW35 and ODinW13 can be subsets of the common OdinW (Object Detection in the Wild) data set. The OdinW data set is a collection of various data sets used for target detection. The comparative detection methods include method 1, method 2 and baseline methods. Method 1 is the DyHeadT method. Method 2 is the DINOSwinT method. The baseline method is the GLIPT method. The experimental method is to use "n-shot" for small sample learning experiments, and n takes values ​​of 1, 3, 5, and 10. n-shot can represent that the model needs to use n samples for training to learn an item category. By Figure 2 It can be seen from the displayed detection accuracy data that the multimodal detection model disclosed in the present invention is significantly better than the experimental comparison method.

[0113] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure, the complexity of the model can be reduced, and the occupation of memory and computing resources can be reduced. Specifically, the reasons for the high complexity of the model and the occupation of more computing resources are: if a meta-network is introduced, the model complexity will be higher, and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image needs to be processed to generate visual cues related to its content, which further makes the model calculation more complex, thereby resulting in the occupation of more computing resources. Based on this, the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure first obtains a training sample set and an initial multimodal detection model. Among them, the training samples in the training sample set include sample text, sample images and sample classification annotated images. The initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model. The language image pre-training model includes a language encoder, a visual encoder and a classifier. The language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator. Here, the initial multimodal detection model and sample data to be trained can be obtained for subsequent model training.Then, training samples are selected from the training sample set, and the following training steps are performed based on the initial multimodal detection model: the sample text included in the selected training samples is input into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and the preset first-layer text prompt vector sequence is spliced ​​with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; the sample image included in the selected training samples is input into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and the first-layer image prompt vector sequence is spliced ​​with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image prompt vector sequence is obtained through the projection layer included in the initial multimodal detection model, and the above-mentioned first-layer image prompt vector sequence is spliced ​​with the above-mentioned first-layer image prompt vector sequence. A layer of text prompt vector sequence is obtained by dimension transformation; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; each image-text similarity matrix in the above-mentioned image-text similarity matrix group is fused to obtain an image-text alignment score matrix; based on the above-mentioned image-text alignment score matrix, a classified annotated image is generated; based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample and the classified annotated image, a multimodal detection loss value is determined; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as a multimodal detection model. Therefore, in some embodiments of the present disclosure, the multimodal detection model training method based on small sample prompt fine-tuning, on the basis of freezing the parameters of the language image pre-training model, splices the prompt vector with the image embedding representation and the text embedding representation respectively, and inputs the splicing result into the visual encoder and the language encoder included in the language image pre-training model for fine-tuning training, thereby achieving the injection of downstream image detection task domain knowledge while maintaining the integrity of the original knowledge of the language image pre-training model. In addition, because the multimodal detection model of the present disclosure preferably includes a projection layer, the corresponding visual prompts can be coupled with the language prompts to strengthen the synergy between the visual and language branches, promote the concept alignment between the two modalities, and further, there is no need to introduce other neural networks and dynamically generate the visual prompts corresponding to each image. Thereby, the complexity of the multimodal detection model can be reduced, and the occupation of memory and computing resources can be reduced.

[0114] refer to Figure 3 The process of some embodiments of the image detection method according to the present disclosure is shown. The image detection method 300 includes the following steps:

[0115] Step 301: Obtain an object image and an object description text corresponding to the object image.

[0116] In some embodiments, the execution subject (e.g., computing device) of the image detection method may obtain an object image and an object description text corresponding to the object image from a client via a wired connection or a wireless connection. The object image may be an image of a target object to be detected. The object description text may be an information text input by the user end describing the target object to be detected.

[0117] Step 302: Input the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image.

[0118] In some embodiments, the execution subject may input the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image. The multimodal detection model is pre-trained by the method of steps 101-1027.

[0119] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the image detection method of some embodiments of the present disclosure can more accurately identify the target object, improve the recognition efficiency of the target object, and reduce the excessive occupation of computing resources caused by inaccurate recognition.

[0120] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multimodal detection model training device based on small sample prompt fine-tuning, and these device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the multimodal detection model training device 400 based on small sample prompt fine-tuning can be specifically applied to various electronic devices.

[0121] like Figure 4As shown, in some embodiments, a multimodal detection model training device 400 based on small sample prompt fine-tuning includes: an acquisition unit 401 and a selection and execution unit 402. The acquisition unit 401 is configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification and annotation images, and the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, and the language image pre-training model includes a language encoder, a visual encoder and a classifier, and the language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator; the selection and execution unit 402 is configured to select training samples from the training sample set, and perform the following training steps based on the initial multimodal detection model: input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splice the preset first layer text prompt vector sequence with the text semantic embedding vector sequence to obtain an initial text embedding vector sequence; input the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and The first layer image hint vector sequence is concatenated with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the above-mentioned first layer image hint vector sequence is obtained by dimensionally transforming the above-mentioned first layer text hint vector sequence through the projection layer included in the initial multimodal detection model; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; each image-text similarity matrix in the above-mentioned image-text similarity matrix group is fused to obtain an image-text alignment score matrix; based on the above-mentioned image-text alignment score matrix, a classified annotated image is generated; based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample and the classified annotated image, a multimodal detection loss value is determined; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as a multimodal detection model.

[0122] It can be understood that the units recorded in the multimodal detection model training device 400 based on small sample prompt fine-tuning are similar to the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal detection model training device 400 based on small sample prompt fine-tuning and the units contained therein, and will not be repeated here.

[0123] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image detection device. These device embodiments are similar to Figure 3 Corresponding to the method embodiments shown, the image detection device 500 can be specifically applied to various electronic devices.

[0124] like Figure 5 As shown, the image anomaly detection device 500 of some embodiments includes: an acquisition unit 501 and an input unit 502. The acquisition unit 501 is configured to acquire an object image and an object description text corresponding to the object image; the input unit 502 is configured to input the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by the method of steps 101-1027.

[0125] It is understandable that the units described in the image anomaly detection device 500 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image anomaly detection device 500 and the units included therein, and will not be described in detail here.

[0126] Further references Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0127] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0128] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0129] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0130] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0131] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0132] The computer-readable medium may be included in the device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, as an example, the electronic device may execute: obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification and annotation images, the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the language image pre-training model includes a language encoder, a visual encoder and a classifier, the language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator; selecting training samples from the training sample set, and performing the following training steps based on the initial multimodal detection model: inputting the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and concatenating a preset first-layer text prompt vector sequence with the text semantic embedding vector sequence to obtain an initial text embedding vector sequence; inputting the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image embedding vector sequence. A group of image block embedding vector sequences, and splicing a first-layer image hint vector sequence with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image hint vector sequence is obtained by dimensionally transforming the above-mentioned first-layer text hint vector sequence through a projection layer included in an initial multimodal detection model; inputting the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; fusing each image-text similarity matrix in the above-mentioned image-text similarity matrix group to obtain an image-text alignment score matrix; generating a classified annotated image based on the above-mentioned image-text alignment score matrix; determining a multimodal detection loss value based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as a multimodal detection model.

[0133] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: an acquisition unit and a selection and execution unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring a training sample set and an initial multimodal detection model".

[0136] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0137] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal detection model training method based on small sample prompt fine-tuning, comprising: Obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample texts, sample images, and sample classification and annotation images, and the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model; Select training samples from the training sample set and perform the following training steps based on the initial multimodal detection model: According to the text embedding module included in the initial multimodal detection model and the preset first-layer text prompt vector sequence, the sample text included in the selected training sample is subjected to text embedding processing to obtain an initial text embedding vector sequence; According to the image embedding module and the first-layer image prompt vector sequence included in the initial multimodal detection model, image embedding processing is performed on the sample images included in the selected training samples to obtain an initial image embedding vector sequence group, wherein the first-layer image prompt vector sequence is obtained by dimensionally transforming the first-layer text prompt vector sequence through the projection layer included in the initial multimodal detection model; Inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training process of the initial multimodal detection model; Determine a multimodal detection loss value based on the image-text similarity matrix group and the sample classification annotation images corresponding to the selected training samples; In response to determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as the multimodal detection model; The language image pre-training model includes a language encoder and a visual encoder, the visual encoder includes an image encoding layer sequence, the language image pre-training model also includes a classifier, and the visual encoder also includes an anchor generator; and the initial text embedding vector sequence and the initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image text similarity matrix group, including: Performing feature extraction processing on the initial text embedding vector sequence through the language encoder in the language image pre-training model to obtain a text feature vector sequence; For each initial image embedding vector sequence in the initial image embedding vector sequence group, performing the following steps: By using the image coding layer sequence included in the visual encoder in the language image pre-training model, encoding each initial image embedding vector in the initial image embedding vector sequence is performed to obtain an image block feature vector sequence; By using the anchor point generator included in the visual encoder, each image block feature vector in the image block feature vector sequence is subjected to a region of interest pooling process to obtain an image region feature vector sequence; An image-text similarity matrix between the text feature vector sequence and the image region feature vector sequence is determined through a classifier in the language image pre-training model.

2. According to the method according to claim 1, the language encoder includes a language coding layer sequence, each language coding layer in the language coding layer sequence corresponds to an image coding layer in the image coding layer sequence, each language coding layer in the language coding layer sequence that meets a preset sequence number condition corresponds to a preset text prompt vector sequence, each image coding layer in the image coding layer sequence that meets the preset sequence number condition corresponds to an image prompt vector sequence, the image prompt vector sequence is obtained by dimensionally transforming the text prompt vector sequence corresponding to the target language coding layer through the projection layer included in the initial multimodal detection model, and the target language coding layer is a language coding layer whose corresponding sequence number in the language coding layer sequence is the same as the sequence number of the image coding layer corresponding to the image prompt vector sequence.

3. The method according to claim 2, wherein: The method further comprises: In response to determining that the multimodal detection loss value is greater than or equal to the preset loss threshold, adjusting the learnable network parameters in the initial multimodal detection model, and reselecting training samples from the training sample set, using the adjusted initial multimodal detection model as the initial multimodal detection model, and continuing to perform the training step, wherein the learnable network parameters include various text prompt vectors, various image prompt vectors and projection layer parameters.

4. The method according to claim 3, wherein: The step of performing feature extraction processing on the initial text embedding vector sequence through the language encoder in the language image pre-training model to obtain a text feature vector sequence includes: Determine the language coding layer with a sequence number of 1 in the language coding layer sequence as the current language coding layer; Based on the initial text embedding vector sequence and the current language encoding layer, the following text feature vector sequence generation steps are performed: In response to determining that the current language coding layer satisfies the preset first position condition, inputting the previous output text embedding vector sequence into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer, wherein the previous output text embedding vector sequence is an output text embedding vector sequence output by the previous language coding layer; In response to determining that the current language encoding layer satisfies a preset end position condition, the output text embedding vector sequence corresponding to the current language encoding layer is determined as a text feature vector sequence.

5. The method according to claim 4, wherein: The method further comprises: In response to determining that the current language coding layer does not meet the preset end position condition, the next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

6. The method according to claim 4, wherein: The method further comprises: In response to determining that the current language coding layer satisfies the preset second position condition, inputting the initial text embedding vector sequence into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer; The next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

7. The method according to claim 4, wherein: The method further comprises: In response to determining that the current language coding layer satisfies the preset third position condition, based on the text prompt vector sequence corresponding to the current language coding layer, the previous output text embedding vector sequence is updated to obtain the to-be-input text embedding vector sequence; Inputting the to-be-input text embedding vector sequence into the current language encoding layer to obtain an output text embedding vector sequence corresponding to the current language encoding layer; The next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

8. An image detection method, comprising: Acquire an item image and an item description text corresponding to the item image; The object image and the object description text are input into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by any one of the methods described in claims 1-7.

9. A multimodal detection model training device based on small sample prompt fine-tuning, comprising: An acquisition unit is configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample texts, sample images, and sample classification and annotation images, and the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model; The selection and execution unit is configured to select training samples from the training sample set and perform the following training steps based on the initial multimodal detection model: According to the text embedding module included in the initial multimodal detection model and the preset first-layer text prompt vector sequence, the sample text included in the selected training sample is subjected to text embedding processing to obtain an initial text embedding vector sequence; According to the image embedding module and the first-layer image prompt vector sequence included in the initial multimodal detection model, image embedding processing is performed on the sample images included in the selected training samples to obtain an initial image embedding vector sequence group, wherein the first-layer image prompt vector sequence is obtained by dimensionally transforming the first-layer text prompt vector sequence through the projection layer included in the initial multimodal detection model; Inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training process of the initial multimodal detection model; Determine a multimodal detection loss value based on the image-text similarity matrix group and the sample classification annotation images corresponding to the selected training samples; In response to determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as the multimodal detection model; The language image pre-training model includes a language encoder and a visual encoder, the visual encoder includes an image encoding layer sequence, the language image pre-training model also includes a classifier, and the visual encoder also includes an anchor generator; and the initial text embedding vector sequence and the initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image text similarity matrix group, including: Performing feature extraction processing on the initial text embedding vector sequence through the language encoder in the language image pre-training model to obtain a text feature vector sequence; For each initial image embedding vector sequence in the initial image embedding vector sequence group, performing the following steps: By using the image coding layer sequence included in the visual encoder in the language image pre-training model, encoding each initial image embedding vector in the initial image embedding vector sequence is performed to obtain an image block feature vector sequence; By using the anchor point generator included in the visual encoder, each image block feature vector in the image block feature vector sequence is subjected to a region of interest pooling process to obtain an image region feature vector sequence; An image-text similarity matrix between the text feature vector sequence and the image region feature vector sequence is determined through a classifier in the language image pre-training model.

Images