Multi-modal detection model training method and device based on small sample prompt fine tuning

Abstract

The embodiment of the invention relates to the field of target detection, in particular to a multi-modal detection model training method and device based on small sample prompt fine tuning. One embodiment of the method comprises the following steps: acquiring a training sample set and an initial multi-modal detection model; executing a training step: obtaining a text semantic embedding vector sequence, and splicing the first-layer text prompt vector sequence and the text semantic embedding vector sequence to obtain an initial text embedding vector sequence; an image block embedding vector sequence group is obtained, the first-layer image prompt vector sequence and the image block embedding vector sequence are spliced, and an initial image embedding vector sequence group is obtained; obtaining an image text similarity matrix group; performing fusion processing on each image text similarity matrix to obtain an image text alignment score matrix; generating a classification annotation image; determining a multi-modal detection loss value; and determining a multi-modal detection model. According to the embodiment, the complexity of a multi-modal detection model can be reduced, and the occupation of computing resources is reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of target detection, and more particularly to a method and apparatus for training a multimodal detection model based on small sample cue fine-tuning. Background Art

[0002] Multimodal detection models can effectively identify unknown categories without relying on a large number of labeled samples. However, when the detection objects are not representative in the original training set or there are significant differences, the multimodal detection model will still face the problem of conceptual inconsistency. In order to solve the problem of conceptual inconsistency faced by the above-mentioned multimodal detection models and enable the multimodal detection models to quickly adapt to downstream detection tasks, researchers have begun to explore small sample fine-tuning strategies. At present, when training multimodal detection models through small sample fine-tuning, the commonly used approach is to keep the pre-trained model parameters unchanged, combine the learnable text prompt vector and the image features learned by the meta-network, or dynamically generate category-aware visual cues through class-aware visual cue tuning to fuse with the original image features.

[0003] However, in practice, it is found that when using the above method to train a multimodal detection model, the following technical problems often occur: If a meta-network is introduced, the model complexity will be higher and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image will need to be processed to generate visual cues related to its content, which will further increase the complexity of the model calculation and thus occupy more computing resources.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a multimodal detection model training method and device based on small sample prompt fine-tuning to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a multimodal detection model training method based on small sample prompt fine-tuning, the method comprising: obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification annotation images, the above-mentioned initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the above-mentioned language image pre-training model includes a language encoder, a visual encoder and a classifier, the above-mentioned language encoder includes a language encoding layer sequence, and the above-mentioned visual encoder includes an image encoding layer sequence and an anchor generator; selecting training samples from the training sample set, and performing the following training steps based on the initial multimodal detection model: inputting the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splicing the preset first-layer text prompt vector sequence with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; inputting the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector The method comprises the steps of: generating a first-layer image hint vector sequence group and concatenating a first-layer image hint vector sequence with each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the first-layer image hint vector sequence is obtained by performing dimensionality transformation on the first-layer text hint vector sequence through a projection layer included in the initial multimodal detection model; inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the parameters of the language image pre-training model are frozen during the training of the initial multimodal detection model; fusing each image-text similarity matrix in the image-text similarity matrix group to obtain an image-text alignment score matrix; generating a classified annotated image based on the image-text alignment score matrix; determining a multimodal detection loss value based on the image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; and determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as a multimodal detection model.

[0008] In a second aspect, some embodiments of the present disclosure provide an image detection method, comprising: obtaining an object image and an object description text corresponding to the object image; inputting the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by the method described in any implementation method of the first aspect.

[0009] In a third aspect, some embodiments of the present disclosure provide a multimodal detection model training device based on small sample prompt fine-tuning, the device comprising: an acquisition unit, configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification annotation images, the above-mentioned initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the above-mentioned language image pre-training model includes a language encoder, a visual encoder and a classifier, the above-mentioned language encoder includes a language encoding layer sequence, and the above-mentioned visual encoder includes an image encoding layer sequence and an anchor generator; a selection and execution unit, configured to select training samples from the training sample set, and perform the following training steps based on the initial multimodal detection model: input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splice the preset first-layer text prompt vector sequence with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; input the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model blocks, obtaining an image block embedding vector sequence group, and concatenating a first-layer image hint vector sequence with each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the first-layer image hint vector sequence is obtained by dimensionally transforming the first-layer text hint vector sequence through a projection layer included in an initial multimodal detection model; inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training of the initial multimodal detection model; fusing each image-text similarity matrix in the image-text similarity matrix group to obtain an image-text alignment score matrix; generating a classified annotated image based on the image-text alignment score matrix; determining a multimodal detection loss value based on the image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; and determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as a multimodal detection model.

[0010] In a fourth aspect, some embodiments of the present disclosure provide an image detection device, comprising: an acquisition unit, configured to acquire an object image and an object description text corresponding to the above-mentioned object image; an input unit, configured to input the above-mentioned object image and the above-mentioned object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the above-mentioned object image, wherein the above-mentioned multimodal detection model is pre-trained by the method described in any implementation method of the first aspect.

[0011] In a fifth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect above.

[0012] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any one of the implementations of the first or second aspect above is implemented.

[0013] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure, the complexity of the model can be reduced, and the occupation of memory and computing resources can be reduced. Specifically, the reason for the high complexity of the model and the occupation of more computing resources is that if a meta-network is introduced, the model complexity will be high, and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image needs to be processed to generate visual cues related to its content, which further makes the model calculation more complex, thereby resulting in the occupation of more computing resources. Based on this, the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure first obtains a training sample set and an initial multimodal detection model. The training samples in the training sample set include sample text, sample images, and sample classified and annotated images. The initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model. The language image pre-training model includes a language encoder, a visual encoder, and a classifier. The language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator. Here, the initial multimodal detection model and sample data to be trained can be obtained for subsequent model training.Then, training samples are selected from the training sample set, and the following training steps are performed based on the initial multimodal detection model: the sample text included in the selected training samples is input into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and the preset first-layer text prompt vector sequence and the above-mentioned text semantic embedding vector sequence are spliced ​​to obtain an initial text embedding vector sequence; the sample image included in the selected training samples is input into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and the first-layer image prompt vector sequence and each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group are spliced ​​to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image prompt vector sequence is obtained through the projection layer included in the initial multimodal detection model. A layer of text prompt vector sequence is obtained by dimensionally transforming the sequence; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; the individual image-text similarity matrices in the above-mentioned image-text similarity matrix group are fused to obtain an image-text alignment score matrix; based on the above-mentioned image-text alignment score matrix, a classified annotated image is generated; based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample and the classified annotated image, a multimodal detection loss value is determined; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model. Therefore, in some embodiments of the present disclosure, the multimodal detection model training method based on small sample prompt fine-tuning, on the basis of freezing the parameters of the language image pre-training model, splices the prompt vector with the image embedding representation and the text embedding representation respectively, and inputs the splicing results into the visual encoder and language encoder included in the language image pre-training model for fine-tuning training. It can achieve the injection of downstream image detection task domain knowledge while maintaining the integrity of the original knowledge of the language image pre-training model. In addition, because the multimodal detection model of the present disclosure also includes a projection layer, the corresponding visual prompts can be coupled with the language prompts to strengthen the collaboration between the visual and language branches, promote the concept alignment between the two modalities, and further, there is no need to introduce other neural networks and dynamically generate visual prompts corresponding to each image. Thereby, the complexity of the multimodal detection model can be reduced, and the memory and computing resources occupied can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of a multimodal detection model training method based on small sample prompt fine-tuning according to the present disclosure; Figure 2 This is a comparison chart of experimental data when the multimodal detection model disclosed herein is used to identify objects with other detection methods; Figure 3 is a flow chart of some embodiments of the image detection method according to the present disclosure; Figure 4 is a schematic structural diagram of some embodiments of a multimodal detection model training device based on small sample prompt fine-tuning according to the present disclosure; Figure 5 is a schematic structural diagram of some embodiments of the image detection device according to the present disclosure; Figure 6 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 The process of some embodiments of the multimodal detection model training method based on small sample prompt fine-tuning according to the present disclosure is shown. The multimodal detection model training method based on small sample prompt fine-tuning 100 includes the following steps: Step 101: Obtain a training sample set and an initial multimodal detection model.

[0023] In some embodiments, an entity (e.g., a computing device) executing a multimodal detection model training method based on small sample prompt fine-tuning can obtain a training sample set and an initial multimodal detection model from a database via a wired or wireless connection. The training samples in the training sample set may include sample text, sample images, and sample classified and annotated images. The sample text may be text describing the sample image. The sample image may be an image to be classified and annotated. The sample classified and annotated image may be the actual result of classifying and annotating the sample image. The initial multimodal detection model may be composed of a text embedding module, an image embedding module, a projection layer, and a language-image pre-trained model. The language-image pre-trained model may include a language encoder, a visual encoder, and a classifier. The text embedding module may be used to embed input text. The image embedding module may be used to embed input images. The projection layer may be used to project the language embedding dimension into the visual embedding dimension. The language encoder may be used to extract text features. The visual encoder may be used to extract image features. The classifier may be used to align the outputs of the language encoder and the visual encoder and output the image classification and annotation results.

[0024] Furthermore, the language encoder may include a sequence of language coding layers. The sequence of language coding layers may be an ordered collection of language coding layers arranged in execution order. The language coding layers in the sequence of language coding layers may be used to perform feature encoding on text to capture contextual information and temporal relationships within the text. The visual encoder may include a sequence of image coding layers and an anchor generator. The sequence of image coding layers may be an ordered collection of image coding layers arranged in execution order. The image coding layers in the sequence of image coding layers may be used to perform feature encoding on images to capture contextual information and spatial relationships within the images. The anchor generator may be used to generate anchor boxes based on image features and extract regional features corresponding to regions of interest. It should be noted that the wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future developed wireless connection methods.

[0025] As an example, the language-image pre-training model can be a Grounded Language-Image Pre-training (GLIP) model. The language encoder can be a BERT-based module. The visual encoder can be a Swin Transformer model.

[0026] Optionally, each language coding layer in the above-mentioned language coding layer sequence can have a one-to-one correspondence with the image coding layer with the same sequence number in the above-mentioned image coding layer sequence. Each language coding layer in the above-mentioned language coding layer sequence that meets a preset sequence number condition can have a one-to-one correspondence with a preset text prompt vector sequence. The preset sequence number condition can be that the layer sequence number corresponding to the coding layer is less than or equal to the preset sequence number. The preset sequence number can be a pre-set layer sequence number. The layer sequence number can be the sequence number of the coding layer in the coding layer sequence. For example, the preset sequence number can be 9. The text prompt vector sequence can be a vector sequence that is adjusted and optimized through a training process to provide additional context or task information. Each image coding layer in the above-mentioned image coding layer sequence that meets the preset sequence number condition can have a one-to-one correspondence with a preset image prompt vector sequence. The image prompt vector sequence can be obtained by performing a dimensional transformation on the text prompt vector sequence corresponding to the target language coding layer using a projection layer included in the initial multimodal detection model. The target language coding layer can be the language coding layer in the above-mentioned language coding layer sequence whose corresponding sequence number is the same as the sequence number of the image coding layer corresponding to the image prompt vector sequence.

[0027] It should be noted that each text hint vector sequence corresponding to each language coding layer and each image hint vector sequence corresponding to each image coding layer have the same sequence length. Each text hint vector in the text hint vector sequence has the same vector length. The image hint vector and the text hint vector have the same vector length. The above sequence length can be the number of vectors in the sequence. The above vector length can be the number of dimensions of the vector. Except for the text hint vector corresponding to the first language coding layer, the hint vectors of other language coding layers are initialized using a random initialization method. The text hint vector corresponding to the first language coding layer can be initialized by a pre-set text embedding. For example, the text hint vector corresponding to the first language coding layer can be initialized to the text embedding corresponding to the example text "a (a) photo (photo) of a".

[0028] In practice, the multimodal detection model of this disclosure facilitates concept alignment between the two modalities by coupling corresponding image cues with text cues layer by layer and implementing layer-by-layer cross-modal interaction through projection layers. This strengthens the collaboration between the vision and language branches, allowing the multimodal detection model to learn the relationship between the two modal features while gradually extracting image and text features. This improves the classification performance of the multimodal detection model.

[0029] Step 102: Select training samples from the training sample set and perform the following training steps based on the initial multimodal detection model: Step 1021: Input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and concatenate the preset first-layer text prompt vector sequence and the text semantic embedding vector sequence to obtain an initial text embedding vector sequence.

[0030] In some embodiments, the execution subject may input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and perform splicing processing on the preset first-layer text prompt vector sequence and the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence. The text semantic embedding vector in the above-mentioned text semantic embedding vector sequence may be an embedding representation corresponding to text word segmentation. Text word segmentation may be obtained by performing word segmentation on the sample text included in the above-mentioned training sample. The above-mentioned first-layer text prompt vector sequence may be a text prompt vector sequence corresponding to the language coding layer with the corresponding layer number 1 in the language coding layer sequence. The above-mentioned initial text embedding vector sequence may be a sequence composed of various first-layer text prompt vectors and various text semantic embedding vectors. The above-mentioned first-layer text prompt vector sequence and the above-mentioned text semantic embedding vector sequence may be spliced ​​in a horizontal splicing manner to obtain an initial text embedding vector sequence.

[0031] As an example, if the first-layer text prompt vector sequence is {text prompt vector 1, text prompt vector 2}, and the text semantic embedding vector sequence is {text semantic embedding vector 1, text semantic embedding vector 2, text semantic embedding vector 3}, then the initial text embedding vector sequence is {text prompt vector 1, text prompt vector 2, text semantic embedding vector 1, text semantic embedding vector 2, text semantic embedding vector 3}.

[0032] Step 1022: Input the sample images included in the selected training samples into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and perform splicing processing on the first-layer image prompt vector sequence and each image block embedding vector sequence in the image block embedding vector sequence group to obtain an initial image embedding vector sequence group.

[0033] In some embodiments, the execution subject may input the sample images included in the selected training samples into the image embedding module included in the initial multimodal detection model, obtain an image block embedding vector sequence group through various methods, and perform splicing processing on the first-layer image hint vector sequence and each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group. The above-mentioned first-layer image hint vector sequence may be obtained by performing dimensional transformation on the above-mentioned first-layer text hint vector sequence through the projection layer included in the initial multimodal detection model. The above-mentioned first-layer image hint vector sequence may correspond to the image coding layer with the layer number 1 in the image coding layer sequence. The initial image embedding vector sequence in the above-mentioned initial image embedding vector sequence group corresponds one-to-one to the image block embedding vector sequence in the above-mentioned image block embedding vector sequence group. The initial image embedding vector sequence in the above-mentioned initial image embedding vector sequence group may be obtained by splicing the corresponding image block embedding vector sequence with the above-mentioned first-layer image hint vector sequence.

[0034] As an example, the execution entity may first output a group of image block embedding vector sequences based on the sample images included in the selected training samples using an image embedding module included in the initial multimodal detection model. Then, for each image block embedding vector sequence in the group of image block embedding vector sequences, the first-layer image hint vector sequence and the image block embedding vector sequence are horizontally spliced ​​together to obtain an initial image embedding vector sequence.

[0035] In some optional implementations of some embodiments, the execution entity may output a group of image block embedding vector sequences based on sample images included in the selected training samples through an image embedding module included in the initial multimodal detection model: In the first step, a multi-scale transformation is performed on the sample images included in the selected training samples to obtain a reduced-scale sample image and an enlarged-scale sample image. The reduced-scale sample image may be a sample image with a lower resolution after downsampling. The enlarged-scale sample image may be a sample image with a higher resolution after upsampling. The multi-scale transformation can be performed on the sample images included in the selected training samples using an image pyramid method to obtain the reduced-scale sample image and the enlarged-scale sample image.

[0036] The second step is to embed the above-mentioned scaled-down sample image to obtain a first image block embedding vector sequence. The first image block embedding vector in the above-mentioned first image block embedding vector sequence can be an embedded representation of the image block after the above-mentioned scaled-down sample image is divided into blocks. The above-mentioned scaled-down sample image can be embedded using a preset image embedding method to obtain a first image block embedding vector sequence. The above-mentioned image embedding method can include image segmentation and image block embedding processing steps. It should be noted that the image segmentation and image block embedding processing steps are currently well-known technologies that are widely studied and applied, and will not be described in detail here.

[0037] In the third step, embedding processing is performed on the sample images included in the selected training samples to obtain a second sequence of image block embedding vectors. The second image block embedding vectors in the second sequence of image block embedding vectors may be embedded representations of image blocks obtained by segmenting the sample images included in the training samples. The second sequence of image block embedding vectors may be obtained by embedding the sample images included in the selected training samples using the aforementioned image embedding method.

[0038] In the fourth step, embedding processing is performed on the enlarged sample image to obtain a third image block embedding vector sequence. The third image block embedding vectors in the third image block embedding vector sequence may be embedded representations of image blocks obtained by dividing the enlarged sample image into blocks. The third image block embedding vector sequence may be obtained by embedding the enlarged sample image using the aforementioned image embedding method.

[0039] Step 5: Generate an image block embedding vector sequence group based on the first image block embedding vector sequence, the second image block embedding vector sequence, and the third image block embedding vector sequence. The first image block embedding vector sequence, the second image block embedding vector sequence, and the third image block embedding vector sequence can be used as image block embedding vector sequences to obtain the image block embedding vector sequence group.

[0040] Optionally, the projection layer may perform a dimensionality transformation on the first-layer text hint vector sequence according to a pre-set learnable weight matrix using a dimensionality transformation formula to obtain a first-layer image hint vector sequence. The dimensionality transformation formula may be: .

[0041] in, Represents a projection layer. Represents the learnable weight matrix of the projection layer. Represents the sequence of text prompt vectors corresponding to the language encoding layer. The target image coding layer can be the image hint vector sequence corresponding to the target image coding layer. The corresponding language coding layer has the image coding layer with the same layer number.

[0042] Step 1023 : Input the initial text embedding vector sequence and the initial image embedding vector sequence group into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group.

[0043] In some embodiments, the execution entity may input the initial text embedding vector sequence and the initial image embedding vector sequence group into the language image pre-training model included in the initial multimodal detection model in various ways to obtain an image-text similarity matrix group.

[0044] In some optional implementations of some embodiments, the execution entity may input the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model, and obtain an image-text similarity matrix group through the following steps: In the first step, the language encoder in the language image pre-training model performs feature extraction on the initial text embedding vector sequence to obtain a text feature vector sequence. The text feature vectors in the text feature vector sequence can represent the characteristics of the text segmentation corresponding to the sample text.

[0045] In some optional implementations of some embodiments, the execution entity may perform feature extraction processing on the initial text embedding vector sequence using the language encoding layer sequence included in the language encoder in the initial multimodal detection model, using the following steps to obtain a text feature vector sequence: Step 1: Determine the language coding layer with sequence number 1 in the above language coding layer sequence as the current language coding layer.

[0046] Step 2: Based on the above initial text embedding vector sequence and the current language encoding layer, perform the following steps to generate a text feature vector sequence: Sub-step 1: In response to determining that the current language coding layer satisfies a preset first position condition, inputting the previous output text embedding vector sequence into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer. The previous output text embedding vector sequence may be an output text embedding vector sequence output by the previous language coding layer. The previous language coding layer may be a language coding layer in the language coding layer sequence that is adjacent to the current language coding layer and has a layer sequence number smaller than the current language coding layer. The preset first position condition may be that the layer sequence number of the current language coding layer is greater than the preset sequence number.

[0047] Sub-step 2: In response to determining that the current language encoding layer satisfies a preset end position condition, determining the output text embedding vector sequence as a text feature vector sequence. The preset end position condition may be that the current language encoding layer is the last language encoding layer in the language encoding layer sequence.

[0048] Optionally, in response to determining that the current language coding layer does not meet the preset end position condition, the execution entity may further determine a next language coding layer as the current language coding layer and execute the text feature vector sequence generation step again. The next language coding layer may be a language coding layer in the language coding layer sequence that is adjacent to the current language coding layer and has a higher layer sequence number than the current language coding layer.

[0049] Optionally, the above execution entity may further perform the following sub-steps: Sub-step 1: In response to determining that the current language encoding layer satisfies a preset second position condition, inputting the initial text embedding vector sequence into the current language encoding layer to obtain an output text embedding vector sequence corresponding to the current language encoding layer. The preset second position condition may be that the current language encoding layer is the first language encoding layer in the sequence of language encoding layers.

[0050] Sub-step 2: determining the next language encoding layer as the current language encoding layer, and executing the above text feature vector sequence generation step again.

[0051] Optionally, the above execution entity may further perform the following sub-steps: Sub-step one, in response to determining that the current language coding layer satisfies the preset third position condition, based on the text prompt vector sequence corresponding to the current language coding layer, the previous output text embedding vector sequence is updated to obtain the text embedding vector sequence to be input. The above-mentioned preset third position condition may be that the layer sequence number corresponding to the current language coding layer is greater than 1 and less than or equal to the above-mentioned preset sequence number. The above-mentioned text embedding vector sequence to be input may be the input data of the current language coding layer. The first preset value vectors of the previous output text embedding vector sequence may be discarded and replaced with the text prompt vector sequence corresponding to the current language coding layer to obtain the text embedding vector sequence to be input. The above-mentioned preset value may be a pre-set number of vectors. The above-mentioned preset value may be the sequence length of the text prompt vector sequence.

[0052] Sub-step 2: input the above-mentioned text embedding vector sequence to be input into the current language encoding layer to obtain the output text embedding vector sequence corresponding to the current language encoding layer.

[0053] Sub-step three: determining the next language encoding layer as the current language encoding layer, and executing the above text feature vector sequence generation step again.

[0054] In the second step, for each initial image embedding vector sequence in the above initial image embedding vector sequence group, perform the following steps: In the first sub-step, each initial image embedding vector in the initial image embedding vector sequence is encoded by the image coding layer sequence included in the above-mentioned visual encoder to obtain an image block feature vector sequence. The image block feature vectors in the above-mentioned image block feature vector sequence can correspond one-to-one to the initial image embedding vector in the above-mentioned initial image embedding vector sequence. The image block feature vectors in the above-mentioned image block feature vector sequence can represent the characteristics of the image block. Specifically, the following sub-steps can be performed through the image coding layer sequence included in the visual encoder in the initial multimodal detection model: Sub-step 1: determining the image coding layer numbered 1 in the above image coding layer sequence as the current image coding layer.

[0055] Sub-step 2: Based on the above initial image embedding vector sequence and the current image coding layer, perform the following steps to generate an image block feature vector sequence: In the first step, in response to determining that the current image coding layer satisfies a preset first sequence position condition, a previous output image embedding vector sequence is input into the current image coding layer to obtain an output image embedding vector sequence corresponding to the current image coding layer. The previous output image embedding vector sequence may be an output image embedding vector sequence output by the previous image coding layer. The previous image coding layer may be an image coding layer in the image coding layer sequence that is adjacent to the current image coding layer and has a layer sequence number smaller than the current image coding layer. The preset first image coding layer position condition may be that the layer sequence number of the current image coding layer is greater than the preset sequence number.

[0056] In a second step, in response to determining that the current image coding layer satisfies a preset end-of-sequence position condition, the output image embedding vector sequence is determined as an image block feature vector sequence. The preset end-of-sequence position condition may be that the current image coding layer is the last image coding layer in the image coding layer sequence.

[0057] Optionally, in response to determining that the current image coding layer does not meet the preset end sequence position condition, the execution entity may further determine a next image coding layer as the current image coding layer and execute the image block feature vector sequence generation step again. The next image coding layer may be an image coding layer in the image coding layer sequence that is adjacent to the current image coding layer and has a layer sequence number greater than that of the current image coding layer.

[0058] Optionally, the above execution entity may further perform the following steps: In a first step, in response to determining that the current image coding layer satisfies a preset second sequence position condition, the initial image embedding vector sequence is input into the current image coding layer to obtain an output image embedding vector sequence corresponding to the current image coding layer. The preset second sequence position condition may be that the current image coding layer is the first image coding layer in the image coding layer sequence.

[0059] In the second step, the next image coding layer is determined as the current image coding layer, and the above-mentioned image block feature vector sequence generation step is performed again.

[0060] Optionally, the above execution entity may further perform the following steps: In the first step, in response to determining that the current image coding layer satisfies a preset third sequence position condition, the previous output image embedding vector sequence is updated based on the image hint vector sequence corresponding to the current image coding layer to obtain the image embedding vector sequence to be input. The preset third sequence position condition may be that the layer sequence number corresponding to the current language coding layer is greater than 1 and less than or equal to the preset sequence number. The image embedding vector sequence to be input may be the input data of the current image coding layer. The first preset number of vectors of the previous output image embedding vector sequence may be discarded and replaced with the image hint vector sequence corresponding to the current image coding layer to obtain the image embedding vector sequence to be input.

[0061] In the second step, the above-mentioned image embedding vector sequence to be input is input into the current image coding layer to obtain the output image embedding vector sequence corresponding to the current image coding layer.

[0062] The third step is to determine the next image coding layer as the current image coding layer, and perform the above-mentioned image block feature vector sequence generation step again.

[0063] In a second sub-step, the anchor generator included in the visual encoder performs region-of-interest pooling on each image block feature vector in the image block feature vector sequence to obtain an image region feature vector sequence. The image region feature vectors in the image region feature vector sequence can represent features of the region of interest in the sample image.

[0064] The third sub-step is to determine the image-text similarity matrix between the text feature vector sequence and the image region feature vector sequence through the classifier in the language image pre-training model. The image-text similarity matrix can be a matrix with each text feature vector as a row, each image region feature vector as a column, and the similarity between the text feature vector and the image region feature vector as an element. First, the text feature vector sequence and the image region feature vector sequence are mapped to the same embedding space through the fully connected layer included in the classifier to obtain a mapped text feature vector sequence and a mapped image region feature vector sequence. The mapped text feature vectors in the mapped text feature vector sequence correspond one-to-one to the text feature vectors in the text feature vector sequence. The mapped image region feature vectors in the mapped image region feature vector sequence correspond one-to-one to the image region feature vectors in the image region feature vector sequence. Then, the mapped text feature vector sequence and the mapped image region feature vector sequence are matched to obtain each image-text feature vector pair. Each image-text feature vector pair can be composed of a mapped text feature vector sequence and a mapped image region feature vector. Afterwards, the similarity between each pair of image-text feature vectors can be determined using the cosine similarity method. Finally, the image-text similarity matrix is ​​obtained by taking each text feature vector as a row, each image region feature vector as a column, and the corresponding similarity between the text feature vector and the image region feature vector as the matrix element.

[0065] Step 1024 , performing fusion processing on each image-text similarity matrix in the image-text similarity matrix group to obtain an image-text alignment score matrix.

[0066] In some embodiments, the execution entity may fuse the individual image-text similarity matrices in the image-text similarity matrix group to obtain an image-text alignment score matrix. The image-text alignment score matrix may be the result of weighted averaging of the individual image-text similarity matrices. The image-text similarity matrices in the image-text similarity matrix group may be fused according to a preset image scale weight information group to obtain an image-text alignment score matrix. The image scale weight information may be information on the weights of images of various scales. It should be noted that, considering that the downstream tasks of the present disclosure generally require attention to image details, a relatively large weight value may be set for low-scale reduced images.

[0067] It should be noted that the above-mentioned execution entity first generates multiple images of different scales based on the sample image. This facilitates the subsequent learning of the features of objects of different scales in the image. Then, for each image in the different scale images, feature extraction is performed by the visual encoder to obtain an image feature sequence, and each obtained image feature sequence is matched with the text feature sequence for similarity. In this way, the image-text similarity matrices corresponding to the images of different scales can be obtained. Finally, the image-text similarity matrices are fused into an image-text alignment score matrix. Therefore, by fusing the matching results of images and texts of different scales, a more accurate image-text matching result can be obtained. Furthermore, the objects in the image can be classified and labeled more accurately.

[0068] Step 1025: Generate a classified annotated image based on the image-text alignment score matrix.

[0069] In some embodiments, the execution entity may generate a classified annotated image based on the image-text alignment score matrix. The classified annotated image may be a sample image that is predicted and annotated with anchor boxes and corresponding classification labels. The classified annotated image may be generated based on the image-text alignment score matrix, referring to the process of generating an image with bounding boxes and classification labels using the GLIP model.

[0070] As an example, the execution entity may generate a classified annotated image based on the image-text alignment score matrix by performing the following steps: The first step is to generate each bounding box information. Each bounding box information includes the coordinates of the center point of the image area, the width of the area, and the height of the area. First, select the elements that meet the first preset score condition from the above-mentioned image text alignment score matrix as target elements to obtain a target element group. The above-mentioned first preset score condition may be that the elements in the above-mentioned image text alignment score matrix are greater than the preset score threshold. The above-mentioned preset score threshold may be the lower limit value of the preset image text alignment score. Then, for each target element in the above-mentioned target element group, the image area feature vector corresponding to the above-mentioned target element is used as the target image area feature vector, and the image area center point coordinates, area width, and area height of the image area corresponding to the above-mentioned target image area feature vector are determined as bounding box information.

[0071] The second step is to generate various classification label information. The classification label information may be the classification label information corresponding to the image area. For each bounding box information in the various bounding box information, a text feature vector that meets the second preset score condition can be selected from the image-text alignment score matrix as the target text feature vector based on the image area feature vector corresponding to the bounding box information, and the text segmentation corresponding to the target text feature vector is determined as the classification label information corresponding to the above-mentioned bounding box information. The second preset score condition may be that the image-text alignment score between the text feature vector and the image area feature vector is the maximum value among the various image-text alignment scores corresponding to the image area feature vectors.

[0072] The third step is to annotate the image based on the bounding box information and the classification label information to obtain a classified annotated image. First, based on the bounding box information, each bounding box can be drawn in the sample image. Then, next to each bounding box, the classification label corresponding to the bounding box is added as text based on the classification label information corresponding to the bounding box.

[0073] Step 1026 : Determine a multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotated image corresponding to the selected training sample, and the classification annotated image.

[0074] In some embodiments, the execution entity may determine a multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotated image corresponding to the selected training sample, and the classification annotated image. The multimodal detection loss value may represent the difference between the actual classification annotated result and the predicted classification annotated result.

[0075] As an example, the execution subject may first determine the classification loss value based on the sample classification annotation image and the classification annotation image corresponding to the selected training sample through a preset first loss function. The first loss function may be a bounding box regression loss function. Then, the image-text alignment loss value is determined based on the image-text alignment score matrix through a preset second loss function. The second loss function may be an image-text alignment loss function. The image-text alignment loss value may characterize the confidence between the most likely aligned word segmentation and image region. Finally, the sum of the classification loss value and the image-text alignment loss value is determined as the multimodal detection loss value. The image-text alignment loss function may be expressed by the following formula: .

[0076] in, Represents the image-text alignment loss function. Represents the image-text alignment score matrix. Represents the Softmax activation function. Represents the maximum value function. Represents a logarithmic function.

[0077] While solving the above-mentioned technical problem 1 mentioned in the background technology part, the present disclosure also takes into account that common multimodal detection models usually have technical problem 2: "When there is inconsistency between visual representation and language representation in a shared space, or when the prompt vector learning relies too much on a certain encoding layer, it is easy to make the matching results of the image and text have large errors, further making the classification accuracy of the multimodal detection model low, thus making it difficult to accurately identify and label objects in the image." Therefore, in the face of the above-mentioned technical problem 2, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decided to adopt the following solution.

[0078] In some optional implementations of some embodiments, the execution entity may determine the multimodal detection loss value based on the image-text alignment score matrix, the sample classification annotated image corresponding to the selected training sample, and the classification annotated image through the following steps: In the first step, a first loss value is determined based on the sample classification annotated image and the classification annotated image corresponding to the selected training sample using a preset classification loss function. The classification loss function may be a bounding box regression loss function, and the first loss value may be the classification loss value.

[0079] In the second step, a second loss value is determined based on the image-text alignment score matrix using a preset image-text consistency loss function. The second loss value can represent the distance difference between image features and text features in a shared space. The image-text consistency loss function can be: .

[0080] in, Represents the second loss value. Indicates the number of image regions. Represents the number of word vectors. Indicates the sequence number of the image area. Indicates the ordinal number of the word segment. Indicates the Image features of an image region. Indicates the The text features corresponding to each word.

[0081] The third step is to obtain each current text hint vector sequence and each current image hint vector. The current text hint vector sequence can be a text hint vector sequence obtained from each language encoder of the language encoder for the current iteration round. The current image hint vector sequence can be a image hint vector sequence obtained from each visual encoder of the visual encoder for the current iteration round.

[0082] In the fourth step, a third loss value is determined based on the above-mentioned current text hint vector sequences and the above-mentioned current image hint vector sequences using a preset hint vector fine-tuning loss function. The third loss value can be used to constrain the size and stability of the hint vector. The hint vector fine-tuning loss function can be: .

[0083] in, Represents the third loss value. Indicates the total number of encoding layers in the language encoder or visual encoder. Indicates the sequence number of the coding layer. Represents the current sequence of text prompt vectors corresponding to the language encoding layer. Represents the current image hint vector sequence corresponding to the image coding layer. Indicates the The current text prompt vector sequence corresponding to the language encoding layer. Indicates the The layer image encodes the current image hint vector sequence corresponding to the layer. represents the L2 norm.

[0084] The fifth step is to perform weighted summation on the first loss value, the second loss value, and the third loss value to obtain a multimodal detection loss value. The multimodal detection loss value can be obtained by a pre-defined formula: .

[0085] in, Represents the multimodal detection loss value. Represents the first loss value. and represents the weight hyperparameter.

[0086] The above-mentioned multimodal detection loss value generation step and its related content, as an inventive feature of an embodiment of the present disclosure, address the second technical problem mentioned above: the difficulty in accurately identifying and labeling objects in images. This difficulty arises from the fact that when visual and linguistic representations are inconsistent in a shared space, or when the hint vector learning relies too heavily on a particular encoding layer, the image-text matching results can easily be inaccurate, further reducing the classification accuracy of the multimodal detection model. This solution, by resolving these issues, can achieve accurate object identification in images. To achieve this, the present disclosure designs an image-text consistency loss function based on the classification loss. This function is used to keep the image and text representations as close as possible during model fine-tuning, thereby increasing their synergy. Furthermore, a hint vector fine-tuning loss function is designed to constrain the size and stability of the hint vector, ensuring balanced learning across the entire network and avoiding over-reliance on a single layer. This reduces the error in subsequent image-text matching results, thereby improving the classification accuracy of the multimodal detection model. Consequently, objects in images can be accurately identified and labeled.

[0087] Step 1027 : In response to determining that the multimodal detection loss value is less than the preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model.

[0088] In some embodiments, the execution entity may, in response to determining that the multimodal detection loss value is less than a preset loss threshold, determine the trained initial multimodal detection model as the multimodal detection model in various ways. The preset loss threshold may be a preset upper limit of the multimodal detection loss value.

[0089] Optionally, the execution entity may further adjust the learnable network parameters in the initial multimodal detection model in response to determining that the multimodal detection loss value is greater than or equal to the preset loss threshold, and reselect training samples from the training sample set, and use the adjusted initial multimodal detection model as the initial multimodal detection model to continue executing the training step, wherein the learnable network parameters may include various text prompt vectors, various image prompt vectors, and projection layer parameters. The projection layer parameters may be the weight matrix of the projection layer.

[0090] As an example, the aforementioned execution entity can adjust the learnable network parameters in the initial multimodal detection model using gradient descent and the AdamW optimizer. Furthermore, during prompt fine-tuning, the depth of the two branch prompts (i.e., the preset sequence number) can be set to 9, and the length of the text prompt and image prompt can be set to 3. All model parameters in the GLIP model, except for the prompt vector, are frozen, and the learning rate of the unfrozen parameters can be set to 0.0001.

[0091] Continuing, as an example, Figure 2 The following figure shows the experimental data comparison of the multimodal detection model of the present disclosure and other detection methods when identifying objects. Figure 2 The data sets, comparative detection methods, experimental methods and detection accuracy data used in the comparative experiments are shown in . The data sets include data set 1 and data set 2. Data set 1 can be the ODinW13 data set, and data set 2 can be the ODinW35 data set. ODinW35 and ODinW13 can be subsets of the common OdinW (Object Detection in the Wild) data set. The OdinW data set is a collection of various data sets used for target detection. The comparative detection methods include method 1, method 2 and baseline methods. Method 1 is the DyHeadT method. Method 2 is the DINOSwinT method. The baseline method is the GLIPT method. The experimental method is to use "n-shot" for small sample learning experiments, and n takes values ​​of 1, 3, 5, and 10. n-shot can represent that the model needs to use n samples for training to learn an item category. By Figure 2 It can be seen from the displayed detection accuracy data that the multimodal detection model disclosed in the present invention is significantly superior to the experimental comparison method.

[0092] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure, the complexity of the model can be reduced, and the occupation of memory and computing resources can be reduced. Specifically, the reason for the high complexity of the model and the occupation of more computing resources is that if a meta-network is introduced, the model complexity will be high, and additional memory and computing resources will be consumed. If class-aware visual cue tuning is used to dynamically generate category-aware visual cues, each image needs to be processed to generate visual cues related to its content, which further makes the model calculation more complex, thereby resulting in the occupation of more computing resources. Based on this, the multimodal detection model training method based on small sample cue fine-tuning of some embodiments of the present disclosure first obtains a training sample set and an initial multimodal detection model. The training samples in the training sample set include sample text, sample images, and sample classified and annotated images. The initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model. The language image pre-training model includes a language encoder, a visual encoder, and a classifier. The language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator. Here, the initial multimodal detection model and sample data to be trained can be obtained for subsequent model training.Then, training samples are selected from the training sample set, and the following training steps are performed based on the initial multimodal detection model: the sample text included in the selected training samples is input into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and the preset first-layer text prompt vector sequence and the above-mentioned text semantic embedding vector sequence are spliced ​​to obtain an initial text embedding vector sequence; the sample image included in the selected training samples is input into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and the first-layer image prompt vector sequence and each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group are spliced ​​to obtain an initial image embedding vector sequence group, wherein the above-mentioned first-layer image prompt vector sequence is obtained through the projection layer included in the initial multimodal detection model. A layer of text prompt vector sequence is obtained by dimensionally transforming the sequence; the above-mentioned initial text embedding vector sequence and the above-mentioned initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the above-mentioned language image pre-training model freezes parameters during the training process of the initial multimodal detection model; the individual image-text similarity matrices in the above-mentioned image-text similarity matrix group are fused to obtain an image-text alignment score matrix; based on the above-mentioned image-text alignment score matrix, a classified annotated image is generated; based on the above-mentioned image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample and the classified annotated image, a multimodal detection loss value is determined; in response to determining that the above-mentioned multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model. Therefore, the multimodal detection model training method based on small sample prompt fine-tuning of some embodiments of the present disclosure, on the basis of freezing the parameters of the language image pre-training model, splices the prompt vector with the image embedding representation and the text embedding representation respectively, and inputs the splicing results into the visual encoder and language encoder included in the language image pre-training model for fine-tuning training. It can achieve the injection of downstream image detection task domain knowledge on the basis of maintaining the integrity of the original knowledge of the language image pre-training model. In addition, because the multimodal detection model of the present disclosure preferably includes a projection layer, the corresponding visual prompts can be coupled with the language prompts to strengthen the collaboration between the visual and language branches, promote the concept alignment between the two modalities, and further, there is no need to introduce other neural networks and dynamically generate visual prompts corresponding to each image. Thereby, the complexity of the multimodal detection model can be reduced, and the memory and computing resources occupied can be reduced.

[0093] refer to Figure 3 The process of some embodiments of the image detection method according to the present disclosure is shown. The image detection method 300 includes the following steps: Step 301: Acquire an item image and an item description text corresponding to the item image.

[0094] In some embodiments, an entity executing the image detection method (e.g., a computing device) can obtain an object image and a corresponding object description from a client via a wired or wireless connection. The object image can be an image of the target object to be detected, and the object description can be textual information entered by the user describing the target object to be detected.

[0095] Step 302: Input the item image and item description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the item image.

[0096] In some embodiments, the execution entity may input the item image and the item description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the item image. The multimodal detection model is pre-trained using the method of steps 101-1027.

[0097] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the image detection method of some embodiments of the present disclosure can more accurately identify the target object, improve the recognition efficiency of the target object, and reduce the excessive occupation of computing resources due to inaccurate recognition.

[0098] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multimodal detection model training device based on small sample prompt fine-tuning. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the multimodal detection model training device 400 based on small sample prompt fine-tuning can be specifically applied to various electronic devices.

[0099] like Figure 4As shown, in some embodiments, a multimodal detection model training apparatus 400 based on small sample prompt fine-tuning includes: an acquisition unit 401 and a selection and execution unit 402 . Among them, the acquisition unit 401 is configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample image and sample classification annotation image, and the above-mentioned initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, and the above-mentioned language image pre-training model includes a language encoder, a visual encoder and a classifier, and the above-mentioned language encoder includes a language encoding layer sequence, and the above-mentioned visual encoder includes an image encoding layer sequence and an anchor generator; the selection and execution unit 402 is configured to select training samples from the training sample set, and perform the following training steps based on the initial multimodal detection model: input the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splice the preset first-layer text prompt vector sequence with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; input the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image block embedding vector sequence group, and The first-layer image cue vector sequence is concatenated with each image block embedding vector sequence in the above-mentioned image block embedding vector sequence group to obtain an initial image embedding vector sequence group, wherein the first-layer image cue vector sequence is obtained by performing dimensionality transformation on the first-layer text cue vector sequence through a projection layer included in the initial multimodal detection model; the initial text embedding vector sequence and the initial image embedding vector sequence group are input into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the parameters of the language image pre-training model are frozen during the training process of the initial multimodal detection model; the image-text similarity matrices in the above-mentioned image-text similarity matrix group are fused to obtain an image-text alignment score matrix; a classified annotated image is generated based on the image-text alignment score matrix; a multimodal detection loss value is determined based on the image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image; and in response to determining that the multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model.

[0100] It is understandable that the units recorded in the multimodal detection model training device 400 based on small sample prompt fine-tuning are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal detection model training device 400 based on small sample prompt fine-tuning and the units contained therein, and will not be repeated here.

[0101] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image detection device. These device embodiments are similar to Figure 3 Corresponding to the method embodiments shown, the image detection device 500 can be specifically applied to various electronic devices.

[0102] like Figure 5 As shown, an image anomaly detection apparatus 500 in some embodiments includes: an acquisition unit 501 and an input unit 502. The acquisition unit 501 is configured to acquire an object image and an object description text corresponding to the object image; the input unit 502 is configured to input the object image and the object description text into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained using the method of steps 101-1027.

[0103] It is understood that the units described in the image anomaly detection device 500 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image anomaly detection device 500 and the units included therein, and will not be described in detail here.

[0104] Further references Figure 6 , which shows a structural diagram of an electronic device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0106] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0107] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0108] It should be noted that in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0109] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0110] The computer-readable medium may be included in the device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, as an example, the electronic device may execute: obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample text, sample images and sample classification annotation images, the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer and a language image pre-training model, the language image pre-training model includes a language encoder, a visual encoder and a classifier, the language encoder includes a language encoding layer sequence, and the visual encoder includes an image encoding layer sequence and an anchor generator; selecting training samples from the training sample set, and performing the following training steps based on the initial multimodal detection model: inputting the sample text included in the selected training sample into the text embedding module included in the initial multimodal detection model to obtain a text semantic embedding vector sequence, and splicing the preset first-layer text prompt vector sequence with the above-mentioned text semantic embedding vector sequence to obtain an initial text embedding vector sequence; inputting the sample image included in the selected training sample into the image embedding module included in the initial multimodal detection model to obtain an image A method for detecting a plurality of image block embedding vector sequences is provided, wherein the plurality of image block embedding vector sequences are obtained by performing a dimensional transformation on the plurality of text hint vector sequences through a projection layer included in an initial multimodal detection model. The method further comprises: inputting the plurality of initial text embedding vector sequences and the plurality of initial image embedding vector sequences into a language image pre-training model included in the initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training of the initial multimodal detection model. The method further comprises: fusing the plurality of image-text similarity matrices in the plurality of image-text similarity matrix group to obtain an image-text alignment score matrix. The method further comprises: generating a classified annotated image based on the image-text alignment score matrix. The method further comprises: determining a multimodal detection loss value based on the image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image. The method further comprises: determining a multimodal detection loss value based on the image-text alignment score matrix, the sample classified annotated image corresponding to the selected training sample, and the classified annotated image. In response to determining that the multimodal detection loss value is less than a preset loss threshold, determining the trained initial multimodal detection model as a multimodal detection model.

[0111] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units described in some embodiments of the present disclosure may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor comprising: an acquisition unit and a selection and execution unit. The names of these units do not, in some cases, limit the units themselves. For example, the acquisition unit may also be described as a "unit for acquiring a training sample set and an initial multimodal detection model."

[0114] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal detection model training method based on small sample prompt fine-tuning, comprising: Obtaining a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample texts, sample images, and sample classification and annotation images, and the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model; Select training samples from the training sample set and perform the following training steps based on the initial multimodal detection model: According to the text embedding module included in the initial multimodal detection model and the preset first-layer text prompt vector sequence, the sample text included in the selected training sample is subjected to text embedding processing to obtain an initial text embedding vector sequence; According to the image embedding module and the first-layer image prompt vector sequence included in the initial multimodal detection model, image embedding processing is performed on the sample images included in the selected training samples to obtain an initial image embedding vector sequence group, wherein the first-layer image prompt vector sequence is obtained by dimensionally transforming the first-layer text prompt vector sequence through the projection layer included in the initial multimodal detection model; Inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training process of the initial multimodal detection model; Determine a multimodal detection loss value based on the image-text similarity matrix group and the sample classification annotation images corresponding to the selected training samples; In response to determining that the multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model.

2. The method according to claim 1, wherein: The language image pre-training model includes a language encoder and a visual encoder, the language encoder includes a language encoding layer sequence, the visual encoder includes an image encoding layer sequence, each language encoding layer in the language encoding layer sequence corresponds to an image encoding layer in the image encoding layer sequence, each language encoding layer in the language encoding layer sequence that meets a preset sequence number condition corresponds to a preset text prompt vector sequence, each image coding layer in the image coding layer sequence that meets the preset sequence number condition corresponds to an image prompt vector sequence, the image prompt vector sequence is obtained by dimensionally transforming a text prompt vector sequence corresponding to a target language encoding layer through a projection layer included in an initial multimodal detection model, and the target language coding layer is a language coding layer whose corresponding sequence number in the language coding layer sequence is the same as the sequence number of the image coding layer corresponding to the image prompt vector sequence.

3. The method according to claim 2, wherein: The method further comprises: In response to determining that the multimodal detection loss value is greater than or equal to the preset loss threshold, adjusting the learnable network parameters in the initial multimodal detection model, and reselecting training samples from the training sample set, using the adjusted initial multimodal detection model as the initial multimodal detection model, and continuing to perform the training step, wherein the learnable network parameters include various text prompt vectors, various image prompt vectors and projection layer parameters.

4. The method according to claim 2, wherein: The language image pre-training model further includes a classifier, and the visual encoder further includes an anchor generator; and the initial text embedding vector sequence and the initial image embedding vector sequence group are input into the language image pre-training model included in the initial multimodal detection model to obtain an image text similarity matrix group, including: Performing feature extraction processing on the initial text embedding vector sequence through the language encoder in the language image pre-training model to obtain a text feature vector sequence; For each initial image embedding vector sequence in the initial image embedding vector sequence group, performing the following steps: By using the image coding layer sequence included in the visual encoder in the language image pre-training model, encoding each initial image embedding vector in the initial image embedding vector sequence is performed to obtain an image block feature vector sequence; By using the anchor point generator included in the visual encoder, each image block feature vector in the image block feature vector sequence is subjected to a region of interest pooling process to obtain an image region feature vector sequence; An image-text similarity matrix between the text feature vector sequence and the image region feature vector sequence is determined through a classifier in the language image pre-training model.

5. The method according to claim 4, wherein: The step of performing feature extraction processing on the initial text embedding vector sequence through the language encoder in the language image pre-training model to obtain a text feature vector sequence includes: Determine the language coding layer with a sequence number of 1 in the language coding layer sequence as the current language coding layer; Based on the initial text embedding vector sequence and the current language encoding layer, the following text feature vector sequence generation steps are performed: In response to determining that the current language coding layer satisfies the preset first position condition, inputting the previous output text embedding vector sequence into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer, wherein the previous output text embedding vector sequence is an output text embedding vector sequence output by the previous language coding layer; In response to determining that the current language encoding layer satisfies a preset end position condition, the output text embedding vector sequence corresponding to the current language encoding layer is determined as a text feature vector sequence.

6. The method according to claim 5, wherein: The method further comprises: In response to determining that the current language coding layer does not meet the preset end position condition, the next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

7. The method according to claim 5, wherein: The method further comprises: In response to determining that the current language coding layer satisfies the preset second position condition, inputting the initial text embedding vector sequence into the current language coding layer to obtain an output text embedding vector sequence corresponding to the current language coding layer; The next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

8. The method according to claim 5, wherein: The method further comprises: In response to determining that the current language coding layer satisfies the preset third position condition, based on the text prompt vector sequence corresponding to the current language coding layer, the previous output text embedding vector sequence is updated to obtain the to-be-input text embedding vector sequence; Inputting the to-be-input text embedding vector sequence into the current language encoding layer to obtain an output text embedding vector sequence corresponding to the current language encoding layer; The next language coding layer is determined as the current language coding layer, and the text feature vector sequence generation step is performed again.

9. An image detection method, comprising: Acquire an item image and an item description text corresponding to the item image; The object image and the object description text are input into a pre-trained multimodal detection model to obtain a classified annotated image corresponding to the object image, wherein the multimodal detection model is pre-trained by any one of the methods described in claims 1-8.

10. A multimodal detection model training device based on small sample prompt fine-tuning, comprising: An acquisition unit is configured to acquire a training sample set and an initial multimodal detection model, wherein the training samples in the training sample set include sample texts, sample images, and sample classification and annotation images, and the initial multimodal detection model is composed of a text embedding module, an image embedding module, a projection layer, and a language image pre-training model; The selection and execution unit is configured to select training samples from the training sample set and perform the following training steps based on the initial multimodal detection model: According to the text embedding module included in the initial multimodal detection model and the preset first-layer text prompt vector sequence, the sample text included in the selected training sample is subjected to text embedding processing to obtain an initial text embedding vector sequence; According to the image embedding module and the first-layer image prompt vector sequence included in the initial multimodal detection model, image embedding processing is performed on the sample images included in the selected training samples to obtain an initial image embedding vector sequence group, wherein the first-layer image prompt vector sequence is obtained by dimensionally transforming the first-layer text prompt vector sequence through the projection layer included in the initial multimodal detection model; Inputting the initial text embedding vector sequence and the initial image embedding vector sequence group into a language image pre-training model included in an initial multimodal detection model to obtain an image-text similarity matrix group, wherein the language image pre-training model freezes parameters during the training process of the initial multimodal detection model; Determine a multimodal detection loss value based on the image-text similarity matrix group and the sample classification annotation images corresponding to the selected training samples; In response to determining that the multimodal detection loss value is less than a preset loss threshold, the trained initial multimodal detection model is determined as the multimodal detection model.

Images