Training method of multi-modal target detector, and target detection method and device

Abstract

The invention discloses a training method of a multi-modal target detector and a target detection method and device.The training method of one embodiment comprises the steps that a data training set is obtained, the data training set comprises multiple pieces of training data, and each piece of training data comprises a labeled target image and a corresponding target text; training the multi-modal target detector by using each piece of training data of the data training set, and performing optimization according to a detection result output by the multi-modal target detector until a preset model index is met, including: inputting one of the plurality of pieces of training data into a data preprocessing model and obtaining a visual feature vector and a text feature vector; inputting the training data into a fusion component and obtaining a fusion feature vector, wherein the fusion feature vector comprises instance features of instances of a target image corresponding to the target text; and splicing the visual feature vector, the text feature vector and the fusion feature vector to form a spliced sequence, and inputting the spliced sequence into a multi-modal language model to generate the detection result. And the detection performance of the model is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a training method, target detection method and apparatus for a multimodal target detector. Background Technology

[0002] In traditional visual-language multimodal deep learning object detection methods, the fusion of image features and language features is usually achieved indirectly, such as by concatenating, adding, or fusing the two features through a complex network structure. However, in practical applications, low accuracy and efficiency in object detection are common problems. Improving the accuracy and efficiency of object detection has become a pressing technical issue for those skilled in the art. Summary of the Invention

[0003] To address at least one of the aforementioned problems, a first embodiment of the present invention provides a training method for a multimodal object detector, wherein the multimodal object detector includes a data preprocessing model, a fusion component, and a multimodal language model, and the training method includes:

[0004] Obtain a data training set, which includes multiple training data sets, each of which includes a labeled target image and target text corresponding to the target image;

[0005] The multimodal target detector is trained using the training data from the aforementioned training set to obtain detection results. Based on these detection results, the multimodal language model is optimized until it meets preset model metrics.

[0006] The step of training the multimodal target detector using each training data from the data training set to obtain detection results includes:

[0007] One training data point from multiple training data sets is input into the data preprocessing model to obtain visual feature vectors and text feature vectors;

[0008] The training data is input into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text;

[0009] The visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

[0010] For example, in some embodiments of the training method provided in this application, the step of inputting the training data into the fusion component and obtaining the fusion feature vector further includes:

[0011] The multimodal target detector includes a data preprocessing model. The target image is processed by the data preprocessing model to obtain target detection results, and the target detection results include multiple detection boxes.

[0012] Alignment operation is performed on each detection box of the target image to obtain at least one instance of the target image, instance features and label information corresponding to each instance;

[0013] The target text is searched to obtain category information. The category information and the tag information of each instance are matched to obtain at least one category instance corresponding to the category information.

[0014] A first fused feature vector is generated by superimposing and averaging the instances of each category, and the first fused feature vector is used as the fused feature vector.

[0015] For example, in some embodiments of the training method provided in this application, the label information includes the coordinates of the detection box position and the category name. The step of searching the target text to obtain category information, and matching the category information with the label information of each instance to obtain at least one category instance corresponding to the category information, further includes:

[0016] Instances whose category names are the same as those in the label information are used as category instances, where the category name is the annotation information of the labeled target image.

[0017] For example, in the training method provided in some embodiments of this application, the step of performing a superposition averaging operation on instances of each category to generate a first fused feature vector further includes:

[0018] The instance features of the instances of each category are superimposed to generate an instance feature set;

[0019] The first fused feature vector is generated by performing an arithmetic mean operation on the set of instance features.

[0020] For example, in some embodiments of the training method provided in this application, after performing a superposition averaging operation on instances of each category to generate a first fused feature vector, the training method further includes:

[0021] The text feature vector and the first fused feature vector are superimposed and averaged to generate a second fused feature vector, and the second fused feature vector is used as the fused feature vector.

[0022] For example, in the training method provided in some embodiments of this application, the step of performing a superposition and averaging operation on the text feature vector and the first fused feature vector to generate a second fused feature vector further includes:

[0023] The text feature vector and the first fused feature vector are superimposed to generate a feature vector set;

[0024] The second fused feature vector is generated by performing an arithmetic mean operation on the set of feature vectors.

[0025] For example, in the training methods provided in some embodiments of this application, optimizing the multimodal language model further includes:

[0026] Adjust the model weights of the multimodal language model.

[0027] For example, in the training methods provided in some embodiments of this application, the target text is a positive question text.

[0028] A second embodiment of the present invention provides a target detection method using a multimodal target detector trained by the training method described in the first embodiment. The multimodal target detector includes a data preprocessing model and a multimodal language model. The detection method includes:

[0029] The image to be detected and the text to be detected are input into the data preprocessing model to obtain visual feature vectors and text feature vectors;

[0030] The visual feature vector and the text feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate detection results.

[0031] For example, in some embodiments of the target detection method provided in this application, the detection text is a positive question text.

[0032] A third embodiment of the present invention provides a training apparatus for a multimodal object detector, the multimodal object detector comprising a data preprocessing model, a fusion component, and a multimodal language model, the training apparatus comprising:

[0033] A training set acquisition unit is used to acquire a data training set, which includes multiple training data, each training data including a labeled target image and target text corresponding to the target image;

[0034] The training unit is used to train the multimodal object detector using training data from the training set, and to optimize the multimodal language model based on the detection results output by the multimodal object detector until a preset model metric is met, including:

[0035] Input training data into the data preprocessing model and obtain visual feature vectors and text feature vectors;

[0036] The training data is input into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text;

[0037] The visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

[0038] A fourth embodiment of the present invention provides a multimodal target detection device, which detects a target image by using a multimodal target detector. The multimodal target detector includes a data preprocessing model and a multimodal language model. The multimodal target detection device includes:

[0039] The data preprocessing unit is used to input the image to be detected and the text to be detected into the data preprocessing model to obtain visual feature vectors and text feature vectors.

[0040] The detection unit is used to concatenate the visual feature vector and the text feature vector to form a concatenated sequence, and input it into the multimodal language model to generate detection results.

[0041] The fifth embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon.

[0042] When the program is executed by the processor, it implements the method described in the first embodiment;

[0043] or

[0044] When the program is executed by the processor, it implements the method described in the second embodiment.

[0045] The sixth embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0046] When the processor executes the program, it implements the method as described in the first embodiment;

[0047] or

[0048] When the processor executes the program, it implements the method described in the second embodiment.

[0049] The beneficial effects of this invention are as follows:

[0050] This invention addresses existing problems by developing a training method, object detection method, and apparatus for a multimodal object detector. During training, it generates a fused feature vector using a fusion component based on labeled target images and corresponding target text, achieving precise matching between visual and textual feature vectors. This adjusts the model weights of the multimodal language model, effectively improving the accuracy of object detection, enhancing its robustness and generalization ability. It overcomes the problems in existing technologies and has broad application prospects. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a training method according to an embodiment of the present invention is shown;

[0053] Figure 2 This diagram illustrates the structural block diagram of a multimodal target detector according to an embodiment of the present invention.

[0054] Figure 3 This diagram shows a structural block diagram of the training device according to an embodiment of the present invention;

[0055] Figure 4 A flowchart illustrating a target detection method according to an embodiment of the present invention is shown;

[0056] Figure 5 This diagram illustrates a structural block diagram of a multimodal target detector according to another embodiment of the present invention.

[0057] Figure 6 This diagram shows a structural block diagram of a target detection device according to an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of the structure of a computer device according to another embodiment of the present invention is shown. Detailed Implementation

[0059] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0060] In related technologies, traditional visual-language multimodal deep learning object detection algorithms typically fuse image features and language features indirectly, such as concatenating, adding, or fusing them through complex network structures. However, these methods often fail to accurately match morphemes with image targets, limiting the accuracy of the detection results. To overcome this challenge...

[0061] For example, in a detection scheme based on a multimodal fusion mechanism of co-attention, features are extracted from text and images respectively, and the text and visual information are exchanged through a constructed co-attention converter layer to achieve feature aggregation. However, the text and visual information do not achieve accurate matching between morphemes and image targets during the exchange.

[0062] For example, in a visual localization scheme based on multimodal feature alignment, an object perception module is introduced to influence the fusion result according to attention weights. However, even if a single convolution or attention mechanism is used for feature fusion only in the final stage, accurate matching of morphemes and image targets is not achieved.

[0063] For example, in a multimodal object detection scheme, a vector transformation model is used to transform the encoded set of image or video feature vectors into content that a large language model can understand. The network structure of the vector transformation model includes, but is not limited to, simple linear transformation layers and complex nonlinear transformations. The established mapping translates image features into semantic information, thereby achieving the fusion of the two. However, this scheme is limited to performing specific operations on specific scenes and specific images, and does not describe how the vector transformation model establishes mappings, how it translates image features, or how it fuses with text instructions.

[0064] For example, in a joint multimodal analysis scheme based on adaptive attention fusion, features are extracted from text and images separately. A multi-head self-attention layer is used to operate on text features through a selective fusion mechanism of images and text, and a cross-modal attention layer is used to perform cross-modal interaction between text and visual modalities to achieve fusion of the two. However, this scheme has problems with computational complexity and multimodal data synchronization.

[0065] To address the problems existing in related technologies, the inventors, after extensive research and experimentation, proposed that accurately matching text morphemes and images is one way to improve object detection performance, such as... Figure 1 and Figure 2 As shown, one embodiment of the present invention provides a training method for a multimodal object detector, the multimodal object detector including a data preprocessing model, a fusion component, and a multimodal language model, the training method including:

[0066] Obtain a data training set, which includes multiple training data sets, each of which includes a labeled target image and target text corresponding to the target image;

[0067] The multimodal target detector is trained using the training data from the training set to obtain detection results. Based on these detection results, the multimodal language model is optimized until it meets preset model metrics. The process of training the multimodal target detector using the training data from the training set to obtain detection results includes: inputting one training data point from multiple training data sets into the data preprocessing model to obtain visual feature vectors and text feature vectors; inputting the training data into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text; concatenating the visual feature vector, text feature vector, and fusion feature vector to form a concatenated sequence, and inputting this sequence into the multimodal language model to generate the detection results.

[0068] In this embodiment, as Figure 2 The diagram illustrates the structure of a multimodal object detector during training, including a data preprocessing model, a fusion component, and a multimodal language model. This embodiment adjusts the structure of the multimodal object detector during training. Based on labeled target images and corresponding target text, a fusion component is added to the multimodal object detector. This component forms a fusion feature vector that accurately matches text morphemes and images. This fusion feature vector is then used as one of the inputs to the multimodal language model for training. The multimodal language model is optimized based on the detection results, for example, by adjusting the model weights. This improves the performance of the trained multimodal language model, further enhancing the target detection accuracy, robustness, and generalization ability of the multimodal object detector that does not include a fusion component.

[0069] To further illustrate the specific implementation methods of this application, such as Figure 1 As shown, the specific process of training a multimodal target detector will be used as an example for explanation.

[0070] The first step is to obtain a data training set, which includes multiple training data sets, each of which includes an labeled target image and the target text corresponding to the target image.

[0071] In this embodiment, the target image used for training is labeled, including the coordinates of the detection boxes, the category of each instance corresponding to the detection box, and the target text corresponding to the target image. For example, the target text includes the category of each instance corresponding to the detection box in the target image. This forms a pair of target images and target text, i.e., training data. The target text includes information such as "category" corresponding to the instances in the target image. Here, an instance is an independently identified target object in the target image, such as a single object with an independent target within a detection box identified by target detection. In a specific example, the target image includes 5 birds and 6 people. The labeling information of the target image includes 11 detection boxes, including the position coordinates and category of each detection box. For example, the category of the bird labeling information is "bird," and the category of the person labeling information is "person." Since each bird and each person in the target image is an independent target, the target image includes 11 instances.

[0072] In an optional embodiment, to further improve training efficiency and optimize the multimodal language model, the target text is a positive question text.

[0073] In this embodiment, training with positively descriptive text is more efficient and yields better optimization results compared to training with non-positively descriptive text. Specifically, using the example above, the target text "Find all the birds" is a positive question text, while "Find all targets that are not people" is a negative question text. Using the positive question text, the category "bird" located in the positive question text is easier to fuse with the morphemes of the target text compared to the category "people" located in the negative question text.

[0074] The second step involves training the multimodal target detector using the training data from the training set, and then optimizing the multimodal language model based on the detection results output by the multimodal target detector until the preset model metrics are met.

[0075] In this embodiment, a large amount of training data is used to train and optimize the model weights of the multimodal language model. For example, based on preset model metrics, such as the expected value of the model's loss function, it is determined whether the optimization of the multimodal language model during training makes the multimodal object detector meet the expected value of the loss function. If it does, training ends, and the trained multimodal object detector is used for object detection applications. It is worth noting that the multimodal object detector in this embodiment removes the fusion component after training. In object detection applications, object detection is achieved by using a data preprocessing model and a multimodal speech model to detect objects in the image and text to be detected.

[0076] Specifically, the training process includes the following steps:

[0077] First, input training data into the data preprocessing model and obtain visual feature vectors and text feature vectors.

[0078] In this embodiment, the data preprocessing model employs the CLIP (Contrastive Language-Image Pre-training) model, a multimodal pre-trained neural network model released by OpenAI in 2021, designed to match images and text. The CLIP model can simultaneously understand information from two different modalities—images and text—and establish connections between them. The CLIP model's network structure mainly consists of an image encoder and a text encoder. The image encoder is responsible for converting images into feature vectors, and can be a convolutional neural network (such as ResNet) or a Transformer model (such as ViT). The text encoder is responsible for converting text into feature vectors, typically a Transformer model. These two encoders achieve cross-modal information interaction and fusion by sharing a vector space. The model is pre-trained using contrastive learning, mapping images and text to a unified vector space, allowing the model to directly calculate the similarity between images and text in the vector space without additional intermediate representations. The core principle of the CLIP model involves pre-training with a large amount of paired image and text data to learn the alignment relationships between images and text.

[0079] Using the previous example, the target images, including 5 birds and 6 people, and the target text "find all the birds" are input into the data preprocessing model. The CLIP model's image encoder converts the target images into image feature vectors, such as image tokens, and the CLIP model's text encoder converts the target text into text feature vectors, such as text tokens.

[0080] It is worth noting that this application does not impose specific limitations on the data preprocessing model. Those skilled in the art should select an appropriate data preprocessing model based on actual application needs. The design criteria should be the ability to extract features from images and text, which will not be elaborated here.

[0081] Second, the training data is input into the fusion component and a fusion feature vector is obtained. The fusion feature vector includes instance features of each instance of the target image corresponding to the target text.

[0082] In this embodiment, although the data preprocessing model has performed some matching and multimodal alignment between the image and text, it has not achieved precise matching of morphemes in the target text with the target image. For example, it cannot accurately correspond the target information in the target text with the label information of each instance in the image. Specifically, in the above example, when the data preprocessing model extracts image features from the target image, it obtains features of the entire image including birds and people, rather than precise features of a specific bird or person in the image. This leads to a problem: the association between the "bird" that needs to be queried in the target text and the specific bird and person instances in the image is not obvious during the training process. In other words, the data preprocessing model cannot accurately identify which part of the target image best matches the description in the target text. This limitation is particularly prominent when the target instance occupies a small proportion of the image. When the target instance, such as a bird instance, appears relatively small in the image and the background is complex and varied, the data preprocessing model is prone to confusion between the foreground and background. This means that the data preprocessing model may mistakenly focus its attention on background objects rather than the target instance itself. This not only affects the model's accurate understanding of image content, but also limits its performance in multimodal alignment tasks.

[0083] In this embodiment, during the training process, based on the labeled target image and target text, the data preprocessing model performs refined target detection and feature extraction on the target image to obtain visual features closely related to the target image, extracts morphemes from the target text and constructs semantic features corresponding to the morphemes, and further accurately matches morphemes in the target text with the target image by accurately identifying and locating specific instances in the target image, and fuses the morphemes and target image, thereby effectively improving the accuracy and reliability of multimodal alignment.

[0084] In an optional embodiment, inputting the training data into the fusion component and obtaining the fused feature vector further includes:

[0085] First, the detection boxes of the target image are aligned to obtain at least one instance of the target image, the instance features and label information corresponding to each instance, wherein the detection boxes of the target image are obtained by the data preprocessing model of the multimodal target detector for target detection.

[0086] In this embodiment, considering that the target image including 5 birds and 6 people has multiple targets, especially multiple targets of different categories, in addition to using a data preprocessing model to extract general image features, an alignment operation is further performed to obtain instance features. The RoI Align model used in this embodiment is a technique used in computer vision tasks, especially in the field of object detection and segmentation. It aims to extract specific regions from the input feature map and adjust these regions to a specified output size. The main purpose of RoI Align is to solve the quantization error problem existing in RoIPooling. By dividing the original RoI into multiple small regions and applying bilinear interpolation in each small region, the alignment problem between the RoI and the feature map grid is handled more finely, which can retain more spatial information and improve the resolution of the feature map and the accuracy of the model. In this embodiment, 11 target instances and feature vectors representing the features of the instances are obtained by performing RoI alignment operations. The label information of each target instance is obtained according to the annotation information of the target image. The label information includes the detection box position coordinates and category name of the corresponding target instance. The detection box position coordinates are the specific coordinate positions of the detection box corresponding to the target instance in the target image, and the category name is the category to which the target instance belongs.

[0087] Secondly, the target text is searched to obtain category information, and the category information and the tag information of each instance are matched to obtain at least one category instance corresponding to the category information.

[0088] In this embodiment, considering that in the task setting of multimodal object detection, the content of the target text usually contains the category name to be queried, such as "bird" or "person" in the example above, it is necessary to find specific category instances in the target image that match the target text description. This embodiment uses the label information of each instance obtained in the previous step to assist in localization.

[0089] In an optional embodiment, instances whose category names are the same as the category information are used as category instances, where the category name is the annotation information of the labeled target image.

[0090] In this embodiment, the target text "find all birds" can be used to locate the category name to be queried as "bird". Based on the category name of the tag information of each instance, five instances with the category name "bird" can be found among the 11 instances of 5 birds and 6 people in the above example. For example, by filtering the category names of the tag information of each embodiment based on the category name "bird" provided in the target text, the five instances with the category name "bird" and their corresponding instance characteristics can be identified.

[0091] Next, an average operation is performed on the instances of each category to generate a first fused feature vector, and the first fused feature vector is used as the fused feature vector.

[0092] In this embodiment, based on the category names in the target text obtained in the aforementioned steps, and the category instances identified from the target image based on the category names, the instance features of each category instance are fused. For example, the instance features of each category instance are first superimposed to generate an instance feature set; then, the instance feature set is arithmetically averaged to generate the first fused feature vector. Specifically, the feature matrices of the instance features of the five instances of the category name "bird" are superimposed to generate an instance feature set, and then an arithmetic average is performed on it. The resulting first fused feature vector retains the uniqueness of each instance while reflecting the commonalities of all instances, thus achieving the fusion of the morphemes of the target text and the image features of the target image.

[0093] To further improve the training efficiency and optimization effect of the multimodal language model in this embodiment, in an optional embodiment, the training method further includes: performing a superposition and averaging operation on the text feature vector and the first fused feature vector to generate a second fused feature vector, and using the second fused feature vector as the fused feature vector.

[0094] In this embodiment, a text encoder based on a data preprocessing model extracts features from the target text to generate a text token, i.e., a text feature vector. This text feature vector is then used to perform a second fusion operation, averaging the first fused feature vector obtained in the previous steps, to form a second fused feature vector. That is, the instance features of the target image's category instances enhance the category features of the target text, so that the target text token not only contains linguistic information but also incorporates visual information. Specifically, the text feature vector and the first fused feature vector are superimposed to generate a feature vector set; the arithmetic average of the feature vector set is then performed to generate the second fused feature vector. This embodiment utilizes the above feature fusion method to further improve the model's understanding of the correspondence between the target text and the target image, thereby improving the accuracy and reliability of multimodal target detection.

[0095] Third, the visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

[0096] In this embodiment, the visual feature vector and text feature vector generated by the data preprocessing model are concatenated with the fusion feature vector generated by the fusion component to form a concatenated sequence. The concatenated sequence is used as the input of the multimodal language model and the detection result output by the multimodal language model is obtained. Depending on the target text, the detection result can be a text describing the image, a response to an object in the image, or a dialogue related to the image, etc.

[0097] In this embodiment, the model weights of the multimodal language model are adjusted to optimize the target detector based on the detection results output by the multimodal language model and the preset model metrics. When the detection results output by the multimodal language model meet the model metrics, the training ends and the trained multimodal target detector is used for target detection applications; otherwise, the above steps are repeated to train the multimodal target detector using training data.

[0098] This completes the training of the object detector using the labeled training data. This embodiment achieves precise matching of visual and textual feature vectors by generating fused feature vectors using a fusion component based on labeled target images and corresponding target text during training. This adjusts the model weights of the multimodal language model, effectively improving the accuracy of object detection and enhancing its robustness and generalization ability.

[0099] Corresponding to the training methods of the aforementioned multimodal target detectors, such as Figure 3 As shown, this application also provides a training apparatus for a multimodal object detector, the multimodal object detector including a data preprocessing model, a fusion component, and a multimodal language model, the training apparatus including:

[0100] A training set acquisition unit is used to acquire a data training set, which includes multiple training data, each training data including a labeled target image and target text corresponding to the target image;

[0101] The training unit is used to train the multimodal object detector using training data from the training set, and to optimize the multimodal language model based on the detection results output by the multimodal object detector until a preset model metric is met, including:

[0102] Input training data into the data preprocessing model and obtain visual feature vectors and text feature vectors;

[0103] The training data is input into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text;

[0104] The visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

[0105] The training device in this embodiment generates a fused feature vector using a fusion component based on the labeled target image and the corresponding target text during the training process to achieve accurate matching between visual feature vectors and text feature vectors. This adjusts the model weights of the multimodal language model, effectively improving the accuracy of target detection and enhancing its robustness and generalization ability.

[0106] Based on the above training method for multimodal target detectors, such as Figure 4 and Figure 5 As shown, this application also provides a target detection method using a multimodal target detector trained by the above training method. The multimodal target detector includes a data preprocessing model and a multimodal language model. The detection method includes:

[0107] The image to be detected and the text to be detected are input into the data preprocessing model to obtain visual feature vectors and text feature vectors;

[0108] The visual feature vector and the text feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate detection results.

[0109] In this embodiment, as Figure 5 The diagram shows the structure of the multimodal object detector during the training process, including a data preprocessing model and a multimodal language model. In this embodiment, the multimodal object detector trained by the above training method, i.e., the multimodal object detector with optimized model weights of the multimodal language model, is used for object detection, which can improve the accuracy of object detection and enhance the robustness and generalization ability of object detection.

[0110] To further improve the accuracy of target detection, in an optional embodiment, the detected text is a positive query text.

[0111] This embodiment uses positive question text with a specified and clear category to facilitate the target detector's understanding of the detection text, thereby improving the accuracy of target detection and enhancing its robustness and generalization ability.

[0112] Corresponding to the target detection methods of the multimodal target detectors mentioned above, such as Figure 6 As shown, this application also provides a multimodal target detection device, which detects a target image by using a multimodal target detector. The multimodal target detector includes a data preprocessing model and a multimodal language model. The multimodal target detection device includes:

[0113] The data preprocessing unit is used to input the image to be detected and the text to be detected into the data preprocessing model to obtain visual feature vectors and text feature vectors.

[0114] The detection unit is used to concatenate the visual feature vector and the text feature vector to form a concatenated sequence, and input it into the multimodal language model to generate detection results.

[0115] The multimodal target detection device in this embodiment uses a multimodal target detector trained by the above training method, i.e., a multimodal target detector with optimized model weights of the multimodal language model, to perform target detection. This can improve the accuracy of target detection and enhance the robustness and generalization ability of target detection.

[0116] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a multimodal target detector or implements a target detection method.

[0117] In practical applications, the computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0119] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0120] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] like Figure 7 As shown, another embodiment of the present invention provides a structural schematic diagram of a computer device. Figure 7 The computer device T12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0122] like Figure 7 As shown, the computer device T12 is represented in the form of a general-purpose computing device. The components of the computer device T12 may include, but are not limited to: one or more processors or processing units T16, system memory T28, and bus T18 connecting different system components (including system memory T28 and processing unit T16).

[0123] Bus T18 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0124] Computer device T12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device T12, including volatile and non-volatile media, removable and non-removable media.

[0125] System memory T28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) T30 and / or cache memory T32. Computer device T12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system T34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus T18 via one or more data media interfaces. The memory T28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0126] A program / utility T40 having a set (at least one) of program modules T42 can be stored, for example, in a memory T28. Such program modules T42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules T42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0127] Computer device T12 can also communicate with one or more external devices T14 (e.g., keyboard, pointing device, display T24, etc.), and with one or more devices that enable a user to interact with computer device T12, and / or with any device that enables computer device T12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface T22. Furthermore, computer device T12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter T20. Figure 7 As shown, network adapter T20 communicates with other modules of computer device T12 via bus T18. It should be understood that, although... Figure 7 As not shown, it can be used in conjunction with computer device T12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0128] The processor unit T16 executes various functional applications and data processing by running programs stored in the system memory T28, such as implementing a training method for a multimodal target detector or a target detection method provided in the embodiments of the present invention.

[0129] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A training method for a multimodal target detector, characterized in that, The multimodal object detector includes a data preprocessing model, a fusion component, and a multimodal language model, and the training method includes: Obtain a data training set, which includes multiple training data sets, each of which includes a labeled target image and target text corresponding to the target image; The multimodal target detector is trained using the training data from the aforementioned training set to obtain detection results. Based on these detection results, the multimodal language model is optimized until it meets preset model metrics. The step of training the multimodal target detector using each training data from the data training set to obtain detection results includes: One training data point from multiple training data sets is input into the data preprocessing model to obtain visual feature vectors and text feature vectors; The training data is input into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text; The visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

2. The training method according to claim 1, characterized in that, The step of inputting the training data into the fusion component and obtaining the fusion feature vector further includes: The multimodal target detector includes a data preprocessing model. The target image is processed by the data preprocessing model to obtain target detection results, and the target detection results include multiple detection boxes. Alignment operation is performed on each detection box of the target image to obtain at least one instance of the target image, instance features and label information corresponding to each instance; The target text is searched to obtain category information. The category information and the tag information of each instance are matched to obtain at least one category instance corresponding to the category information. A first fused feature vector is generated by superimposing and averaging the instances of each category, and the first fused feature vector is used as the fused feature vector.

3. The training method according to claim 2, characterized in that, The label information includes the coordinates of the detection box position and the category name. The step of searching the target text to obtain category information, and matching the category information with the label information of each instance to obtain at least one category instance corresponding to the category information, further includes: Instances whose category names are the same as those in the label information are used as category instances, where the category name is the annotation information of the labeled target image.

4. The training method according to claim 2, characterized in that, The step of generating the first fused feature vector by superimposing and averaging instances of each category further includes: The instance features of the instances of each category are superimposed to generate an instance feature set; The first fused feature vector is generated by performing an arithmetic mean operation on the set of instance features.

5. The training method according to claim 2, characterized in that, After performing the averaging operation on instances of each category to generate the first fused feature vector, the training method further includes: The text feature vector and the first fused feature vector are superimposed and averaged to generate a second fused feature vector, and the second fused feature vector is used as the fused feature vector.

6. The training method according to claim 5, characterized in that, The step of generating a second fused feature vector by superimposing and averaging the text feature vector and the first fused feature vector further includes: The text feature vector and the first fused feature vector are superimposed to generate a feature vector set; The second fused feature vector is generated by performing an arithmetic mean operation on the set of feature vectors.

7. The training method according to claim 1, characterized in that, Optimizing the multimodal language model further includes: Adjust the model weights of the multimodal language model.

8. The training method according to any one of claims 1-7, characterized in that, The target text is a positive question text.

9. A target detection method using a multimodal target detector trained by any one of claims 1-8, characterized in that, The multimodal target detector includes a data preprocessing model and a multimodal language model, and the detection method includes: The image to be detected and the text to be detected are input into the data preprocessing model to obtain visual feature vectors and text feature vectors; The visual feature vector and the text feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate detection results.

10. The target detection method according to claim 9, characterized in that, The detection text is a positive question text.

11. A training device for a multimodal target detector, characterized in that, The multimodal target detector includes a data preprocessing model, a fusion component, and a multimodal language model; the training device includes: A training set acquisition unit is used to acquire a data training set, which includes multiple training data, each training data including a labeled target image and target text corresponding to the target image; The training unit is used to train the multimodal object detector using training data from the training set, and to optimize the multimodal language model based on the detection results output by the multimodal object detector until a preset model metric is met, including: Input training data into the data preprocessing model and obtain visual feature vectors and text feature vectors; The training data is input into the fusion component to obtain a fusion feature vector, which includes instance features of each instance of the target image corresponding to the target text; The visual feature vector, text feature vector, and fused feature vector are concatenated to form a concatenated sequence, which is then input into the multimodal language model to generate the detection result.

12. A multimodal target detection device, characterized in that, The multimodal target detection device includes a data preprocessing model and a multimodal language model, and detects target images by using a multimodal target detector. The data preprocessing unit is used to input the image to be detected and the text to be detected into the data preprocessing model to obtain visual feature vectors and text feature vectors. The detection unit is used to concatenate the visual feature vector and the text feature vector to form a concatenated sequence, and input it into the multimodal language model to generate detection results.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8; or When the program is executed by the processor, it implements the method as described in any one of claims 9-10.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8; or When the processor executes the program, it implements the method as described in any one of claims 9-10.

Images