Target detection method, device, electronic device and medium based on multimodal prompting

By using unlabeled images and multimodal data to construct pseudo-labels for unsupervised training in the object detection model, and combining image features and text description information for detection, the dependence on massive annotated data and single prompt limitations in the prior art is solved. The problem improves the accuracy and reliability of target detection.

CN118628846BActive Publication Date: 2024-11-08HEFEI ZHONGKE LEINAO INTELLIGENCE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411111045.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The existing object detection model relies on massive annotated data, limiting its capability limit and only supports a single information as a prompt word, limiting the accuracy and reliability of the detection.

Method used

Using pseudo-labels built on unlabeled sample images, their multimodal data and preset features, training the object detection model through unsupervised training, and using multimodal prompts (image features and text description information) for object detection .

Benefits of technology

Reduce dependence on labeled data, reduce data collection costs, improve the generalization ability of the model and the accuracy of object detection, and better handle object detection tasks in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628846B_ABST
    Figure CN118628846B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device, electronic device and medium for target detection based on multimodal prompts, which are related to the field of image processing technology. The method comprises: obtaining an image to be detected and a detection instruction of a target to be detected; the detection instruction comprises a first image feature and / or a first text description information of the target to be detected; the image to be detected and the detection instruction are input into a pre-trained target detection model for target detection, and a detection result of the target to be detected in the image to be detected is obtained; wherein the target detection model is a pseudo-label constructed based on an unlabeled sample image and its multimodal data and preset features, and is obtained by training in an unsupervised training manner. The method can solve the limitations of the capability, accuracy and reliability of the target detection pre-training model, and improve the accuracy and reliability of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, device, electronic device and medium based on multimodal prompting. Background Art

[0002] Object detection is a core algorithm in the computer field. The training of high-precision object detection models requires the use of annotation boxes to label objects in a large number of images, which obviously greatly increases the time and manpower costs of developing high-precision object detection models.

[0003] With the sudden and rapid development of big models and pre-training technology this year, a new development paradigm has emerged, which is to pre-train basic models based on massive data, and then fine-tune them with a small amount of downstream task data. However, the target detection big model pre-training algorithm in related technologies still strongly relies on massive amounts of high-quality annotated data, which limits the upper limit of the target detection pre-training model. In addition, the target detection pre-training model in related technologies only supports single information as a prompt word for detection, which limits the accuracy and reliability of target detection. Summary of the invention

[0004] Based on this, it is necessary to provide a target detection method, device, electronic device and medium based on multimodal prompts that can improve detection capabilities in response to the above technical problems.

[0005] A target detection method based on multimodal prompts, comprising:

[0006] Acquire an image to be detected and a detection instruction of a target to be detected; the detection instruction includes a first image feature and / or a first text description information of the target to be detected;

[0007] The image to be detected and the detection instruction are input into a pre-trained target detection large model for target detection, and a detection result of the target to be detected in the image to be detected is obtained; wherein the target detection large model is a pseudo-label constructed based on unlabeled sample images, their multimodal data and preset features, and is trained using an unsupervised training method.

[0008] In the above scheme, the number of sample images is multiple, and the training process of the target detection large model includes:

[0009] Build a large model for target detection;

[0010] For each sample image, target segmentation is performed on the sample image to obtain a segmented image, and second text description information and a second image feature of each segmented image are determined;

[0011] Randomly selecting one of the sample images as the image to be detected, and selecting one or more from the segmented images of the image to be detected as the targets to be detected;

[0012] Determining a detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected;

[0013] Determine positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature;

[0014] The pseudo labels are constructed by using the image to be detected, the detection instruction, the positive sample and the irrelevant sample, and the target detection large model is trained based on the pseudo labels.

[0015] In the above solution, the training of the target detection large model based on the pseudo-label includes:

[0016] Inputting the image to be detected and the detection instruction into the target detection large model to obtain a prediction result;

[0017] Performing Hungarian matching on the predicted result, the positive sample and the irrelevant sample respectively to obtain a matching result;

[0018] Calculate the loss function value according to the matching result;

[0019] The gradient of the loss function value is returned, and the next sample image is randomly selected until the target detection model reaches the convergence condition.

[0020] In the above solution, determining the detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected includes:

[0021] Calculating a text instruction feature corresponding to the second text description information of the target to be detected, and determining the second image feature as an image instruction feature;

[0022] A detection instruction of the target to be detected is determined according to the text instruction feature and / or the image instruction feature.

[0023] In the above solution, determining the positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature includes:

[0024] Determining a similarity value between the target to be detected and all the segmented images according to the second image feature corresponding to the target to be detected and the second image features corresponding to all the segmented images;

[0025] If the similarity value is greater than a first threshold, determining the corresponding segmented image as the positive sample;

[0026] If the similarity value is between a first threshold and a second threshold, the corresponding segmented image is determined as the irrelevant sample; wherein the first threshold is greater than the second threshold.

[0027] In the above solution, the calculation of the loss function value according to the matching result includes:

[0028] If the matching result indicates that the detection result matches the positive sample, calculating a classification loss function and a detection box coordinate loss function;

[0029] If the matching result indicates that the detection result matches the irrelevant sample, the loss function value is not calculated;

[0030] If the matching result indicates that the detection result does not match the positive sample and the irrelevant sample, a classification loss function is calculated.

[0031] In the above scheme, the construction of the target detection large model includes:

[0032] Calculating a detection instruction for the detection target according to the second text description information and the second image feature of the target to be detected;

[0033] Extracting a third image feature of the image to be detected;

[0034] Performing feature fusion on the third image feature and the detection instruction;

[0035] The fused features are input into the decoder, and the prediction result about the image to be detected is output.

[0036] A target detection device based on multimodal prompting, comprising:

[0037] An acquisition module, used to acquire an image to be detected and a detection instruction of a target to be detected; the detection instruction includes a first image feature and / or a first text description information of the target to be detected;

[0038] A detection module is used to input the image to be detected and the detection instruction into a pre-trained target detection model to perform target detection, and obtain a detection result of the target to be detected in the image to be detected; wherein the target detection model is a pseudo-label constructed based on unlabeled sample images and their multimodal data and preset features, and is trained using an unsupervised training method.

[0039] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned target detection method based on multimodal prompts when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned target detection method based on multimodal prompts when executed by a processor.

[0041] The above-mentioned target detection method, device, electronic device and medium based on multimodal cues use pseudo labels constructed based on unlabeled sample images and their multimodal data and preset features, and adopt an unsupervised training method to train the obtained target detection large model, perform target detection on the image to be detected and the detection instruction, and obtain the detection result of the target to be detected, wherein the detection instruction includes the image features and / or text description information of the image to be detected, so that multimodal information can be used as the detection instruction, enriching the input data of the target detection large model, so that the target detection large model can better understand and process the target detection tasks in complex scenes, and improve the accuracy and reliability of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a target detection method based on multimodal prompts in one embodiment;

[0043] Figure 2 A schematic diagram of the training process of a large target detection model in one embodiment;

[0044] Figure 3 is a schematic diagram of a processing flow of a sample image in one embodiment;

[0045] Figure 4 A framework diagram of model training in one embodiment;

[0046] Figure 5 A framework diagram of model training in yet another embodiment;

[0047] Figure 6 A flowchart of building a large target detection model in one embodiment;

[0048] Figure 7 4 is a structural block diagram of a target detection device based on multimodal prompts in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The implementation details of the technical solution of the embodiment of the present application are described in detail below.

[0051] In one embodiment, Figure 1 As shown, a flow chart of a target detection method based on multimodal prompts is provided, and the target detection method based on multimodal prompts may include the following steps:

[0052] Step S101, obtaining an image to be detected and a detection instruction of a target to be detected.

[0053] An image to be detected is obtained, which may be collected by different sensors or devices. After the image to be detected is obtained, the image to be detected is processed to obtain the detection instruction of the target to be detected.

[0054] The detection instruction is multimodal information used to describe the target to be detected, including the first image feature and / or the first text description information of the target to be detected. The detection instruction can provide richer and more comprehensive information, which helps to improve the accuracy and efficiency of target detection.

[0055] It should be noted that the target detection large model used in this embodiment supports multimodal prompts, that is, it can accept prompts or guidance in various forms. Based on this, the target detection large model supports input detection instructions that contain both image features and text description information. Of course, it can also support input detection instructions that only contain image features or text description information.

[0056] The first text description information is the relevant information of the target to be detected described in text. In practical applications, the first text description information here can be determined according to the target to be detected. For example, if it is necessary to detect a football in an image, the corresponding first text description information can be "football". More features about the target to be detected can be further limited, such as size, color, etc. In one case, the example image can be processed using a large image description model to obtain the first text description information of the target to be detected. Among them, the large image description model refers to a large deep learning model that can understand the content of the image and generate a natural language description from the image, so that the first text description information can accurately describe the scene, objects and relationships in the example image. Among them, the example image can be an image of the target to be detected. For example, assuming that the processing task is to identify the football in the image to be detected, the football image can be used as an example image.

[0057] The first image feature is an image feature about the target to be detected. In practical applications, a large image feature extraction model can be used to process the sample image. The large image feature extraction model has been pre-trained on a large scale and can extract rich semantic features from the sample image to generate the first image feature. The first image feature usually captures the voice information and visual features of the target to be detected, which helps to identify the target to be detected.

[0058] Step S102, inputting the image to be detected and the detection instruction into a pre-trained target detection model to perform target detection, and obtaining the detection result of the target to be detected in the image to be detected.

[0059] Here, the image to be detected and the detection instructions are input into a pre-trained target detection model for target detection. The target detection model can more comprehensively understand the features and context of the target to be detected by combining the detection instructions, thereby obtaining accurate detection results.

[0060] In actual applications, if the detection instruction includes a first image feature, the target detection model supports using the first image feature as the detection instruction to find the position of the target to be detected represented by the first image feature in the image to be detected; if the detection instruction includes the first text description information, the target detection model supports using the first text description information as the detection instruction to find the position of the target to be detected represented by the first text description information in the detection image; if the detection instruction includes the first image feature and the first text description information, the target detection model supports using the combination of "first image feature + first text description information" as a multimodal description to find the position on the image to be detected that is consistent with the described target to be detected.

[0061] It should be noted that in order to reduce the workload of image annotation and improve the training efficiency of the target detection large model, in this embodiment, an unsupervised training method is used to train the target detection large model, so that there is no need to annotate a large amount of sample data. In the training process, the training is based on the unlabeled sample images and their multimodal data and pseudo-labels constructed with preset features, so that the trained target detection large model can support target detection using multimodal cues.

[0062] The following is a detailed description of the training process of the large target detection model.

[0063] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of the training process of the target detection large model is shown.

[0064] Step S201, constructing a large target detection model.

[0065] Here, a target detection large model structure that supports multimodal prompts is constructed. The target detection large model structure can support the position of the target to be detected in the image to be detected with text description information, image features and "text description information + image features", so that a more powerful and accurate target detection large model can be built, and the performance and generalization ability of the model can be improved, so that it can better cope with complex target detection tasks.

[0066] Step S202: for each sample image, perform target segmentation on the sample image to obtain segmented images, and determine second text description information and second image features of each segmented image.

[0067] like Figure 3 As shown, Figure 3 A schematic diagram of the processing flow of sample images is shown. By processing the sample images, multimodal data and feature libraries for training large target detection models are constructed.

[0068] First, a large number of unlabeled sample images are collected, and the large segmentation model is used to segment the sample images to obtain segmented images. In the process of target segmentation, the large segmentation model will try to identify and segment all possible images in the sample images instead of segmenting targets of specific categories. This means that the large segmentation model does not need to know in advance which specific categories of targets are in the sample images, but tries to segment the sample images into independent regions.

[0069] The specific steps of determining the segmented image may be:

[0070] (1) After the large segmentation model processes the sample image, it will obtain the segmentation result of the sample image, which can be converted into the corresponding segmentation mask. The segmentation mask is usually used to represent the boundaries and contours of different regions or objects in the image. Each pixel is assigned a label to indicate which region or object the pixel belongs to.

[0071] (2) Filtering extremely small and irregular segmentation masks can improve the quality of the segmentation results, making them more consistent with the shape and structure of the actual object and obtaining a high-quality segmentation mask map.

[0072] (3) For each segmentation mask in the segmentation mask map, calculate a minimum rectangular box that can enclose the mask. This rectangular box is usually called an outer rectangular box, and its boundary is tangent to or contains the boundary of the object or region enclosed by the mask.

[0073] (3) Use the calculated bounding rectangle to crop the sample image to obtain a segmented image, where each segmented image corresponds to a bounding rectangle.

[0074] Next, the image description model is used to generate second text description information for each segmented image, so that the second text description information corresponding to all segmented images obtained by segmenting all sample images constitutes a multimodal database.

[0075] Then, the large image feature extraction model is used to calculate the second image features for each segmented image, so that the second image features corresponding to all segmented images obtained by segmenting all sample images constitute a feature library.

[0076] Step S203 : randomly select a sample image as the image to be detected, and select one or more from the segmented images of the image to be detected as the objects to be detected.

[0077] Here, one is randomly selected from a plurality of sample images as the image to be detected, wherein the image to be detected includes a plurality of segmented images, and one or more are selected from all the segmented images of the image to be detected as the target to be detected.

[0078] Step S204: determining a detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected.

[0079] Here, the second text description information and the second image feature of the target to be detected are determined in the process of processing the sample image, and the detection instruction of the target to be detected is calculated using the second text description information and the second image feature corresponding to the target to be detected.

[0080] In practical applications, detection instructions can be divided into text instruction features, image instruction features and multi-modal instruction features. The following describes the determination of different detection instructions.

[0081] Among them, for the text instruction features, a pre-trained text encoding model can be used to process the second text description information, and the text description can be encoded into a vector representation of a fixed length, which represents the voice information and features of the second text description, thereby obtaining the text instruction features.

[0082] For the image instruction feature, since the second image feature is represented by a fixed-length vector, the second image feature can be determined as the image instruction feature.

[0083] For multimodal instruction features, on the basis of determining text instruction features and image instruction features, the text instruction features and image instruction features are combined, and the multimodal instruction features can be obtained by simply adding them or fusing them in other ways. The multimodal instruction feature integrates the detection instructions of text description and image information, and contains both text description and image feature information, which can be used for subsequent multimodal tasks or applications.

[0084] Step S205 , determining positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature.

[0085] In the process of model training, positive samples and irrelevant samples help optimize the model training process, improve detection performance and accuracy, and reduce the false detection rate, thereby achieving better target detection results. Therefore, determining positive samples and irrelevant samples is an important step in the target detection task. Among them, positive samples refer to the target objects or target areas contained in the image to be detected, and irrelevant samples refer to objects or areas in the image to be detected that are irrelevant to the target. Irrelevant samples and positive samples can be used to guide the model to learn target detection tasks during the training process.

[0086] In this embodiment, all segmented images in the image to be detected contain positive samples and irrelevant samples of the target to be detected, wherein the positive samples and irrelevant samples of the target to be detected can be screened by the second image features of the target to be detected, thereby determining the positive samples and irrelevant samples of the target to be detected in other segmented images of the image to be detected.

[0087] In one embodiment, the positive samples and irrelevant samples of the target to be detected may be determined by feature similarity.

[0088] First, determine the similarity value between the second image feature corresponding to the target to be detected and the second image features corresponding to all segmented images in the image to be detected, wherein the similarity value can be obtained by calculating the cosine distance between the two second image features, or calculated using other similarity measurement methods, so as to measure the content similarity between the two images.

[0089] In practical applications, in the process of processing unlabeled sample images, the second image features corresponding to each segmented image in the sample image will be determined. After the second image features corresponding to all segmented images in all sample images are determined, the similarity values ​​between the second image features corresponding to any two segmented images can be calculated to generate a similarity matrix. Therefore, in the process of determining the positive samples and irrelevant samples of the target to be detected, the similarity values ​​are searched in the image similarity matrix, and threshold screening is performed to obtain positive samples and irrelevant samples.

[0090] Next, according to the size of the similarity value, positive samples and irrelevant samples are screened from all the segmented images in the image to be detected. Among them, if the similarity value is greater than the first threshold, the corresponding segmented image is determined as a positive sample of the target to be detected, indicating that the segmented image has a high similarity with the target to be detected and may be an object of the same type; if the similarity value is between the first threshold and the second threshold, the corresponding segmented image is determined as an irrelevant sample of the target to be detected, indicating that the segmented image has a moderate similarity with the target to be detected and may be an object of a different type. Of course, if the similarity value is less than the second threshold, where the second threshold is less than the first threshold, the corresponding segmented image is a negative sample of the target to be detected.

[0091] Step S206, construct pseudo labels for the image to be detected, the detection instruction, the positive sample and the irrelevant sample, and train the target detection model based on the pseudo labels.

[0092] Pseudo labeling is a commonly used technique in unsupervised learning. In the absence of real labels, pseudo labels are generated and used as part of the training data to continue training the model. In this way, the labels generated by the model itself can be used to increase the training data and help the model better learn the distribution and characteristics of the data.

[0093] Here, the pseudo-labels constructed by the image to be detected, the detection instructions, the positive samples and the irrelevant samples are used to guide the model to learn the target detection task during the training process and train the target detection model. During the model training process, the image to be detected in the pseudo-label contains the image of the target to be detected. The target detection model needs to perform target detection on the image to be detected, that is, identify the target object or target area in the image to be detected. The detection instructions in the pseudo-label refer to the prompts used to guide the target detection model to perform the target detection task, which can be text descriptions, image features, etc. The detection instructions can help the target detection model better understand the task requirements of target detection. The positive samples in the pseudo-label refer to the target objects or target areas contained in the image to be detected, that is, the targets that the target detection model needs to correctly detect and identify, and are used to train the target detection task of the model. The irrelevant samples in the pseudo-label refer to objects or areas in the image to be detected that are irrelevant to the target, that is, the model should not identify them as targets, and are used to help the model learn how to distinguish between targets and non-targets.

[0094] In the process of model training, pseudo labels can be automatically extracted based on a large number of unlabeled images, thereby reducing the dependence on labeled data, reducing the cost of data collection, and improving the generalization ability of the model. The target detection large model that supports text descriptions and sample images as instructions is pre-trained through pseudo labels, so that the model can better understand and handle target detection tasks in complex scenes. In addition, the training method provided by the present embodiment has extremely strong scalability, supports the automatic generation of target detection pseudo labels for any image in any field, and uses this to train a powerful and scalable target detection basic large model, which can provide a basis for training a high-precision target detection model with heavy downstream tasks. In addition, in the process of model training, it is based on the pre-trained model, and through fine-tuning on any downstream detection task data set, the detection performance of the model in a specific task can be further improved, so that it has better performance in practical applications.

[0095] In one embodiment, reference Figure 4 As shown, Figure 4 The following is a framework diagram of model training. Figure 4 Detailed explanation of how to train large object detection models with pseudo labels.

[0096] First, the image to be detected and the detection instructions in the pseudo-label are input into the target detection model, and the detection results are output. Among them, the image to be detected and the detection instructions are information from different modalities. Inputting them into the target detection model at the same time enables the model to process multiple types of data such as images and text at the same time, which helps the model to understand the target detection task more comprehensively and improve the accuracy of detection. In the model, the image to be detected and the detection instructions are usually fused and interacted to better guide the model to perform target detection and finally output the corresponding prediction results.

[0097] Next, the prediction results are Hungarian matched with the positive samples and irrelevant samples to obtain matching results, so that the performance and accuracy of the model can be evaluated in the object detection task. Among them, Hungarian matching is an optimal matching algorithm used to optimally match the correspondence between two groups of objects. Hungarian matching of the prediction results with positive samples and irrelevant samples can be achieved by following the following steps:

[0098] (1) Calculate the matching cost matrix. First, it is necessary to calculate the cost matrix between the prediction result and the positive sample and the irrelevant sample. Each element of the cost matrix represents the distance or similarity between the prediction result and the positive sample or the irrelevant sample. Various distance measurement methods, such as Euclidean distance, can be used to calculate the cost matrix.

[0099] (2) Apply the Hungarian algorithm. The Hungarian algorithm is used to solve the optimal matching problem. The Hungarian algorithm can find the optimal matching result based on the cost matrix to minimize the total cost.

[0100] (3) Obtaining matching results. Based on the output of the Hungarian algorithm, the best matching relationship between each prediction result and the positive sample or irrelevant sample can be obtained. These matching relationships include the matching object and the matching distance.

[0101] In the above description, the positive samples and irrelevant samples that match the detection results are determined first, and then Hungarian matching is performed. Figure 5 As shown, Figure 5 Another model training framework diagram is shown. Figure 5 After the detection result is Hungarian matched with other segmented images, it is determined whether the segmented image matched by the detection result belongs to a positive sample or an irrelevant sample based on the similarity.

[0102] Then, the loss function value is calculated based on the matching results. Among them, the model's prediction results are matched with positive samples to perform Hungarian matching, which can evaluate the model's performance in detecting target objects; the model's prediction results are matched with irrelevant samples to perform Hungarian matching, which can evaluate the model's false detection rate.

[0103] Finally, the calculated loss function value is back-propagated through the gradient. Back-propagation of gradients is a key step in the optimization algorithm, which is used to update the model parameters to minimize the loss function. The general steps of gradient back-propagation are:

[0104] (1) Calculate the gradient of the loss function with respect to the model parameters. Based on the calculation results of the loss function and the model parameters, the chain rule is used to calculate the gradient of the loss function with respect to the model parameters. These gradients represent the rate of change of the loss function under the current parameter values.

[0105] (2) Gradient back propagation. The calculated gradient is propagated from the output layer to the input layer to update the parameters of each layer. In this process, an optimization algorithm (such as stochastic gradient descent) can be used to update the parameters so that the loss function gradually decreases.

[0106] (3) Parameter update. Based on the gradient calculated by the gradient back-propagation process, the optimization algorithm is used to update the model parameters. The parameter update method is usually to update the parameter value in the opposite direction of the gradient according to a certain step size (learning rate).

[0107] After completing the gradient backpropagation, continue to randomly select the next sample image as the image to be detected, generate a new pseudo label corresponding to the image to be detected, and carry out the next round of model training until the target detection model reaches the convergence conditions (such as reaching the maximum number of iterations, loss function convergence, etc.), then the entire model training of the complete target detection model is completed.

[0108] In one embodiment, the calculation of the loss function value is related to the matching result, and the calculation method of the loss function value under different matching results is different. The matching results can be divided into: the detection result matches the positive sample, the detection result matches the irrelevant sample, and the matching fails, where the matching failure means that the detection result neither matches the irrelevant sample nor the positive sample, that is, the model fails to correctly classify the detection box as any target object.

[0109] For the detection results that match the positive samples, the classification loss function and the detection box coordinate loss function are calculated. The focal loss function can be used to calculate the classification loss, which can help solve the problem of category imbalance and improve the model's attention to difficult samples. The L1 distance or generalized intersection over union distance (GIoU) ​​is used to calculate the detection box coordinate loss, which can help the model learn the correct target position.

[0110] For detection results that match irrelevant samples, there is no need to calculate the loss function, because irrelevant samples are not the target of the model.

[0111] For detection results that fail to match, it is necessary to calculate the classification loss function so that the model can learn how to classify correctly to improve the classification accuracy of the model.

[0112] In one embodiment, Figure 6 As shown, Figure 6 A flowchart for building a large target detection model is shown.

[0113] In this embodiment, for any image to be detected, the constructed target detection model supports finding the positions of all targets to be detected in the image to be detected using text descriptions of the target to be detected, sample images, or a combination thereof as instructions. The specific construction process of the target detection model is as follows:

[0114] Step 1: Calculate the detection instruction for the multimodal information of the target to be detected. The target to be detected here may refer to any segmented image in the image to be detected. The detection instruction may include text instruction features, image instruction features and multimodal instruction features. For the text instruction feature, the second text description information is processed and calculated using the text encoding model; for the image instruction feature, since the second image feature of the target to be detected has been extracted using the image feature extraction model, the second image feature can be determined as the image instruction feature; for the multimodal instruction feature, the sum of the text instruction feature and the image instruction feature is calculated as the multimodal instruction feature.

[0115] Step 2: Use the image encoder large model to extract the third image feature of the image to be detected.

[0116] Step 3: Fuse the third image feature of the image to be detected with the detection instruction of the target to be detected. Here, an image-instruction fuser can be used to achieve feature fusion, and a more informative feature representation is obtained by fusing the image feature and the detection instruction. In the process of feature fusion, different methods can be used, such as splicing, weighted summation, attention mechanism, etc.

[0117] Step 4: Input the fused features into the decoder and output the prediction structure of the model. The decoder is usually a neural network model that converts the input feature representation into the final prediction output.

[0118] Based on this, by comprehensively utilizing multimodal information, feature fusion and decoder prediction, this process can build a large target detection model that supports multimodal cues. This model design can improve the performance of the model when processing multimodal information, enabling it to more comprehensively understand the target object and achieve accurate target detection.

[0119] In the above embodiment, the target detection large model is a pseudo-label constructed based on unlabeled sample images and their multimodal information and preset features, and is trained using an unsupervised learning method, thereby reducing the reliance on labeled data, reducing the cost of data collection, and improving the generalization ability of the model. In the application process of the target detection large model, the detection instructions of the image to be detected can be input into the target detection large model for processing and target detection, and the corresponding detection results can be obtained, so that the model can use multimodal information as instructions, enriching the input information of the target detection large model, and can better understand and process target detection tasks in complex scenes, and improve the accuracy of target detection.

[0120] In one embodiment, a target detection device based on multimodal prompts is provided, referring to Figure 7As shown, the target detection device 700 based on multimodal prompts may include: an acquisition module 701, a prediction module 702 and a training module 703.

[0121] Among them, the acquisition module 701 is used to acquire the image to be detected and the detection instruction of the target to be detected; the detection instruction includes the first image feature and / or the first text description information of the target to be detected; the prediction module 702 is used to input the image to be detected and the detection instruction into a pre-trained target detection large model for target detection, and obtain the detection result of the target to be detected in the image to be detected; wherein the target detection large model is a pseudo-label constructed based on unlabeled sample images and their multimodal data and preset features, and is trained using an unsupervised training method.

[0122] In one embodiment, there are multiple sample images, and the training module 703 is used to: construct a large model for target detection; for each sample image, perform target segmentation on the sample image to obtain a segmented image, and determine the second text description information and the second image feature of each segmented image; randomly select a sample image as the image to be detected, and select one or more from the segmented images of the image to be detected as the targets to be detected; determine the detection instructions of the target to be detected according to the second text description information and the second image feature of the target to be detected; determine the positive samples and irrelevant samples of the target to be detected from all the segmented images of the image to be detected according to the second image feature; construct pseudo labels for the image to be detected, the detection instructions, the positive samples and the irrelevant samples, and train the large model for target detection based on the pseudo labels.

[0123] In one embodiment, the training module 703 is specifically used to: input the image to be detected and the detection instruction into the target detection model to obtain a prediction result; perform Hungarian matching on the prediction result with the positive sample and the irrelevant sample respectively to obtain a matching result; calculate the loss function value according to the matching result; return the gradient of the loss function value, and continue to randomly select the next sample image until the target detection model reaches the convergence condition.

[0124] In one embodiment, the training module 703 is specifically used to: calculate the text instruction feature corresponding to the second text description information of the target to be detected, and determine the second image feature as the image instruction feature; determine the detection instruction of the target to be detected based on the text instruction feature and / or the image instruction feature.

[0125] In one embodiment, the training module 703 is specifically used to: determine the similarity value between the target to be detected and the segmented image excluding the target to be detected based on the second image feature corresponding to the target to be detected and the second image feature corresponding to the segmented image excluding the target to be detected; if the similarity value is greater than a first threshold, the corresponding segmented image is determined as a positive sample; if the similarity value is between the first threshold and the second threshold, the corresponding segmented image is determined as an irrelevant sample; wherein the first threshold is greater than the second threshold.

[0126] In one embodiment, the training module 703 is specifically used to: if the matching result represents that the detection result matches the positive sample, calculate the classification loss function and the detection box coordinate loss function; if the matching result represents that the detection result matches the irrelevant sample, do not calculate the loss function value; if the matching result represents that the detection result does not match both the positive sample and the irrelevant sample, calculate the classification loss function.

[0127] In one embodiment, the training module 703 is specifically used to: calculate the detection instructions of the detection target based on the second text description information and the second image features of the target to be detected; extract the third image features of the image to be detected; fuse the third image features with the detection instructions; input the fused features into the decoder, and output the prediction results about the image to be detected.

[0128] For the specific limitations of the target detection device 700 based on multimodal prompts, please refer to the limitations of the target detection method based on multimodal prompts above, which will not be repeated here. The various modules in the above-mentioned target detection device 700 based on multimodal prompts can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0129] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a target detection method based on multimodal prompts is implemented.

[0130] In one embodiment, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a target detection method based on multimodal prompts is implemented.

[0131] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or in combination with these instruction execution systems, devices or equipment. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or equipment, or in combination with these instruction execution systems, devices or equipment. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0132] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0134] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0135] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A target detection method based on multimodal prompts, characterized in that: The method comprises: Acquire an image to be detected and a detection instruction of a target to be detected; the detection instruction includes a first image feature and / or a first text description information of the target to be detected; The image to be detected and the detection instruction are input into a pre-trained target detection model for target detection, and a detection result of the target to be detected in the image to be detected is obtained; wherein the target detection model is a pseudo-label constructed based on an unlabeled sample image and its multimodal data and preset features, and is trained in an unsupervised training manner; wherein the number of the sample images is multiple, and the training process of the target detection model includes: Build a large model for target detection; For each sample image, target segmentation is performed on the sample image to obtain a segmented image, and second text description information and a second image feature of each segmented image are determined; Randomly selecting one of the sample images as the image to be detected, and selecting one or more from the segmented images of the image to be detected as the targets to be detected; Determining a detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected; Determine positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature; The pseudo labels are constructed by using the image to be detected, the detection instruction, the positive sample and the irrelevant sample, and the target detection large model is trained based on the pseudo labels.

2. The method according to claim 1, characterized in that The training of the target detection large model based on the pseudo-label comprises: Inputting the image to be detected and the detection instruction into the target detection large model to obtain a prediction result; Performing Hungarian matching on the predicted result, the positive sample and the irrelevant sample respectively to obtain a matching result; Calculate the loss function value according to the matching result; The gradient of the loss function value is returned, and the next sample image is randomly selected until the target detection model reaches the convergence condition.

3. The method according to claim 1, characterized in that The step of determining the detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected includes: Calculating a text instruction feature corresponding to the second text description information of the target to be detected, and determining the second image feature as an image instruction feature; A detection instruction of the target to be detected is determined according to the text instruction feature and / or the image instruction feature.

4. The method according to claim 1, characterized in that: The step of determining the positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature comprises: Determining a similarity value between the target to be detected and all the segmented images according to the second image feature corresponding to the target to be detected and the second image features corresponding to all the segmented images; If the similarity value is greater than a first threshold, determining the corresponding segmented image as the positive sample; If the similarity value is between a first threshold and a second threshold, the corresponding segmented image is determined as the irrelevant sample; wherein the first threshold is greater than the second threshold.

5. The method according to claim 2, characterized in that: The calculating the loss function value according to the matching result includes: If the matching result indicates that the detection result matches the positive sample, calculating a classification loss function and a detection box coordinate loss function; If the matching result indicates that the detection result matches the irrelevant sample, the loss function value is not calculated; If the matching result indicates that the detection result does not match the positive sample and the irrelevant sample, a classification loss function is calculated.

6. The method according to claim 1, characterized in that The construction of the target detection large model includes: Calculating a detection instruction for the detection target according to the second text description information and the second image feature of the target to be detected; Extracting a third image feature of the image to be detected; Performing feature fusion on the third image feature and the detection instruction; The fused features are input into the decoder, and the prediction result about the image to be detected is output.

7. A target detection device based on multimodal prompts, characterized in that: The device comprises: An acquisition module, used to acquire an image to be detected and a detection instruction of a target to be detected; the detection instruction includes a first image feature and / or a first text description information of the target to be detected; A detection module is used to input the image to be detected and the detection instruction into a pre-trained target detection model to perform target detection, and obtain a detection result of the target to be detected in the image to be detected; wherein the target detection model is a pseudo-label constructed based on an unlabeled sample image and its multimodal data and preset features, and is trained in an unsupervised training manner; wherein the number of the sample images is multiple, and the training process of the target detection model includes: Build a large model for target detection; For each sample image, target segmentation is performed on the sample image to obtain a segmented image, and second text description information and a second image feature of each segmented image are determined; Randomly selecting one of the sample images as the image to be detected, and selecting one or more from the segmented images of the image to be detected as the targets to be detected; Determining a detection instruction of the target to be detected according to the second text description information and the second image feature of the target to be detected; Determine positive samples and irrelevant samples of the target to be detected from all segmented images of the image to be detected according to the second image feature; The pseudo labels are constructed by using the image to be detected, the detection instruction, the positive sample and the irrelevant sample, and the target detection large model is trained based on the pseudo labels.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the target detection method based on multimodal prompts described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method based on multimodal prompts described in any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Rail surface peeling and spalling small target detection method and system based on p2-pdg-yolo-world

    CN123141814A