Target detection method, electronic equipment, storage medium and program product

Abstract

The invention discloses a target detection method, electronic equipment, a storage medium and a program product, belongs to the technical field of computer languages, and is used for reducing the time delay of target detection of a large model. The method comprises the following steps: preprocessing a to-be-detected image to obtain a mapping feature sequence of the to-be-detected image in a text embedding space; preprocessing category text information of a target object in the to-be-detected image to obtain a text embedding sequence of the category text information in the text embedding space; and determining detection features of the target object according to a preset query vector, the mapping feature sequence and the text embedding sequence.

Description

Technical Field

[0001] The present application belongs to the field of computer language technology, and specifically relates to a target detection method, electronic equipment, storage medium and program product. Background Art

[0002] Multimodal large language models (MLLMs) are cutting-edge innovations in artificial intelligence that combine the power of language and vision models to tackle complex tasks such as visual question answering and image captioning. These models leverage large-scale pre-training and integrate multiple data modalities, significantly improving their performance across a wide range of applications.

[0003] Current large-scale multimodal model object detection solutions generally use a text-image question-answering approach, outputting normalized coordinate values ​​in a self-looping manner. These coordinate values ​​can be directly expressed in natural language (e.g., Shikra, Griffion) or using additional special tokens (e.g., Florence2). Current large-scale multimodal models use an autoregressive approach to generate tokens. Generating a complete conversation requires T iterations, where T is the number of answer tokens. Furthermore, a location coordinate typically needs to be modeled as four or more tokens. Therefore, in existing large-scale multimodal model object detection solutions, detecting a single image often requires generating more than 30 tokens, which translates to more than 30 iterations. Given the large number of parameters and computational complexity inherent in the large-scale multimodal model, inferring a single image requires a long time and significant computational overhead. This results in high detection latency for large-scale multimodal models, making it difficult to achieve the latency required in practical applications. Summary of the Invention

[0004] The embodiments of the present application provide a target detection method, electronic device, storage medium, and program product, which can solve the problem of excessively high detection latency in multimodal large models.

[0005] In a first aspect, an embodiment of the present application provides a target detection method, which includes: preprocessing an image to be tested to obtain a mapping feature sequence of the image to be tested in a text embedding space; preprocessing the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; and determining the detection features of the target object based on a preset query vector, the mapping feature sequence and the text embedding sequence.

[0006] In second aspect, an embodiment of the present application provides a target detection device, which includes: a first processing module for preprocessing the image to be tested to obtain a mapping feature sequence of the image to be tested in a text embedding space; a second processing module for preprocessing the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; a detection module for determining the detection features of the target object based on a preset query vector, the mapping feature sequence and the text embedding sequence.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first aspect.

[0010] In an embodiment of the present application, the image to be tested is preprocessed to obtain a mapping feature sequence of the image to be tested in a text embedding space; the category text information of the target object in the image to be tested is preprocessed to obtain a text embedding sequence of the category text information in the text embedding space; the detection features of the target object are determined based on a preset query vector, the mapping feature sequence and the text embedding sequence. The detection features of the target object can be detected at one time through the query vector, thereby avoiding the problem of excessively high detection delay of a multimodal large model and improving detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of a target detection method provided in an embodiment of the present application; Figure 2 This is a schematic diagram of an architecture for target detection provided by an embodiment of the present application; Figure 3 This is a schematic diagram of another target detection architecture provided by an embodiment of the present application; Figure 4 This is a schematic diagram of another target detection architecture provided by an embodiment of the present application; Figure 5 This is a schematic diagram of another target detection architecture provided by an embodiment of the present application; Figure 6 is a structural diagram of a target detection device provided in an embodiment of the present application; Figure 7 It is a structural diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0014] Some general-purpose multimodal large language models: Shikra has reference conversation capabilities, can understand the location information input by the user, and output accordingly. Shikra can understand the point / bounding box input by the user and supports point / bounding box output, making it suitable for object detection tasks. Shikra's design is simple and straightforward, requiring no additional position encoders, front / back object detectors, external plug-in modules, or even an additional vocabulary. Shikra can accurately understand the location area input by the user and can reference different areas in its output to communicate effectively through conversation and location, just like humans. Shikra can make inferences based on location information, but when directly applied to traditional object detection, Shikra's performance is poor and is often plagued by hallucinations.

[0015] Griffon is a large-scale visual perception model that unifies multiple tasks. Specifically, based on the multimodal large-scale LLaVA model, it unifies localization tasks such as object detection, phrase localization, and referential expression understanding into a large-scale image-text model. Using the LLM's autoregressive unified encoding training and prediction, the LLM achieves fine-grained understanding of object locations and categories in images, strengthening the local perception capabilities of the large-scale image-text model. Its development primarily encompasses four aspects: the construction of a language-cued localization dataset, the design of a unified input-output representation, a progressive training approach, and a training-free confidence assessment method. To avoid introducing special placeholders and specialized structures, Griffon designs a unified input-output representation consistent with image-text tasks. Griffon consists of a visual encoder, CLIP-ViT-L / 14, a connector, and a large language model, Llama2-13B. Because it employs a large language model dialogue format, the model can directly process any form of text input, naturally unifying localization input across different scenarios. From the perspective of output unification, Griffon normalizes coordinates into a fixed 3-bit precision [0-1] encoding representation. A single target is uniformly represented in the form of "category-[x1, y1, x2, y2]". When outputting multiple targets, the targets are connected with &, thus achieving output unification for multiple positioning scenarios without introducing any priors or placeholders.

[0016] Florence-2 is a general-purpose, multimodal, large-scale model designed to address the diverse challenges of computer vision and vision-language tasks. Florence-2 is carefully designed to overcome the limitations of traditional single-task frameworks. It adopts a sequence-to-sequence learning paradigm, integrating various tasks under a common language modeling objective. It accepts images and text instructions as input data and outputs textual results. The input or output text can be in the form of plain text or descriptions of specific regions in the image. Florence-2 is built on the classic encoder-decoder transformer architecture. During task execution, the model flexibly manages image and text processing.

[0017] Existing large-scale, multimodal object detection solutions, represented by Florence2, Griffon, and Shikra, generally employ a text-image question-answering approach, outputting normalized coordinate values ​​in a self-looping fashion. These coordinate values ​​can be directly expressed in natural language (Shikra, Griffion) or using additional special tokens (Florence2). These generative object detection solutions suffer from the following common issues: High inference latency: Large multimodal models use an autoregressive approach to generate tokens. Generating a complete conversation requires T iterations, where T is the number of answer tokens. Furthermore, a location coordinate is typically modeled as four or more tokens. Therefore, existing large multimodal model object detection solutions often require generating more than 30 tokens to detect a single image, requiring more than 30 iterations. Given the large number of parameters and computational complexity inherent in large multimodal models, inference on a single image requires a long time and significant computational overhead. This makes it difficult for large multimodal models to achieve the same latency as traditional detection models and meet the real-time detection requirements (>30 fps) in practical applications.

[0018] No confidence score: Mainstream detection solutions based on large multimodal models only output category names and corresponding coordinates. Compared to traditional object detection models, they lack confidence score output. Some duplicate detection boxes cannot be filtered out using the traditional non-maximum suppression (NMS) method, seriously affecting business performance.

[0019] Poor detection results: Large multimodal models are susceptible to language model hallucinations. Furthermore, the output length of large models is uncertain, resulting in many repeated detections. This makes the model's performance in detection tasks generally lower than that of traditional models.

[0020] In order to solve the above problems, the embodiments of the present application provide a target detection method, an electronic device, a storage medium and a program product. The target detection method provided in the embodiments of the present application is used to Represents the visual encoder, using J represents the mapper, represents a generative language model, then the multimodal language model Indicated as M In the embodiment of the present application, the generative language model can be an encoder-decoder architecture or a decoder-only architecture, which is not limited in the embodiment of the present application. The object detection methods of the two architectures follow the same idea.

[0021] In the embodiment of the present application, there is only one adjusted generative language model, which is represented by , used to fuse visual and text features and output the corresponding location coordinates through a learnable query. The target detection method provided by the application embodiment is to organically combine the traditional Transformer-based target detection method (Detr) with the target detection mechanism of the multimodal large model. The specific implementation is to adjust the generative language model in the multimodal large model and add a Learnable query vectors , used to extract location and category information from visual features and text features, and add a box regression head and a classification head , decode the word element vector output by the Decoder into the final target box position coordinates and confidence.

[0022] The target detection method, electronic device, storage medium and program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0023] Figure 1 An embodiment of the present application provides a target detection method, which can be performed by an electronic device, which may include a server and / or a terminal device. In other words, the method can be performed by software or hardware installed in the electronic device, and the method includes the following steps: Step S102: pre-processing the image to be tested to obtain a mapping feature sequence of the image to be tested in the text embedding space.

[0024] In an embodiment of the present application, the large model can obtain the image to be tested input by the user, and then preprocess the image to be tested to obtain a mapping feature sequence corresponding to the image to be tested in the text embedding space.

[0025] In one implementation, the preprocessing of the image to be tested to obtain a mapping feature sequence of the image to be tested in a text embedding space includes: encoding the image to be tested into a visual feature sequence; and mapping the visual feature sequence to the text embedding space to obtain the mapping feature sequence.

[0026] Specifically, input the image to be tested , the visual encoder VE of the large model can first encode the image to be tested into a length of , the dimension is Visual feature sequence :

[0027] in, is the hidden layer dimension of the visual encoder.

[0028] For visual features , which can be transformed from a large model to a large model through the mapper PJ The visual space is mapped to the dimension The text embedding space is used to obtain the mapping feature sequence of visual features in the text embedding space :

[0029] Step S104: pre-processing the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space.

[0030] In the embodiment of the present application, mainly for the open set target detection task, the category information of the target object to be detected in the image to be detected can be given in the form of text prompts, and the category text information is obtained. . Text information for this category Preprocessing can obtain the text information of this category A sequence of text embeddings in the text embedding space.

[0031] In one implementation, the category text information is mapped to the text embedding space to obtain the text embedding sequence.

[0032] Specifically, the word segmenter TK and embedding layer of the generative language model GL The category text information can be encoded by BPE Mapping to the text embedding space, so that the text embedding sequence can be obtained :

[0033] Step S106: determining the detection features of the target object according to the preset query vector, the mapping feature sequence and the text embedding sequence.

[0034] The embodiment of the present application adjusts the generative language model GL in the multimodal large model and adds Learnable query vectors , which is used to extract the detection features of the target object from the mapping feature sequence and the text embedding sequence. The detection features may include the location information and category information of the target object.

[0035] Learnable query vectors are automatically optimized and adjusted during the training process in machine learning and deep learning models. These vectors play a crucial role in attention mechanisms, particularly in the Transformer model. The query vector, key vector, and value vector are the three fundamental vector representations used to describe input sequences, calculate similarity, and output weighted information, respectively.

[0036] The target detection method provided in the embodiment of the present application obtains a mapping feature sequence of the image to be tested in a text embedding space by preprocessing the image to be tested; preprocesses the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; determines the detection features of the target object based on a preset query vector, the mapping feature sequence and the text embedding sequence, and detects the detection features of the target object at one time by implementing a forward propagation process through the query vector, replacing the multiple-loop inference process based on the autoregressive process of the existing multimodal large model, which can greatly reduce the inference delay, avoid the problem of excessively high detection delay of the multimodal large model, and improve the detection efficiency.

[0037] In one implementation, the detection features of the target object are determined based on a preset query vector, the mapping feature sequence, and the text embedding sequence, including: concatenating the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; using the query vector as a query for cross-attention and the multimodal sequence as a key and value for the cross-attention; and determining the detection features based on the cross-attention, the query for the cross-attention, and the key and value for the cross-attention.

[0038] In the embodiment of the present application, the mapping feature sequence obtained after mapping can be and the encoded text embedding sequence Concatenate into a multimodal sequence:

[0039] Among them, concat represents the concatenation operation of vectors in the sequence dimension. and a learnable query vector Together they serve as input to the generative language model GL to decode the final detection features.

[0040] Since there are two architectures of Encoder-Decoder and Decoder-only in the current multimodal large model, and the forward propagation processes of the two architectures are different, the embodiments of the present application respectively describe the decoding process based on these two architectures.

[0041] The decoding process based on the Encoder-Decoder architecture is as follows: Encoder-Decoder architecture is as follows Figure 2 As shown, the main calculation step is cross attention CA, at this time, the query vector of target detection Can be used as a query for cross attention, and multimodal sequence As the key and value of cross attention, the decoding process can be expressed as , in, That is, the detection features output by the generative language model GLM, which includes position feature information and category feature information.

[0042] In this way, the Transformer-based target detection method (Detr) is organically combined with the target detection mechanism of the multimodal large model, and the generative language model in the multimodal large model is adjusted, and the input of its Decoder structure is added. Learnable query vectors , a single forward propagation process can detect the detection features of the target object at one time, replacing the multiple cyclic reasoning processes based on the autoregressive process of the existing multimodal large model, which can greatly reduce the reasoning delay, avoid the problem of excessively high detection delay of the multimodal large model, and improve the detection efficiency.

[0043] In one implementation, determining the detection features of the target object based on a preset query vector, the mapping feature sequence, and the text embedding sequence includes: concatenating the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; concatenating the multimodal sequence and the query vector to obtain an input sequence; using the input sequence as a query, key, and value of self-attention; and determining the detection features based on the self-attention, the query, key, and value of the self-attention.

[0044] The decoding process based on the decoder-only architecture also needs to first concatenate the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence.

[0045] The mapping feature sequence obtained after mapping and the encoded text embedding sequence Concatenate into a multimodal sequence:

[0046] Among them, concat represents the concatenation operation of vectors in the sequence dimension. and a learnable query vector Together they serve as input to the generative language model GL to decode the final detection features.

[0047] Decoder-only architecture Figure 3 As shown, the main calculation step is self-attention SA, so we multimodal sequence and query vector Splicing to get the input sequence :

[0048] The input sequence is used as the query, key, and value of the self-attention to obtain the detection feature output:

[0049] In this way, the Transformer-based target detection method (Detr) is organically combined with the target detection mechanism of the multimodal large model, and the generative language model in the multimodal large model is adjusted, and the input of its Decoder structure is added. Learnable query vectors , a single forward propagation process can detect the detection features of the target object at one time, replacing the multiple cyclic reasoning processes based on the autoregressive process of the existing multimodal large model, which can greatly reduce the reasoning delay, avoid the problem of excessively high detection delay of the multimodal large model, and improve the detection efficiency.

[0050] In one implementation, the detection features include position feature information and category feature information. After determining the detection features of the target object, it also includes: processing the detection features through a classification head, and outputting the matching confidence between the category feature information in each detection feature and the category text information.

[0051] In the embodiment of the present application, a classification header can also be used , decode the detection features output by the Decoder into the final matching confidence. Specifically, for the output length of The sequence of detection features , for each of the detection features , first use the classification head to get the classification probability:

[0052] The embodiment of this application is aimed at the open set target detection task, so the purpose of classification is to determine whether the category to which the detection feature belongs matches the given category text information. Therefore, the It is a binary probability, 0 represents no match and 1 represents a match.

[0053] The classification head is used to output the binary classification probability of the category. Compared with the large model that only outputs the location coordinates and category name, the probability can be used as the confidence level to filter out low-confidence outputs or use NMS technology to further improve the detection accuracy. At the same time, the classification head limits the category space of the model output, which can effectively reduce the occurrence of hallucinations.

[0054] In one implementation, after outputting the matching confidence between the category feature information and the category text information in each of the detection features, it also includes: obtaining a target detection feature whose matching confidence meets a preset condition from the detection feature; processing the target detection feature through a regression head, and outputting the position coordinates according to the position feature information of the target detection feature.

[0055] In an embodiment of the present application, the preset condition may be that the binary classification probability is 1. All target detection features with a classification result of 1 may be obtained from the detection features, and then the target detection features may be processed using a regression head. Position coordinates may be output based on the position feature information of the target detection features. The position coordinates may be four-dimensional box coordinates, and the regression head may be a box regression head. The computing device in the embodiment of the present application may be a GPU, a CPU, an NPU, etc., and the embodiment of the present application does not specifically limit this.

[0056] The analytical method is introduced into the multimodal large-model target detection solution. The detection features are extracted using a learnable query vector, and the regression head and classification head are used to obtain the detection results, replacing the original natural language output. The regression head is used to regress the position coordinates in one go. Compared with using a large model to autoregressively output the position coordinates, this can greatly reduce the number of inferences and significantly save inference latency, making it more conducive to the application of large models in actual business.

[0057] The target inspection method provided by this application is specifically described below through two specific examples: Example 1: Encoder-Decoder architecture detection model based on Florence like Figure 4 The figure shows the implementation of the Encoder-Decoder architecture of the embodiment of the present application. The backbone model of the architecture is based on the open source Florence2 model. This model is pre-trained with a total of 5B of graphic data. Therefore, it originally has target detection capabilities, but uses natural language format to output the detection frame of the model.

[0058] In the embodiment of this application, we keep the main structure of Florence2 unchanged, and the visual encoder still uses its pre-trained DaViT model. On this basis, we add a Learnable query vectors In this embodiment, N=20. And replace the LM Head for text generation with the classification head and frame return header is used to obtain the final classification information and location box coordinates from the decoder output.

[0059] Because the Florence2 backbone already has object detection capabilities, only a small amount of data is needed to train the newly added structure to achieve the same effect by changing the model input method. Taking Florence2-base-ft as an example, the original mAP indicator on the COCO dataset was 41.4. After adopting the technical solution of this embodiment, it can reach 42.8 with simple fine-tuning, surpassing the original indicator.

[0060] At the same time, this embodiment can significantly improve the inference efficiency of multimodal language models. Based on the inference speed of Florence2's autoregressive process of 0.06 seconds per token, and considering that each image in the COCO dataset has an average of seven objects, the decoding process for predicting all objects in an image takes approximately 0.42 seconds. However, this embodiment only requires the decoder to perform inference once, taking an average of only 0.06 seconds, which is only 1 / 7 the time of the autoregressive method. This significantly alleviates the bottleneck of slow inference speed in large multimodal models.

[0061] Example 2: Decoder-only architecture detection model based on GPT.

[0062] like Figure 5 The example above illustrates the decoder-only architecture implementation described in this example. The model's backbone is the same as that of Shikra and Griffon, both of which employ the classic visual encoder-mapping layer-decoder-only large language model architecture proposed by LLaVA. DaViT from Florence2 is used as the visual encoder (VE) to provide a broad visual representation, and GPT2 is used as the generative language model (GL). Two fully connected layers serve as mappers (PJs) between these two structures to connect the visual encoder and the generative language model.

[0063] Referring to the training method of LLaVA, we first use image and text data to align the visual and textual modalities. At this time, we still use the language modeling head that comes with GPT2 to output the results, and use the standard text modeling loss to constrain the training of the model. During this stage, the visual encoder VE remains frozen. In the second stage, we fine-tune the detection ability of the model and replace the language modeling head with the classification head. and frame return header and fine-tune all parameters of the model using object detection data.

[0064] Under the premise of fine-tuning with the same data, compared with directly outputting the results in the form of natural language, the embodiment of the present application can improve COCO mAP from 12.9 to 33, significantly improving the possibility of applying large multimodal models to target detection. Compared with Shikra and Griffon, the parameter size of the generative language model GPT2 used in this embodiment is only 135 million, which is better than the other two models with a parameter size of 7 billion, and has achieved better detection indicators. At the same time, the embodiment of the present application can directly use the data set used by the traditional detection model, without the need to use paid models such as GPT-4V to generate a large amount of multimodal dialogue data, which greatly reduces development costs.

[0065] In an embodiment of the present application, for any multimodal large model, a certain number of learnable query vectors can be added to extract all detection-related features in the image through a forward propagation process, and the position coordinates can be obtained in batches using the regression head, replacing the multiple loop reasoning processes based on the autoregressive process of the existing multimodal large model, which can greatly reduce the reasoning delay. It is also possible to increase the classification head to output the classification probability, add confidence output to the model, filter out low-confidence outputs and use NMS to remove duplicate detection outputs, further improving the detection accuracy of the model. The classification head limits the category output space of the model, which effectively reduces the occurrence of hallucinations compared to the large model that makes predictions in all word vector spaces.

[0066] It should be noted that the target detection method provided in the embodiments of the present application can be executed by a target detection device or a control module in the target detection device for executing the target detection method. In the embodiments of the present application, the target detection device provided in the embodiments of the present application is described by taking the target detection device executing the target detection method as an example.

[0067] Figure 6 Schematic diagram of the structure of the target detection device according to the embodiment of the present application. Figure 6 As shown, the target detection device 600 includes: a first processing module 610 , a second processing module 620 and a detection module 630 .

[0068] The first processing module 610 is used to preprocess the image to be tested to obtain a mapping feature sequence of the image to be tested in the text embedding space; the second processing module 620 is used to preprocess the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; the detection module 630 is used to determine the detection features of the target object based on a preset query vector, the mapping feature sequence and the text embedding sequence.

[0069] In one implementation, the detection features include position feature information and category feature information. The detection module 630 is further used to process the detection features through a classification head and output the matching confidence between the category feature information in each detection feature and the category text information.

[0070] In one implementation, the detection module 630 is further used to obtain the target detection feature whose matching confidence meets the preset conditions from the detection feature; process the target detection feature through the regression head, and output the position coordinates according to the position feature information of the target detection feature.

[0071] In one implementation, the first processing module 610 is configured to encode the image to be tested into a visual feature sequence; and map the visual feature sequence to the text embedding space to obtain the mapped feature sequence.

[0072] In one implementation, the second processing module 620 is configured to map the category text information to the text embedding space to obtain the text embedding sequence.

[0073] In one implementation, the detection module 630 is used to splice the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; use the query vector as the query of the cross-attention and the multimodal sequence as the key and value of the cross-attention; and determine the detection feature based on the cross-attention, the query of the cross-attention, and the key and value of the cross-attention.

[0074] In one implementation, the detection module 630 is used to splice the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; splice the multimodal sequence and the query vector to obtain an input sequence; use the input sequence as the query, key and value of self-attention; and determine the detection feature based on the self-attention, the query, key and value of the self-attention.

[0075] The target detection device in the embodiments of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0076] The target detection device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0077] The target detection device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0078] like Figure 7 As shown, an embodiment of the present application further provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores a program or instruction that can be run on the processor 701, and when the program or instruction is executed by the processor 701, it implements: preprocessing the image to be tested to obtain a mapping feature sequence of the image to be tested in the text embedding space; preprocessing the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; and determining the detection features of the target object based on a preset query vector, the mapping feature sequence and the text embedding sequence.

[0079] In one implementation, the detection features include position feature information and category feature information. After determining the detection features of the target object, the detection features are processed by a classification head to output the matching confidence between the category feature information in each detection feature and the category text information.

[0080] In one implementation, after outputting the matching confidence between the category feature information and the category text information in each of the detection features, a target detection feature whose matching confidence meets a preset condition is obtained from the detection feature; the target detection feature is processed by a regression head, and the position coordinates are output according to the position feature information of the target detection feature.

[0081] In one implementation, the image to be tested is encoded into a visual feature sequence; and the visual feature sequence is mapped to the text embedding space to obtain the mapped feature sequence.

[0082] In one implementation, the category text information is mapped to the text embedding space to obtain the text embedding sequence.

[0083] In one implementation, the mapping feature sequence and the text embedding sequence are concatenated to obtain a multimodal sequence; the query vector is used as a query for cross-attention, and the multimodal sequence is used as a key and value for the cross-attention; and the detection feature is determined based on the cross-attention, the query for the cross-attention, and the key and value for the cross-attention.

[0084] In one implementation, the mapping feature sequence and the text embedding sequence are concatenated to obtain a multimodal sequence; the multimodal sequence and the query vector are concatenated to obtain an input sequence; the input sequence is used as the query, key, and value of self-attention; and the detection feature is determined based on the self-attention, the query, key, and value of the self-attention.

[0085] The specific execution steps can refer to the various steps of the above-mentioned target detection method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0086] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices other than terminals.

[0087] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured as a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes at least one of a touch panel and other input devices. A touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be detailed here.

[0088] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory may include volatile memory or non-volatile memory, or the memory may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus random access memory (DRRAM).

[0089] The processor may include one or more processing units; optionally, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and application programs, and the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor.

[0090] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned target detection method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0091] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0092] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the various processes of the embodiment of the above-mentioned target detection method and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0093] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of this application.

[0095] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A target detection method, characterized in that: include: Preprocessing the image to be tested to obtain a mapping feature sequence of the image to be tested in the text embedding space; Preprocessing the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space; The detection features of the target object are determined according to a preset query vector, the mapping feature sequence and the text embedding sequence.

2. The method according to claim 1, characterized in that The detection features include position feature information and category feature information. After determining the detection features of the target object, the method further includes: The detection features are processed by a classification head, and a matching confidence between the category feature information and the category text information in each of the detection features is output.

3. The method according to claim 2, characterized in that After outputting the matching confidence between the category feature information and the category text information in each of the detection features, the method further includes: Obtaining target detection features whose matching confidence meets preset conditions from the detection features; The target detection features are processed by a regression head, and position coordinates are output according to position feature information of the target detection features.

4. The method according to claim 1, wherein The preprocessing of the image to be tested to obtain a mapping feature sequence of the image to be tested in the text embedding space includes: Encoding the image to be tested into a visual feature sequence; The visual feature sequence is mapped to the text embedding space to obtain the mapped feature sequence.

5. The method according to claim 1, wherein The preprocessing of the category text information of the target object in the image to be tested to obtain a text embedding sequence of the category text information in the text embedding space includes: The category text information is mapped to the text embedding space to obtain the text embedding sequence.

6. The method according to claim 1, characterized in that The determining the detection feature of the target object according to the preset query vector, the mapping feature sequence, and the text embedding sequence includes: splicing the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; Using the query vector as a query for cross-attention and the multimodal sequence as a key and a value for the cross-attention; The detection feature is determined based on the cross-attention, the query of the cross-attention, the key and the value of the cross-attention.

7. The method according to claim 1, characterized in that The determining the detection feature of the target object according to the preset query vector, the mapping feature sequence, and the text embedding sequence includes: splicing the mapping feature sequence and the text embedding sequence to obtain a multimodal sequence; Concatenating the multimodal sequence and the query vector to obtain an input sequence; Treat the input sequence as the query, key, and value of self-attention; The detection feature is determined based on the self-attention, the query of the self-attention, the key and the value.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the target detection method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to perform the steps of the target detection method according to any one of claims 1 to 7.

Images