Parallel visual inspection method and device based on large language model and electronic equipment

Abstract

The application discloses a kind of parallel visual detection method, device and electronic equipment based on large language model related to image detection technical field, the method includes: obtaining the visual feature sequence indicating image to be detected;Learnable visual query vector is constructed;Visual feature sequence and visual query vector are spliced, combined with the natural language input sequence of the target to be detected Commonly input large language model, update visual query vector, and generate the category token of target to be detected and the semantic feature representation for describing category token in self-recurrence mode;Each visual query vector is fused with semantic feature representation based on detection head, and the target feature of fusion semantic information is obtained;Target feature is processed based on the prediction branch, and multiple detection results are output in parallel.The method can complete the category identification and position positioning of multiple targets in parallel mode, improve overall detection efficiency and detection performance.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a parallel visual detection method, apparatus, and electronic device based on a large language model. Background Technology

[0002] With the development of multimodal artificial intelligence technology, the Vision-Language Model (VLM), which combines a Large Language Model (LLM) with a visual encoder, has been widely applied in tasks such as image understanding, visual question answering, and cross-modal retrieval. Building on this, some studies have attempted to introduce object detection tasks into the VLM framework. By designing natural language prompts, the model is guided to output the object's category and its location information in the image in the form of a text sequence, thus achieving open vocabulary detection and zero-shot transfer capabilities to some extent. These methods typically utilize a visual encoder to extract features from the input image, and then input the obtained visual features and text prompts together into a large language model. The model then generates text tokens describing the detection results one by one using an autoregressive approach, and finally parses the generated text into corresponding detection boxes.

[0003] However, the aforementioned object detection methods based on language generation paradigms inherently rely on the autoregressive generation mechanism of large language models. Their detection results must be output step-by-step according to the token sequence, making it difficult to predict multiple targets in parallel. When there are many targets in the scene to be detected, the inference process needs to generate a large number of tokens, leading to a significant increase in overall inference latency, making it difficult to meet the requirements of applications with high real-time requirements. Furthermore, since the output space of large language models is a discrete vocabulary space, continuous target bounding box coordinates usually need to be represented through discretization encoding or by introducing special coordinate tokens. This representation inevitably introduces quantization errors, limiting the performance of detection results in high-precision localization tasks, especially in applications with high requirements for bounding box overlap, making it difficult to further improve localization accuracy. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, the objective of this invention is to propose a parallel visual detection method, apparatus, and electronic device based on a large language model, to improve the parallel inference efficiency of multi-target detection and enhance target localization accuracy while maintaining language semantic alignment capabilities.

[0005] To achieve the above objectives, a first aspect of the present invention proposes a parallel visual detection method based on a large language model, comprising:

[0006] Visual encoding is performed on the image to be detected to obtain a sequence of visual features representing the image to be detected;

[0007] Construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image to be detected in parallel;

[0008] The visual feature sequence is concatenated with the visual query vector, and then combined with the natural language input sequence of the target to be detected and input into a pre-trained large language model to update the visual query vector. The category token of the target to be detected and the semantic feature representation for describing the category token are generated in an autoregressive manner.

[0009] Based on the detection head in the large language model, each visual query vector is fused with the semantic feature representation to obtain the target feature with fused semantic information;

[0010] The target features are processed based on the prediction branch in the large language model, and multiple detection results are output in parallel; wherein each detection result includes at least the probability information of the category to which the target belongs and the coordinates of the target bounding box.

[0011] In addition, the method of the above embodiments of the present invention may also have the following additional technical features:

[0012] According to one embodiment of the present invention, generating a semantic feature representation for describing the category token includes:

[0013] After generating the category token, a predefined placeholder token is introduced, and the latent features of the placeholder token in the large language model are extracted as semantic feature representations to describe the category token.

[0014] According to one embodiment of the present invention, generating a semantic feature representation for describing the category token includes:

[0015] Extract the hidden features of the category word token corresponding to the generated category token in the large language model, and use the hidden features as semantic feature representation independent of the category token decoding result.

[0016] According to an embodiment of the present invention, concatenating the visual feature sequence with the visual query vector includes:

[0017] The visual query vector is used as a prefix or suffix sequence and concatenated with the visual feature sequence.

[0018] According to one embodiment of the present invention, before fusing each of the visual query vectors with the semantic feature representation, the method further includes:

[0019] Parallel self-attention processing is performed on the updated visual query vector.

[0020] According to one embodiment of the present invention, fusing each of the visual query vectors with the semantic feature representation includes:

[0021] Using the visual query vector obtained after interaction as the query term, and the semantic feature representation as the key input and value input, cross-attention computation is performed to obtain the target feature.

[0022] According to an embodiment of the present invention, the step of fusing each of the visual query vectors with the semantic feature representation further includes:

[0023] The visual query vector and the semantic feature representation are fused by feature concatenation, element-by-element addition, or through a gating mechanism.

[0024] According to one embodiment of the present invention, the large language model includes the following during the training process:

[0025] Cross-entropy loss is used to supervise the intermediate tokens and category tokens generated by the large language model; the intermediate tokens refer to other intermediate generated tokens in the feature sequence output by the large language model, excluding the visual query vector, category token, and the semantic feature representation of the category token;

[0026] The detection loss is introduced through the detection results output by the detection head, and joint backpropagation optimization is performed on the update of the visual query vector and the model parameters related to the semantic feature representation; wherein,

[0027] The detection loss includes classification loss and regression loss after sample allocation based on the matching strategy. The classification loss is used to constrain the target category prediction result, and the regression loss is used to constrain the target bounding box coordinate prediction result.

[0028] To achieve the above objectives, a second aspect of the present invention provides a parallel visual detection device based on a large language model, comprising:

[0029] The visual encoding module is used to perform visual encoding on the image to be detected to obtain a sequence of visual features representing the image to be detected.

[0030] A visual query vector construction module is used to construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image to be detected in parallel.

[0031] The detection module is used to concatenate the visual feature sequence with the visual query vector, and input them together with the natural language input sequence of the target to be detected into a pre-trained large language model to update the visual query vector. It then generates a category token for the target to be detected and a semantic feature representation describing the category token using an autoregressive approach. Based on the detection head in the large language model, it fuses each visual query vector with the semantic feature representation to obtain target features with fused semantic information. Finally, it processes the target features based on the prediction branch in the large language model and outputs multiple detection results in parallel. Each detection result includes at least the probability information of the category to which the target to be detected belongs and the coordinates of the target bounding box.

[0032] To achieve the above objectives, a third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described parallel visual detection method based on a large language model.

[0033] The parallel visual detection method, apparatus, and electronic device based on a large language model in this invention introduces a learnable visual query vector on top of visual feature encoding, enabling the spatial information of multiple targets to be detected to be modeled in parallel. The updated visual query vector is then fused with the semantic feature representation corresponding to the category token generated by the large language model, thereby completing target category recognition and bounding box regression without relying on discrete coordinate token output. By decoupling the target detection process from the pure text autoregressive generation paradigm, parallel output of multiple detection results is achieved, effectively reducing inference latency. Simultaneously, the target bounding box coordinates in the detection results are predicted as continuous values, reducing quantization errors introduced by discrete encoding. This results in better accuracy and stability in high-precision positioning scenarios, thereby improving the practicality of the large language model-based visual detection method in complex scenes. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating a parallel visual detection method based on a large language model in one embodiment.

[0035] Figure 2 This is a flowchart illustrating the training process loss function in one embodiment;

[0036] Figure 3 This is a schematic diagram of the data processing flow of a large language model in one embodiment;

[0037] Figure 4 This is a block diagram of a parallel visual inspection device based on a large language model in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] The implementation details of the technical solutions of the embodiments of the present invention are described in detail below.

[0040] In one embodiment, such as Figure 1 The diagram illustrates a flowchart of a parallel visual detection method based on a large language model, which may include the following steps:

[0041] Step S101: Visually encode the image to be detected to obtain a visual feature sequence representing the image to be detected.

[0042] The image to be detected is input into a pre-selected visual encoding method. The image is divided into blocks, and features are extracted from each block, thereby converting the image to be detected into a set of structured visual feature representations. Through this visual encoding process, a visual feature sequence composed of multiple visual tokens can be obtained. ,in, This indicates the corresponding number of pixels in the image to be detected. Visual feature representation of an image patch This indicates the number of image patches after the image has been divided. In practical applications, the image to be detected can come from an object detection dataset, such as the COCO dataset.

[0043] In the above implementation, the visual encoding method can be implemented based on the pre-configured visual encoding capabilities in the large language model, or it can be implemented using an external visual encoding method that works in conjunction with the large language model. This method transforms the original image information into a visual feature sequence suitable for joint modeling with language features.

[0044] Step S102: Construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image in parallel.

[0045] Construct a sequence of visual query vectors composed of multiple learnable features ,in, Indicates the first A visual query vector This represents the number of visual query vectors, in practical applications. 300 can be selected.

[0046] The visual query vector is a trainable vector introduced into the large language model. Its initial value can be set through random initialization or a predefined distribution, and it participates in the update during the subsequent forward computation of the large language model. Specifically, during the model training phase, the visual query vector is optimized as a learnable parameter according to the training objective; during the model inference phase, the visual query vector uses the parameters after training and participates in the forward computation without being updated through backpropagation.

[0047] By constructing multiple sets of visual query vectors simultaneously, different visual query vectors can be used to focus on and represent different potential target regions in the image to be detected in parallel during the same computation process. This allows them to carry the spatial information representation of multiple targets to be detected and provides a foundation for the subsequent fusion of target category semantic information and the parallel prediction of detection results.

[0048] Step S103: The visual feature sequence and the visual query vector are concatenated and input into the pre-trained large language model together with the natural language input sequence of the target to be detected. The visual query vector is updated, and the category token of the target to be detected and the semantic feature representation used to describe the category token are generated in an autoregressive manner.

[0049] After constructing the visual feature sequence and the visual query vector, the visual feature sequence and the visual query vector are concatenated to form a joint visual input sequence. The input interface of the large language model can simultaneously receive visual tokens encoded from visual features and text tokens encoded from natural language, and perform unified modeling on both. Based on this, the joint visual input sequence is used as the visual token, and together with the text token corresponding to the natural language input sequence describing the target to be detected, it is input into the large language model for processing. This allows visual information and linguistic semantic information to be aligned and interactively modeled within the same representation space. The natural language input sequence provides category semantic constraints relevant to the detection task, guiding the large language model to understand category semantics during multimodal joint modeling.

[0050] The joint input sequence is computed using an autoregressive modeling approach. During this process, the visual query vector interacts multimodally with the visual feature sequence and the natural language input sequence within the same representation space. This allows the visual query vector to be updated by incorporating image content and category semantic information, resulting in an updated visual query vector that includes spatial information of the target object to be detected. .

[0051] Simultaneously, during the autoregressive generation of the output sequence, the large language model generates a category token corresponding to the target to be detected, denoted as... And generate a semantic feature representation for describing the category token, denoted as . .

[0052] Based on this, after performing autoregressive calculations on the joint input sequence, the large language model outputs a set of structured feature sequences, which include:

[0053]

[0054] Among them, the updated visual query vector The output token, which corresponds to the input visual query vector after fusing visual features and linguistic semantic information, is used to carry spatial information representations related to different target regions. This refers to intermediate generated tokens other than the visual query vector, category token, and their corresponding semantic feature representations. This represents a category token generated by a large language model to indicate the category of the target to be detected. Description Semantic features.

[0055] In practical applications, since LLMs typically use an autoregressive generation mechanism based on the prediction of the next token for modeling, and the semantic feature representations corresponding to the category tokens are not directly used as predictable discrete token outputs, it will be difficult to effectively process the hidden features corresponding to the categories in the training or inference phases while maintaining the consistency of the autoregressive generation mechanism.

[0056] Based on this, in one embodiment, after generating the category token, a predefined placeholder token is introduced. This placeholder token is located in the sequence following the category token and carries the semantic feature representation corresponding to the category token in the hidden layer of the large language model. The placeholder token is inserted into the input or output sequence of the large language model according to a predetermined positional relationship, providing a stable and aligned representation position for the semantic features corresponding to the category token during autoregressive computation.

[0057] When a large language model performs forward computation on sequences containing placeholder tokens, the placeholder tokens form hidden feature representations in the model's hidden layers that are associated with category tokens. By extracting the feature vectors of the placeholder tokens in the hidden layers of the large language model, semantic feature representations corresponding to category tokens are obtained. This allows for the effective extraction and utilization of category tokens without altering the original autoregressive generation mechanism of the large language model.

[0058] In another embodiment, instead of introducing an additional placeholder token, the latent features formed by the corresponding category word token during the generation of the category token are directly used as semantic feature representations to describe the category token.

[0059] Specifically, during the forward computation of the large language model, the category word token participates in multimodal modeling together with the joint visual input sequence and the natural language input sequence. The corresponding hidden state naturally integrates visual information and category semantic information in the specific hidden output after the category word token generation position. By extracting the feature vector of the category word token in the hidden layer of the large language model, the semantic feature representation corresponding to the category token is obtained. This semantic feature representation is independent of the decoding result of the category token.

[0060] In practical applications, extracting hidden features of category words (tokens) can utilize hidden features closer to the output side of a large language model, such as the output features of the penultimate layer, to ensure rich semantic information and include visual contextual interaction information. The natural language input describing the target to be detected can be uniformly set as a general descriptive statement used to instruct the detection task, such as "detect people, cars, etc. in this image."

[0061] It should be noted that the category token is used to decode and output the target category token, while the category word token refers to the discrete word unit used to represent the semantics of the target category. It can be the category name itself or its corresponding sub-word token, and is used to extract the semantic features of the target category.

[0062] This approach allows for the extraction of semantic features of category tokens without introducing additional placeholders into the model's input or output sequences, thus providing semantic information support for subsequent fusion processing with visual query vectors.

[0063] In one embodiment, the visual query vector and the visual feature sequence are concatenated in a predetermined order according to the processing method of the large language model for the input sequence to form a joint visual input sequence.

[0064] Specifically, the visual query vector can be appended as a prefix sequence to the visual feature sequence, allowing it to participate preferentially in context modeling during the autoregressive computation of the large language model. Alternatively, the visual query vector can be appended as a suffix sequence to the visual feature sequence, further guiding attention to the target region based on complete visual feature modeling. Through these different appending methods, the position of the visual query vector can be flexibly configured without changing the visual feature generation method.

[0065] In practical applications, the stitched joint visual input sequence When input into a large language model along with the natural language input sequence, the concatenation position of the natural language input sequence can also be selected according to specific task requirements. For example, the natural language input sequence can be concatenated as a prefix before the joint visual input sequence to provide global semantic constraints in the early stages of model inference. Alternatively, the natural language input sequence can be concatenated as a suffix after the joint visual input sequence, allowing the model to fully perceive visual information before introducing linguistic semantic guidance. By flexibly setting the concatenation order of the visual query vector, visual feature sequence, and natural language input sequence, the collaborative modeling capability of multimodal information in the large language model can be enhanced.

[0066] Step S104: Based on the detection head in the large language model, each visual query vector is fused with the semantic feature representation to obtain the target feature with fused semantic information.

[0067] The detection head in a large language model is used to fuse each visual query vector with its semantic feature representation. Specifically, the detection head receives a visual query vector and a set of semantic features consisting of semantic feature representations corresponding to tokens of multiple categories. During the forward computation process, the category semantic information is explicitly injected into the feature representation of the visual query vector through the feature interaction mechanism, so that each visual query vector can establish a relationship with the semantic feature representation based on the image region it is interested in, thereby introducing the semantic constraints of the category at the feature level.

[0068] Through the above fusion process, the visual query vector is transformed from an intermediate representation that only contains spatial and appearance information into target features that simultaneously encode the target's spatial location, visual appearance features, and category semantic constraints. The target features obtained in this way can directly reflect the matching relationship between visual regions and categories at the feature level, providing a unified and semantically enhanced feature foundation for subsequent category judgment and detection result output based on these target features.

[0069] In one embodiment, before performing the fusion operation between the visual query vectors and semantic feature representations, parallel self-attention processing is simultaneously performed on multiple updated visual query vectors. Specifically, during the self-attention computation, each visual query vector interacts with other visual query vectors and passes information within a parallel computing framework, enabling different visual query vectors to share contextual information.

[0070] This parallel self-attention processing can encourage different visual query vectors to form differentiated attention in the feature space, reducing the situation where multiple visual query vectors are concentrated in the same target region, thereby reducing the possibility of repeated detection and providing a more stable and complementary query feature foundation for subsequent fusion with semantic feature representation.

[0071] In practical applications, the detection head includes a Self-Attention module for performing parallel self-attention processing, and the updated visual query vector is input to the detection head. and semantic feature representation set Apply the Self-Attention module to This enables interaction between N visual query vectors. The detection head can employ two layers of Self-Attention modules.

[0072] In one embodiment, after completing the parallel self-attention processing of the visual query vector, the processed visual query vector is used as the query term, and the semantic feature representation corresponding to the target category is used as the key input and value input to perform cross-attention computation. Through this cross-attention computation, each visual query vector can actively select and aggregate target category semantic information with a high degree of matching from multiple semantic feature representations based on its spatial attention region.

[0073] Through the cross-attention mechanism, the visual query vector adaptively extracts semantic information with a high degree of matching from the semantic feature representation based on the features of the currently focused image region. This achieves alignment and fusion of visual information and category semantic information at the feature level, generating a target feature representation that incorporates the semantic constraints of the target category. This approach enables explicit alignment between visual spatial information and category semantic information, improving the discriminative ability of target features for category classification.

[0074] In practical applications, the detection head also includes a Cross-Attention module for performing cross-attention fusion. For query items, with The inputs are key and value, fed into the Cross-Attention module for cross-attention fusion, enabling each visual query to actively seek semantic features in the current text domain. For example, when a visual query vector focuses on the lower left corner of the image, if its semantic features match the category "Car" well, the visual query vector will integrate the semantic information of "Car" during the fusion process. The detection head can employ two layers of Cross-Attention modules.

[0075] In one embodiment, when fusing the visual query vector with the semantic feature representation, cross-attention computation is not used; instead, semantic information injection is performed based on a feature-level fusion approach. Specifically, this may include the following methods:

[0076] (1) The visual query vector and the semantic feature representation are concatenated along the feature dimension to form a joint feature representation containing visual and semantic information;

[0077] (2) Perform an element-wise addition operation on the visual query vector and the semantic feature representation to directly enhance the visual features with the semantic features;

[0078] (3) Introduce a gating mechanism to adaptively adjust the weight of different features in the fusion result based on the correlation between the visual query vector and the semantic feature representation.

[0079] By using the above fusion method, while maintaining a simple implementation structure, the visual query vector and semantic feature representation are effectively combined to generate target features containing category semantic information for subsequent detection result prediction.

[0080] Step S105: Process the target features based on the prediction branch in the large language model and output multiple detection results in parallel.

[0081] The target features are input into the prediction branch for processing. For each target feature, the prediction branch performs independent forward computations, outputting the corresponding target category prediction and target bounding box prediction results in parallel. The prediction branch includes a classification branch and a regression branch. The classification branch performs the classification process to determine the target category, while the regression branch performs the regression prediction of the target's position in the image to be detected. In practical applications, both the classification and regression branches employ a two-layer multilayer perceptron (MLP) structure.

[0082] For the Each target feature is used to predict a set of detection results. ,in, This indicates the bounding box coordinates of the target object in the image to be detected. Category prediction result. This represents the probability value of a target feature belonging to each target category. Multiple target categories correspond to multiple category tokens generated by the large language model in the preceding steps, and the probability of each category reflects the degree of matching between the target feature and the corresponding category semantic feature.

[0083] Since each target feature is processed independently in the prediction branch, the above-mentioned category prediction and bounding box regression processes can be performed simultaneously on all target features, thus forming a set of detection results:

[0084]

[0085] This method achieves parallel output of category recognition and location localization for multiple targets in the image to be detected. While avoiding the inference delay and result coupling caused by generating detection boxes sequentially using a large language model, it avoids discretizing the target position by using an independent regression branch to perform continuous value regression prediction on the target bounding box coordinates during the prediction stage, thus balancing detection efficiency and target localization accuracy.

[0086] In one embodiment, a large language model is trained using an end-to-end approach during the model training phase. During training, such as... Figure 2 The flowchart shown illustrates the training process loss function. For different types of output generated by the large language model during model training, text loss and detection loss are jointly calculated to achieve joint optimization of language generation and object detection tasks.

[0087] In practical applications, the calculation of text loss reflects the differentiated supervision constraints imposed on different types of output tokens. Specifically, for intermediate tokens generated by a large language model (i.e., those in the output feature sequence)... ) and category token (i.e., in the output feature sequence) The model employs a cross-entropy-based loss function for supervision and constraint, enabling it to maintain stable language generation and category identification prediction capabilities during training.

[0088] Meanwhile, for the updated visual query vector and semantic feature representations corresponding to category tokens Instead of directly introducing cross-entropy loss for constraint, it optimizes the detection loss constructed based on the detection results through backpropagation, enabling this type of feature to adaptively adjust around the detection task objective.

[0089] For detection results based on target features, a sample matching strategy is introduced during the training phase to assign predicted results to ground truth labels. Specifically, a one-to-one correspondence between predicted and ground truth targets is established based on a matching strategy (such as Hungarian matching), and classification loss and regression loss are calculated accordingly. The classification loss constrains the consistency between the predicted target category and the ground truth category label to improve the accuracy of target category discrimination. The regression loss constrains the spatial deviation between the predicted target bounding box and the ground truth bounding box to improve target localization accuracy. In practical applications, the regression loss can be composed of a distance loss (such as L1 loss) to constrain bounding box position errors and a shape consistency loss (such as GIoU loss) to constrain bounding box overlap relationships.

[0090] Figure 3The diagram illustrates the data processing flow of a large language model. This model leverages its powerful semantic understanding capabilities to serially generate descriptions of target categories present in the image to be detected. Simultaneously, it uses a set of learnable visual query vectors to extract spatial features from the image in parallel. Finally, an attention mechanism aligns the visual queries with the semantic descriptions, achieving efficient and high-precision detection. The data processing flow begins at the input stage, where the image to be detected is input into the visual encoder. Through block processing and feature extraction, it is transformed into a sequence of visual features composed of multiple visual tokens.

[0091] Simultaneously, a set of visual query vector sequences containing N learnable features is constructed, and these are linearly concatenated with the visual feature sequence and the text prompt word sequence in a preset order to form the concatenated input sequence Y. During the concatenation process, the position of the visual query vectors is flexible and can be inserted as prefixes, suffixes, or intermediate sequences to enhance the collaborative modeling capability of multimodal information.

[0092] The input sequence Y is concatenated and input into the Transformer layer of a large language model for multimodal joint modeling. During the LLM forward propagation, the processing logic is divided into two paths: on the one hand, the visual query vector interacts multimodally with visual and textual features to complete parallel feature updates to carry spatial information; on the other hand, the model autoregressively generates a text sequence based on a next-token prediction mechanism, which contains at least one category token. To effectively extract semantic features under the autoregressive mechanism, the hidden features corresponding to the category token are extracted to obtain the semantic feature representation. Specifically, this can be achieved by generating a predefined placeholder token (denoted as FEAT) after the category token and extracting its hidden features.

[0093] After completing the LLM internal processing, the updated visual query vector is used as the Query, and the extracted semantic features are represented. Both the key and value are input into the image-text alignment detection head H. Inside the detection head, the target category semantics are explicitly injected into the visual query vector through cross-attention mechanisms or other feature-level fusion methods, transforming the original spatial representation into target features that simultaneously encode position, appearance, and semantics. To reduce duplicate detections, parallel self-attention processing can be performed on the query vector before fusion to create differentiated attention.

[0094] Finally, these fused target features are processed in parallel by the prediction branch in LLM, directly outputting a set of detection results containing bounding box coordinates and class probability distributions, thereby achieving parallel localization and recognition of multiple targets in the image.

[0095] In the above embodiments, by visually encoding the image to be detected and constructing a learnable visual query vector, information from different target regions can be represented in parallel in the feature space. This is then combined with the natural language input sequence of the target to be detected in a pre-trained large language model for joint modeling, thereby achieving effective fusion of visual features and category semantic information. Based on this, the detection head fuses each visual query vector with semantic feature representations to generate target features that simultaneously encode the target's spatial location, visual appearance features, and category semantic constraints. The prediction branch then processes the target features and outputs multiple detection results in parallel, achieving category discrimination and bounding box localization for each target. This method fully leverages the advantages of large language models in multimodal information modeling, enabling deep alignment of visual and category semantic information at the feature level, improving the detection accuracy of multiple targets in complex scenes, and simultaneously achieving parallel processing of the detection process, thus improving detection efficiency.

[0096] In one embodiment, a parallel visual detection device based on a large language model is provided, with reference to... Figure 4 As shown, the parallel visual detection device 400 based on a large language model may include: a visual encoding module 401, a visual query vector construction module 402, and a detection module 403. Among them,

[0097] The visual encoding module 401 is used to perform visual encoding on the image to be detected to obtain a sequence of visual features representing the image to be detected.

[0098] The visual query vector construction module 402 is used to construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image in parallel.

[0099] The detection module 403 is used to concatenate the visual feature sequence with the visual query vector, and input it together with the natural language input sequence of the target to be detected into a pre-trained large language model to update the visual query vector. It also generates a category token of the target to be detected and a semantic feature representation describing the category token in an autoregressive manner. Based on the detection head in the large language model, it fuses each visual query vector with the semantic feature representation to obtain target features with fused semantic information. Based on the prediction branch in the large language model, it processes the target features and outputs multiple detection results in parallel. Each detection result includes at least the probability information of the category to which the target belongs and the coordinates of the target bounding box.

[0100] In one embodiment, the detection module 403 is specifically used to introduce a predefined placeholder token after generating the category token, and extract the hidden features of the placeholder token in the large language model as a semantic feature representation to describe the category token.

[0101] In one embodiment, the detection module 403 is specifically used to extract the hidden features of the category word token corresponding to the generated category token in the large language model, and use the hidden features as a semantic feature representation independent of the category token decoding result.

[0102] In one embodiment, the detection module 403 is specifically used to concatenate the visual query vector as a prefix sequence or a suffix sequence with the visual feature sequence.

[0103] In one embodiment, before fusing each visual query vector with a semantic feature representation, the detection module 403 is specifically configured to perform parallel self-attention processing on the updated visual query vectors.

[0104] In one embodiment, the detection module 403 is specifically used to perform cross-attention computation to obtain target features by using the visual query vector after interaction as the query item and the semantic feature representation as the key input and value input.

[0105] In one embodiment, the detection module 403 is specifically used to perform feature concatenation, element-wise addition, or fusion of the visual query vector and semantic feature representation through a gating mechanism.

[0106] In one embodiment, the detection module 403 is also used during the training of the large language model to: supervise the intermediate tokens and category tokens generated by the large language model using cross-entropy loss; the intermediate tokens refer to other intermediate generated tokens in the feature sequence output by the large language model, excluding the visual query vector, the category token, and the semantic feature representation corresponding to the category token;

[0107] The detection loss is introduced using the detection results output by the detection head, and joint backpropagation optimization is performed on the model parameters related to the update of the visual query vector and the semantic feature representation; among them,

[0108] The detection loss includes classification loss and regression loss after sample allocation based on the matching strategy. The classification loss is used to constrain the target category prediction result, and the regression loss is used to constrain the target bounding box coordinate prediction result.

[0109] Specific limitations regarding the parallel visual inspection device 400 based on a large language model can be found in the limitations of the parallel visual inspection method based on a large language model described above, and will not be repeated here. Each module in the parallel visual inspection device 400 based on a large language model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0110] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a parallel visual detection method based on a large language model.

[0111] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A parallel visual detection method based on a large language model, characterized in that, include: Visual encoding is performed on the image to be detected to obtain a sequence of visual features representing the image to be detected; Construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image to be detected in parallel; The visual feature sequence is concatenated with the visual query vector, and then combined with the natural language input sequence of the target to be detected and input into a pre-trained large language model to update the visual query vector. An autoregressive approach is used to generate a category token for the target to be detected and a semantic feature representation describing the category token. Generating the semantic feature representation describing the category token includes: after generating the category token, introducing a predefined placeholder token and extracting the latent features of the placeholder token in the large language model as the semantic feature representation describing the category token; or, extracting the latent features of the category word token corresponding to the generated category token in the large language model and using the latent features as a semantic feature representation independent of the category token decoding result. Based on the detection head in the large language model, each visual query vector is fused with the semantic feature representation to obtain target features with fused semantic information; the semantic feature representation participates in the feature interaction with the visual query vector as key input and value input, so as to introduce semantic constraints of category on the visual query vector based on the semantic feature representation; The target features are processed based on the prediction branch in the large language model, and multiple detection results are output in parallel. Each detection result includes at least the probability information of the category to which the target belongs and the target bounding box coordinates. The target bounding box coordinates are predicted by continuous value regression through an independent regression branch.

2. The parallel visual detection method based on a large language model according to claim 1, characterized in that, The step of concatenating the visual feature sequence with the visual query vector includes: The visual query vector is used as a prefix or suffix sequence and concatenated with the visual feature sequence.

3. The parallel visual detection method based on a large language model according to claim 1, characterized in that, Before fusing each of the visual query vectors with the semantic feature representation, the method further includes: Parallel self-attention processing is performed on the updated visual query vector.

4. The parallel visual detection method based on a large language model according to claim 3, characterized in that, The step of fusing each visual query vector with the semantic feature representation includes: Using the visual query vector obtained after interaction as the query term, and the semantic feature representation as the key input and value input, cross-attention computation is performed to obtain the target feature.

5. The parallel visual detection method based on a large language model according to claim 3, characterized in that, The step of fusing each of the visual query vectors with the semantic feature representation further includes: The visual query vector and the semantic feature representation are fused by feature concatenation, element-by-element addition, or through a gating mechanism.

6. The parallel visual detection method based on a large language model according to claim 1, characterized in that, During the training of the large language model, the method further includes: Cross-entropy loss is used to supervise the intermediate tokens and category tokens generated by the large language model; the intermediate tokens refer to other intermediate generated tokens in the feature sequence output by the large language model, excluding the visual query vector, category token, and the semantic feature representation of the category token; The detection loss is introduced through the detection results output by the detection head, and joint backpropagation optimization is performed on the update of the visual query vector and the model parameters related to the semantic feature representation; wherein, The detection loss includes classification loss and regression loss after sample allocation based on the matching strategy. The classification loss is used to constrain the target category prediction result, and the regression loss is used to constrain the target bounding box coordinate prediction result.

7. A parallel visual inspection device based on a large language model, characterized in that, include: The visual encoding module is used to perform visual encoding on the image to be detected to obtain a sequence of visual features representing the image to be detected. A visual query vector construction module is used to construct a set of learnable visual query vectors to represent the spatial information of the target to be detected in the image to be detected in parallel. The detection module is used to concatenate the visual feature sequence with the visual query vector, and input them together with the natural language input sequence of the target to be detected into a pre-trained large language model to update the visual query vector. It then generates a category token for the target to be detected and a semantic feature representation describing the category token using an autoregressive approach. Generating the semantic feature representation describing the category token includes: after generating the category token, introducing a predefined placeholder token and extracting the hidden features of the placeholder token in the large language model as the semantic feature representation describing the category token; or, extracting the category word token corresponding to the generated category token from the large language model. The model extracts hidden features, which are then used as semantic feature representations independent of the category token decoding results. Based on the detection head in the large language model, each visual query vector is fused with the semantic feature representation to obtain target features with fused semantic information. The semantic feature representation serves as both key and value inputs in the feature interaction with the visual query vector, introducing category-based semantic constraints on the visual query vector. The target features are processed based on the prediction branch in the large language model, outputting multiple detection results in parallel. Each detection result includes at least the probability information of the category to which the target belongs and the target bounding box coordinates. The target bounding box coordinates are predicted using continuous value regression through an independent regression branch.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the parallel visual detection method based on a large language model as described in any one of claims 1 to 6.

Images