Detection method and device based on multi-modal target detection model, and electronic device

Abstract

This invention discloses a detection method, apparatus, and electronic device based on a multimodal object detection model, belonging to the field of model detection technology. The method includes: acquiring a pre-constructed multimodal object detection model, which employs a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between state text and video images is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target bounding box regression head are trained to adapt to a specified industry distribution; industry-specific prompt text and extracted image frames are input into the multimodal object detection model to extract target object features and output target bounding box coordinates to locate the target object position. This invention solves the technical problem of low accuracy in locating dense target scenes in related technologies.

Description

Technical Field

[0001] This invention relates to the field of model detection technology or other related fields. Specifically, it relates to a detection method, apparatus, and electronic device based on a multimodal target detection model. Background Technology

[0002] Object detection aims to identify multiple targets in an image and output their category and spatial location (bounding box). It has been widely used in key fields such as autonomous driving, safety inspection, industrial quality inspection, and smart healthcare.

[0003] In related technologies, when performing object detection, the object detection models used can only identify the preset categories in the training set, which cannot meet the detection needs in open scenarios. To solve this drawback, Open-Vocabulary Object Detection (OVD) has emerged. The main approach is a detection scheme based on generative large language models. These models are based on the Transformer architecture and output the location coordinates and category of the target through text generation. However, although this detection method has powerful complex semantic understanding capabilities and general knowledge reasoning capabilities, it has obvious drawbacks: the positioning accuracy is low in dense target and small target scenarios, and the number of parameters is huge, resulting in low inference efficiency. It is difficult to meet the engineering deployment requirements with high real-time requirements. It not only cannot accurately identify specific industry terms, but also cannot balance the semantic understanding capabilities of open vocabulary with the accurate detection capabilities of dense / small targets. The detection effect is not ideal in complex security detection and industrial scenarios.

[0004] On the other hand, dataset fusion solutions in related technologies require unified annotation rules, category systems, and label formats. They rely on manual screening and data cleaning, cannot directly integrate historical datasets with different category systems and annotation specifications, and have low data utilization. Different datasets have inconsistent category systems and annotation specifications, with significant differences in the number of categories. They require manual format unification, resulting in low data reuse and high training costs.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a detection method, apparatus, and electronic device based on a multimodal target detection model, to at least solve the technical problem of low accuracy in target detection models for locating dense target scenes in related technologies.

[0007] To achieve the above objectives, according to one aspect of this application, a detection method based on a multimodal target detection model is provided, comprising: responding to a regional security monitoring instruction, acquiring real-time video data of a target region, and extracting image frames from the real-time video data; acquiring a pre-constructed multimodal target detection model, wherein the multimodal target detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and the video image is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the bimodal architecture and feature fusion layer of the backbone are frozen, and only the projection layer, type feature representation head, and target bounding box regression head are trained to adapt to a specified industry distribution; inputting pre-set industry prompt text and the extracted image frames into the multimodal target detection model, and having the multimodal target detection model extract target object features and output target bounding box coordinates to locate the target object position.

[0008] Optionally, the multimodal object detection model includes: a text encoder, which performs text segmentation on the input diverse Chinese tag list and generates semantic vectors based on the text segmentation, and generates a text semantic feature matrix by combining the semantic vector generation results; a visual encoder, which extracts multi-scale features from video images to obtain multi-level image features; a feature fusion layer, which performs cross-modal fusion on the text semantic feature matrix and the image features to output multi-level image features; a projection layer, which uses a multi-layer perceptron structure connected to the text encoder to map the text semantic feature matrix to the object detection space, wherein the multi-layer perceptron structure uses two fully connected layers; a type feature representation layer, which connects to the feature fusion layer and uses a three-layer convolutional network to replace the category prediction branch, which performs convolution processing on each layer of image features output by the feature fusion layer to output a multi-dimensional feature matrix; and a target bounding box regression layer, which outputs the target bounding box coordinates based on the multi-dimensional feature matrix and the text feature matrix output by the object detection space.

[0009] Optionally, the diversified Chinese tag list is obtained through the following methods: obtaining the original tag category name and industry target dictionary, wherein the industry target dictionary includes the parent category name, the list of synonyms and aliases, and the list of random descriptive phrases; searching the industry target dictionary for the parent category name, the list of synonyms and aliases, and the list of random descriptive phrases corresponding to each original tag category name; randomly selecting a random descriptive phrase, and randomly selecting a placeholder from the original tag category name, synonyms and aliases, parent category name, and parent category name to replace the placeholder in the phrase, thereby generating the diversified Chinese tag list, wherein the diversified Chinese tag list includes positive sample tags and negative sample tags.

[0010] Optionally, the step of performing cross-modal fusion of the text semantic feature matrix and the image features to output multi-layer image features includes: performing cross-modal fusion of the text semantic feature matrix with multi-scale image features at each level, enhancing the image feature expression through text semantics, and retaining the multi-layer image features after fusion enhancement.

[0011] Optionally, the pre-training stage of the multimodal object detection model includes: collecting an open-source image-text detection dataset, batch-annotating the open-source image-text detection dataset with Chinese characters to obtain a Chinese image-text pre-training dataset, and performing a cross-dataset batch processing fusion strategy on the Chinese image-text pre-training dataset; in the pre-training stage, freezing the parameters of the text encoder and opening the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation layer, and target box regression layer head for training; using the Chinese image-text pre-training data to train the initialized multimodal object detection model, the initialized multimodal object detection model learns the cross-modal alignment relationship between text and image, establishes text semantic representation capabilities, and obtains the parameters of the pre-trained multimodal object detection model.

[0012] Optionally, the fine-tuning stage of the multimodal object detection model includes: retrieving an open-source image and text detection dataset and constructing an industry-specific object dictionary based on the open-source image and text detection dataset; freezing the parameters of the text encoder, visual encoder, and feature fusion layer during the fine-tuning stage, and only opening the parameters of the projection layer, type feature representation layer, and target box regression layer for training; training the multimodal object detection model using an industry-specific dataset and the industry-specific object dictionary; performing cross-dataset batch fusion processing on different historical detection datasets in the industry-specific object dictionary; controlling the multimodal object detection model to complete industry data distribution and industry terminology recognition operations; and outputting the parameters of the industry-adapted multimodal object detection model. The industry-specific dataset includes a dataset that has been collected from datasets related to a specified industry and then fused.

[0013] Optionally, when performing cross-dataset batch fusion processing, the process includes: randomly selecting N training samples from different datasets to form a training batch, where N is a positive integer; setting a maximum number of negative samples M, and for each training sample, randomly selecting a specified number I of categories from the non-target categories of the corresponding dataset as negative sample categories, and the other categories as positive sample categories; and generating a D-dimensional feature vector from the positive and negative sample categories of each training sample using a text encoder to form I... A D-dimensional category feature matrix; calculate the maximum value S of all training samples within the batch; for training samples that are less than the maximum value S, fill the gap to S using a random vector. D dimensions, ultimately outputting N S The industry-specific dataset is obtained by analyzing D-dimensional batch feature data.

[0014] According to another aspect of the present invention, a detection device based on a multimodal target detection model is also provided, comprising: a data acquisition unit, configured to respond to a regional security monitoring instruction, acquire real-time video data of a target region, and extract image frames from the real-time video data; a detection model acquisition unit, configured to acquire a pre-constructed multimodal target detection model, wherein the multimodal target detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and the video image is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and the feature fusion layer are frozen, and only the projection layer, the type feature representation head, and the target bounding box regression head are trained to adapt to a specified industry distribution; and a target detection unit, configured to input a pre-set industry prompt text and the extracted image frames into the multimodal target detection model, and the multimodal target detection model extracts target object features and outputs target bounding box coordinates to locate the target object position.

[0015] Optionally, the multimodal object detection model includes: a text encoder, which performs text segmentation on the input diverse Chinese tag list and generates semantic vectors based on the text segmentation, and generates a text semantic feature matrix by combining the semantic vector generation results; a visual encoder, which extracts multi-scale features from video images to obtain multi-level image features; a feature fusion layer, which performs cross-modal fusion on the text semantic feature matrix and the image features to output multi-level image features; a projection layer, which uses a multi-layer perceptron structure connected to the text encoder to map the text semantic feature matrix to the object detection space, wherein the multi-layer perceptron structure uses two fully connected layers; a type feature representation layer, which connects to the feature fusion layer and uses a three-layer convolutional network to replace the category prediction branch, which performs convolution processing on each layer of image features output by the feature fusion layer to output a multi-dimensional feature matrix; and a target bounding box regression layer, which outputs the target bounding box coordinates based on the multi-dimensional feature matrix and the text feature matrix output by the object detection space.

[0016] Optionally, the detection device based on the multimodal target detection model, when generating the diversified Chinese label list, includes: an industry target dictionary acquisition unit, used to acquire the original labeled category names and the industry target dictionary, wherein the industry target dictionary includes parent category names, a list of synonyms, and a list of random descriptive phrases; a similar name search unit, used to search for the parent category names, the list of synonyms, and the list of random descriptive phrases corresponding to each original labeled category name from the industry target dictionary; and a phrase replacement unit, used to randomly select a random descriptive phrase and randomly select a placeholder from the original labeled category names, synonyms, parent category names, and parent category names to generate the diversified Chinese label list, wherein the diversified Chinese label list includes positive sample labels and negative sample labels.

[0017] Optionally, the feature fusion layer includes an image enhancement module, which is used to perform cross-modal fusion of the text semantic feature matrix with multi-scale image features at each level, enhance the image feature expression through text semantics, and retain the multi-layer image features after fusion enhancement.

[0018] Optionally, the pre-training stage of the multimodal object detection model includes: an image-text detection dataset acquisition module, used to acquire open-source image-text detection datasets, perform batch Chinese annotation on the open-source image-text detection datasets to obtain Chinese image-text pre-training datasets, and execute a cross-dataset batch processing fusion strategy on the Chinese image-text pre-training datasets; a first freezing module, used to freeze the parameters of the text encoder and train the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation layer, and target box regression layer head during the pre-training stage; and a first training module, used to train the initialized multimodal object detection model using the Chinese image-text pre-training data, so that the initialized multimodal object detection model learns the cross-modal alignment relationship between text and images, establishes text semantic representation capabilities, and obtains the parameters of the pre-trained multimodal object detection model.

[0019] Optionally, the fine-tuning stage of the multimodal object detection model includes: an industry target dictionary retrieval module, used to retrieve an open-source image and text detection dataset and construct an industry target dictionary based on the open-source image and text detection dataset; a second freezing module, used to freeze the parameters of the text encoder, visual encoder, and feature fusion layer during the fine-tuning stage, and only open the parameters of the projection layer, type feature representation layer, and target box regression layer for training; and a second training module, used to train the multimodal object detection model using an industry-specific dataset and the industry target dictionary, wherein the multimodal object detection model performs cross-dataset batch fusion processing on different historical detection datasets in the industry target dictionary, and controls the multimodal object detection model to complete industry data distribution and industry terminology recognition operations, and outputs the parameters of the industry-adapted multimodal object detection model, wherein the industry-specific dataset includes a dataset that has been collected from datasets related to a specified industry and then fused.

[0020] Optionally, the detection device based on the multimodal target detection model, when generating the above, includes: a training sample random selection unit, used to randomly select N training samples from different datasets to form a training batch, where N is a positive integer; a sample category determination unit, used to set a maximum negative sampling number M, and for each training sample, randomly select a specified number I categories from the non-target categories of the corresponding dataset as negative sample categories, and other categories as positive sample categories; and a category feature matrix generation unit, used to generate a D-dimensional feature vector by combining the positive and negative sample categories of each training sample through a text encoder, forming an I-dimensional feature matrix. A D-dimensional category feature matrix; an industry-specific dataset generation unit used to calculate the maximum value S of all training samples within a batch. For training samples that are less than the maximum value S, random vectors are used to fill in the gaps to S. D dimensions, ultimately outputting N S The industry-specific dataset is obtained by analyzing D-dimensional batch feature data.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the detection method based on the multimodal target detection model described above.

[0022] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the detection method based on the multimodal target detection model described above.

[0023] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the detection method based on the multimodal target detection model described in any one of the above embodiments.

[0024] In this disclosure, in response to regional security monitoring instructions, real-time video data of the target area is collected, and image frames are extracted from the real-time video data. A pre-constructed multimodal target detection model is obtained, wherein the multimodal target detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and video images is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution; the pre-set industry prompt text and the extracted image frames are input into the multimodal target detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object position.

[0025] In this disclosure, a bimodal architecture (visual encoder and text encoder) in a multimodal object detection model can be used for multi-scale feature extraction. By combining a feature fusion layer and a type feature representation head, text semantic features and image multi-scale features can be fused to improve the understanding of dense targets. In terms of training strategy, a two-stage training process is adopted. The pre-training stage ensures the model's understanding of general scenes. In the fine-tuning stage, by freezing the parameters of the text encoder and visual encoder, only the projection layer, type feature representation head, and target box regression head are trained. This allows the model to focus on optimizing the bounding box regression accuracy of dense small targets, avoiding the destruction of pre-trained knowledge, reducing training costs, significantly reducing localization errors, and improving detection accuracy. This solves the technical problem of low accuracy in localizing dense target scenes in related technologies. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a detection method based on a multimodal target detection model is shown.

[0028] Figure 2 This is a flowchart of an optional detection method based on a multimodal target detection model according to an embodiment of the present invention;

[0029] Figure 3 This is an architecture diagram of an optional multimodal target detection model according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of an optional industry target dictionary according to an embodiment of the present invention;

[0031] Figure 5 This is an example diagram of processing an optional model batch dataset according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of an optional detection device based on a multimodal target detection model according to an embodiment of the present invention;

[0033] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0037] The Gaussian Error Linear Unit (GELU) is a commonly used non-linear activation function in deep learning. It is used to enhance the non-linear expressive power of neural networks. Compared with ReLU, it has smooth gradient characteristics, which helps with model convergence and performance improvement.

[0038] A multilayer perceptron (MLP) is a fully connected feedforward neural network structure containing at least one hidden layer. In this invention, it is used to construct a projection layer to map text-encoded features to a semantic space that matches image features, thereby achieving cross-modal feature alignment and enhancement.

[0039] The Vision-Language Path Aggregation Network (Vision-Language PAN) is used to fuse textual semantic features with multi-scale image features across modalities. By enhancing the semantic expression of image features, it improves the model's ability to locate open-word targets with fine-grained precision.

[0040] It should be noted that the detection method and apparatus based on the multimodal target detection model in this disclosure can be used in the field of model detection technology to achieve target object detection based on the multimodal target detection model, and can also be used in any field other than the field of model detection technology. In the case of achieving target object detection based on the multimodal target detection model, this disclosure does not limit the application field of the detection method and apparatus based on the multimodal target detection model.

[0041] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) collected in this public disclosure are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data are handled with corresponding access points, allowing users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0042] It should be noted that in this disclosure, customer information is collected and analyzed, and users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision-making results; if the user chooses to reject, the process proceeds to the expert decision-making process.

[0043] The following embodiments of the present invention can be applied to various detection systems / applications / devices based on multimodal target detection models. The present invention is applicable to scenarios involving the intersection of computer vision and natural language processing, security detection, and autonomous driving. Through industry-specific target dictionaries and tag replacement strategies, it can improve the semantic understanding of industry-specific terms and subcategories, thereby enhancing the generalization ability in vertical scenarios.

[0044] This invention achieves efficient unification of historical datasets with different category systems and annotation specifications through a cross-dataset batch processing fusion strategy, reducing data cleaning and training costs, and significantly reducing positioning errors and improving measurement accuracy in scenarios with dense targets, small targets, and multiple targets.

[0045] This invention constructs a multimodal detection model that supports Chinese. By using a text encoder, an industry target dictionary label replacement strategy, and cross-dataset batch processing fusion training, it solves the problems of weak Chinese support, poor industry terminology recognition, difficulty in fusion of multiple datasets, and low accuracy of dense small targets in existing models.

[0046] This invention employs a two-stage training approach (pre-training stage + industry fine-tuning stage) to enhance Chinese language understanding and industry adaptability. The batch processing strategy unifies different dataset formats, significantly improving detection accuracy, generalization ability, and engineering practicality.

[0047] Furthermore, addressing the issue of low data utilization due to the difficulty in directly integrating historical datasets with different category systems and annotation specifications in related technologies, this invention employs a cross-dataset batch processing fusion strategy. This eliminates the need for manual standardization of annotation specifications, directly integrating historical datasets from different category systems and reducing training costs.

[0048] The present invention will now be described in detail with reference to various embodiments.

[0049] Example 1

[0050] According to an embodiment of the present invention, an embodiment of a detection method based on a multimodal target detection model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0051] The detection method based on a multimodal target detection model provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a detection method based on a multimodal target detection model is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1The processor 102 (which may include, but is not limited to, a microprocessor MCU (Microcontroller Unit) or a programmable gate array (FPGA)) is shown as 102a, 102b, ..., 102n. It also includes a memory 104 for storing data and a transmission device 106 for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0052] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0053] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the detection method based on the multimodal target detection model in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned detection method based on the multimodal target detection model. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0055] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0056] Under the aforementioned operating environment, this application provides the following: Figure 2 The detection method shown is based on a multimodal target detection model. Figure 2 This is a flowchart of an optional detection method based on a multimodal target detection model according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps S201 to S203. The present invention will be described in detail below with reference to each implementation step.

[0057] Step S201: In response to the area security monitoring command, collect real-time video data of the target area and extract image frames from the real-time video data.

[0058] Step S202: Obtain a pre-built multimodal object detection model. The multimodal object detection model adopts a two-stage training strategy: In the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between state text and video images is learned using other model architectures to establish basic semantic alignment; In the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution.

[0059] During the pre-training phase, the parameters of the text encoder are frozen, and the trainable parameters of the visual encoder, feature fusion layer, projection layer, type feature representation head, and target bounding box regression head are used to learn the cross-modal alignment relationship between state text and video images, thereby establishing basic semantic alignment capabilities. This helps improve the model's basic understanding of Chinese object detection in general scenarios. During the fine-tuning phase, the parameters of the text encoder, visual encoder, and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target bounding box regression head are trained. This helps the model quickly adapt to specific data distributions in designated industries, reducing industry adaptation costs, while retaining the general semantic knowledge learned during the pre-training phase.

[0060] Optionally, the multimodal object detection model includes: a text encoder, which performs text segmentation on the input diverse Chinese tag list and generates semantic vectors based on the text segmentation, and generates a text semantic feature matrix by combining the semantic vector generation results; a visual encoder, which extracts multi-scale features from video images to obtain multi-level image features; a feature fusion layer, which performs cross-modal fusion on the text semantic feature matrix and image features to output multi-level image features; a projection layer, which uses a multi-layer perceptron structure connected to the text encoder to map the text semantic feature matrix to the object detection space, wherein the multi-layer perceptron structure uses two fully connected layers; a type feature representation layer, which connects to the feature fusion layer and uses a three-layer convolutional network to replace the category prediction branch, which performs convolution processing on each layer of image features output by the feature fusion layer to output a multi-dimensional feature matrix; and a target bounding box regression layer, which outputs the target bounding box coordinates based on the multi-dimensional feature matrix and the text feature matrix output from the object detection space.

[0061] The multimodal object detection model's architecture comprises several cooperating sub-modules. A text encoder receives a diverse list of Chinese tags as input, performs text segmentation, and generates semantic vectors based on the segmentation results. These semantic vectors are then combined to generate a text semantic feature matrix, thus transforming the Chinese text into a vector representation that the model can understand. A visual encoder extracts multi-scale features from video images. Through a multi-level network structure, it obtains image features containing different levels of abstraction, helping to capture multi-dimensional information from local details to global semantics in the image.

[0062] The feature fusion layer receives the text semantic feature matrix and image features, performs cross-modal fusion operations, injects text semantics into the image features, and outputs semantically enhanced multi-layer image features, which can strengthen the fine-grained alignment capability of image features with open-word targets. The projection layer uses a multi-layer perceptron (MLP) structure to connect to the output of the text encoder. This MLP structure uses two fully connected layers to map the text semantic feature matrix to the target detection space that matches the image features, making up for the insufficient optimization of industry corpora caused by freezing the text encoder parameters, while not destroying the original massive knowledge of the language model.

[0063] The type feature representation layer connects to the feature fusion layer, replacing the original category prediction branch with a three-layer convolutional network. This network independently convolves each layer of image features output from the feature fusion layer, outputting a multi-dimensional feature matrix representing the semantic vector of the target category. This facilitates fine-grained matching of open-vocabulary categories. The target bounding box regression layer calculates and outputs the target's bounding box coordinates based on the multi-dimensional feature matrix output from the type feature representation layer and the text feature matrix output from the projection layer. This helps improve localization accuracy in dense or complex scenes by leveraging features with rich semantic information.

[0064] Optionally, the diversified Chinese tag list is obtained through the following methods: obtaining the original tag category names and industry target dictionary, wherein the industry target dictionary includes parent category names, a list of synonyms and aliases, and a list of random descriptive phrases; searching the industry target dictionary for the parent category names, the list of synonyms and aliases, and the list of random descriptive phrases corresponding to each original tag category name; randomly selecting a random descriptive phrase, and randomly selecting a placeholder from the original tag category names, synonyms and aliases, parent category names, and parent category names to replace the phrase, thereby generating a diversified Chinese tag list, wherein the diversified Chinese tag list includes positive sample tags and negative sample tags.

[0065] This embodiment involves the construction process of a diverse Chinese label list, enriching the semantic diversity of training data. Specifically, the original labeled category names and a pre-constructed industry target dictionary are first obtained. The industry target dictionary stores the parent category name, a list of synonyms, and a list of random descriptive phrases corresponding to each target category, establishing semantic relationships between categories and the diversity of expression forms. Further, records matching the original labeled category names of the current training sample are retrieved from the industry target dictionary, and the corresponding parent category name, a list of synonyms, and a list of random descriptive phrases are extracted, expanding the general category labels into a rich semantic set containing hierarchical relationships and multiple expression forms.

[0066] It should be noted that a phrase is randomly selected from the list of random descriptive phrases as the base template, and a name is randomly selected from the candidate set consisting of the original labeled category name, synonyms, parent class name, and parent class name to replace the placeholders in the phrase, generating diverse Chinese labels. This increases the textual expressions the model encounters during training, enhancing its semantic understanding of industry terms and their variations. Optionally, the generated diverse Chinese label list includes positive sample labels and negative sample labels. Positive sample labels represent actual targets in the image, while negative sample labels represent interference categories that are not present in the image. This combination of positive and negative sample labels helps the model more clearly distinguish between targets and non-targets during training, improving detection accuracy.

[0067] In this embodiment of the invention, the diverse labels are manually created based on the characteristics of the dataset and the annotation rules. For example, if a new dataset of "highway cyclist recognition" needs to be introduced as a training set, different descriptions of the target need to be determined according to the data annotation rules and the scenario. At this time, the descriptive phrase can be "There is a target cyclist on the highway".

[0068] Optionally, the step of cross-modal fusion of text semantic feature matrix and image features to output multi-layer image features includes: cross-modal fusion of text semantic feature matrix with multi-scale image features at each level, enhancing image feature expression through text semantics, and retaining the multi-layer image features after fusion enhancement.

[0069] This embodiment relates to the specific process of cross-modal fusion of text semantic feature matrix and image features to generate multi-layer image features. By cross-modal fusion of text semantic feature matrix with multi-scale image features at various levels, text semantic information is introduced to enhance the expressive power of image features, and the multi-layer image features after fusion and enhancement are retained, thereby improving the model's understanding of target semantics during the detection process and enhancing the recognition accuracy of open vocabulary targets.

[0070] Optionally, the pre-training stage of the multimodal object detection model includes: collecting an open-source image-text detection dataset, batch-annotating the open-source image-text detection dataset with Chinese characters to obtain a Chinese image-text pre-training dataset; during the pre-training stage, freezing the parameters of the text encoder, and training the parameters of the open visual encoder, feature fusion layer, projection layer, type feature representation layer, and target box regression layer head; using the Chinese image-text pre-training data to train the initialized multimodal object detection model, which learns the cross-modal alignment relationship between text and image, establishes text semantic representation capabilities, and obtains the parameters of the pre-trained multimodal object detection model.

[0071] In the pre-training phase of the multimodal object detection model, an open-source image-text detection dataset was first collected, and the annotation information in this dataset was batch-translated into Chinese annotations to construct a Chinese image-text pre-training dataset. During the pre-training phase, the parameters of the text encoder were frozen to maintain its stable Chinese semantic representation capability, while the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation layer, and bounding box regression layer were made available to allow the model to learn. The initialized multimodal object detection model was trained using the Chinese image-text pre-training data, enabling the model to learn the cross-modal alignment relationships between state text images and establish basic text semantic representation capabilities, thereby obtaining the parameters of the pre-trained multimodal object detection model.

[0072] Optionally, the fine-tuning stage of the multimodal object detection model includes: retrieving an open-source image and text detection dataset and constructing an industry-specific object dictionary based on the open-source image and text detection dataset; freezing the parameters of the text encoder, visual encoder, and feature fusion layer, and only opening the parameters of the projection layer, type feature representation layer, and object box regression layer for training; training the multimodal object detection model using an industry-specific dataset and an industry-specific object dictionary; performing cross-dataset batch fusion processing on different historical detection datasets in the industry-specific object dictionary; controlling the multimodal object detection model to complete industry data distribution and industry terminology recognition operations; and outputting the parameters of the industry-adapted multimodal object detection model. The industry-specific dataset includes a dataset that has been collected from datasets related to a specified industry and then fused.

[0073] This embodiment describes the specific operation process of a multimodal object detection model in the fine-tuning stage. First, an open-source image and text detection dataset is retrieved. Based on this dataset, an industry-specific target dictionary is constructed to obtain industry-specific terminology and semantic association information. During the fine-tuning stage, the parameters of the text encoder, visual encoder, and feature fusion layer are frozen to maintain the general semantic and visual feature extraction capabilities obtained in the pre-training stage. Simultaneously, only the parameters of the projection layer, type feature representation layer, and target box regression layer are opened for training. The multimodal object detection model is trained using an industry-specific dataset and an industry-specific target dictionary. The multimodal object detection model then performs cross-dataset batch fusion processing on different historical detection datasets in the industry-specific target dictionary, allowing the model to focus on learning the industry data distribution and the recognition features of industry terms. This results in the output of industry-adapted multimodal object detection model parameters, which helps improve the model's detection performance in vertical domains.

[0074] Step S203: Input the pre-set industry prompt text and the extracted image frame into the multimodal target detection model. The multimodal target detection model extracts the target object features and outputs the target bounding box coordinates to locate the target object position.

[0075] The multimodal object detection model extracts features from input state text and real-time video data. A text encoder encodes the state text to generate semantic vectors, while a visual encoder extracts features from video frames to generate multi-scale image features. Subsequently, cross-modal fusion and mapping are performed through a feature fusion layer and a projection layer to extract target object features containing rich semantic information, accurately distinguishing targets from interference in complex video backgrounds. Based on the extracted target object features and combined with text semantic information, the multimodal object detection model performs category matching and bounding box regression calculations, ultimately outputting the target bounding box coordinates. By outputting precise bounding box coordinates, the specific location of the target object in the video frame can be determined, achieving real-time target object localization. This helps meet the real-time and positioning accuracy requirements of scenarios such as security detection and autonomous driving.

[0076] Through the above steps, in response to regional security monitoring instructions, real-time video data of the target area can be collected, and image frames can be extracted from the real-time video data. A pre-constructed multimodal object detection model can be obtained, wherein the multimodal object detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and video images is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution; the pre-set industry prompt text and the extracted image frames are input into the multimodal object detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object position. In this embodiment, multi-scale feature extraction can be performed using a bimodal architecture (visual encoder and text encoder) in a multimodal object detection model. By combining a feature fusion layer and a type feature representation head, text semantic features and image multi-scale features can be fused, improving the ability to understand dense targets. In terms of training strategy, a two-stage training process is adopted. The pre-training stage ensures the model's understanding of general scenes. In the fine-tuning stage, by freezing the parameters of the text encoder and visual encoder, only the projection layer, type feature representation head, and target box regression head are trained, enabling the model to focus on optimizing the bounding box regression accuracy of dense small targets, avoiding the destruction of pre-trained knowledge, reducing training costs, significantly reducing localization errors, and improving detection accuracy. This solves the technical problem of low accuracy in localizing dense target scenes in related technologies.

[0077] To address the technical problems of difficulty in directly integrating historical datasets with different categories and annotation standards for training, and low data utilization in related technologies, this invention adopts a cross-dataset batch processing fusion strategy. This eliminates the need for manual standardization of annotation standards and directly integrates historical datasets from different categories, thereby reducing training costs.

[0078] Specifically, the cross-dataset batch fusion process includes: randomly selecting N training samples from different datasets to form a training batch, where N is a positive integer; setting a maximum negative sampling number M, and for each training sample, randomly selecting a specified number I of categories from the non-target categories of the corresponding dataset as negative sample categories, and the other categories as positive sample categories; and generating a D-dimensional feature vector from the positive and negative sample categories of each training sample using a text encoder to form I... A D-dimensional category feature matrix; calculate the maximum value S of all training samples within the batch; for training samples that are less than the maximum value S, fill the gap to S using a random vector. D dimensions, ultimately outputting N S D-dimensional batch feature data yields an industry-specific dataset.

[0079] N training samples from different datasets are randomly selected to form a training batch, enabling mixed training of samples from different datasets. Then, a maximum number of negative samples M is set. For each training sample, a specified number I of categories are randomly selected from the non-target categories of the corresponding dataset as negative sample categories, and the remaining categories are used as positive sample categories. This sampling method enables the model to distinguish between target and non-target categories during training, thereby enhancing its classification and discrimination capabilities.

[0080] Furthermore, the positive and negative sample categories of each training sample are used by a text encoder to generate a D-dimensional feature vector, forming I... A D-dimensional category feature matrix is ​​generated, containing semantic information for each category in the current batch. Then, the maximum value S of all training samples within the batch is calculated, which determines the maximum number of negative sample categories in the batch. For training samples where the number of negative sample categories is less than the maximum value S, random vectors are used to fill in S. The D dimension is used to unify the dimension of the feature matrix for all samples within a batch. The final output is N. S D-dimensional batch feature data yields industry-specific datasets with a unified format, which helps eliminate training obstacles caused by inconsistencies in category systems between different datasets and enables efficient fusion training of multi-source data.

[0081] The following describes in detail another optional implementation method.

[0082] The embodiments of this invention mainly describe the training method of the multimodal target detection model. It adopts an image-text dual input architecture and achieves efficient and high-precision open vocabulary target detection through multimodal encoder, industry dictionary label replacement, cross-dataset batch processing fusion, and two-stage training strategy.

[0083] Part 1: Architecture Design of Multimodal Target Detection Model

[0084] The multimodal object detection model takes text and image modalities as input data, Figure 3 is an optional architecture diagram of the multimodal object detection model according to the embodiment of the present invention, as Figure 3 shown, the sub-structures of the model include:

[0085] a. Chinese text encoder: A pre-trained model of cn_clip (Chinese Contrastive Language-Image Pre-training) can be adopted, which segments Chinese text and generates semantic vectors. Parameters are frozen in the pre-training stage to ensure the stability of Chinese semantic representation. For example, with the input of ["a red car", "pedestrian"], the encoder will encode the text into an N-dimensional vector, i.e., a 2 dim] matrix, where dim can be preset to 512.

[0086] b. Visual encoder: Extracts multi-scale features of the image, balancing detection accuracy and inference speed.

[0087] c. Feature fusion layer: The Vision-Language PAN structure proposed in yolo-world can be adopted to perform cross-modal fusion of Chinese text features and image features, so as to enhance the expression of image features with text semantics and strengthen the fine-grained alignment capability, and finally only output the image feature layer part (the original structure also performs feature fusion on the text part encoding).

[0088] d. Projection layer: A MLP (Multilayer Perceptron) structure is used to connect the output end of the Chinese text encoder, mapping text features to the object detection space. In the present invention, 2 fully connected layers (Linear[512x256] + GELU Activation + Linear[256x512]) can be used as the MLP layer. This module can update parameters during the training phase to adapt to industry semantic features, which solves the problem that the text encoder parameters are always frozen during training and cannot be optimized for industry corpora, while not damaging the knowledge learned by the language model from massive data.

[0089] e. Type feature representation: A 3-layer convolutional network is applied to each layer of image features output by the feature fusion layer respectively to replace the original class prediction branch of YOLO26. After processing by this structure, each output feature layer is a [H W dim] dimensional feature matrix, which is used to represent the semantic vector of the object class to realize open-vocabulary class matching, where H and W are 32, 16, and 8 times downsampling of the input image size respectively for each layer in the PAN.

[0090] f. Target bounding box regression: Using the YOLO26 bounding box prediction structure, output the coordinates of the target bounding box.

[0091] Part Two: Two-Phase Training Process

[0092] 2.1 Pre-training phase (Chinese language capability building).

[0093] (1) Data preparation: Collect a large-scale English open-source image and text detection dataset, translate it into Chinese annotation in batches, and construct a Chinese image and text pre-training dataset.

[0094] (2) Parameter configuration: Only the parameters of the Chinese text encoder are frozen, while the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation head, and target box regression head are all trainable.

[0095] (3) Training objective: To learn the cross-modal alignment relationship between Chinese text and images, establish basic Chinese semantic representation capabilities, and improve the accuracy of Chinese object detection in general scenarios.

[0096] 2.2 Industry Model Fine-tuning Stage (Vertical Scenario Adaptation).

[0097] (1) Data preparation: Collect industry-specific image and text detection datasets and construct industry target dictionaries. Figure 4 This is a schematic diagram of an optional industry target dictionary according to an embodiment of the present invention, such as... Figure 4 As shown, its directory includes class name, parent class, same name representation, and random phrase representation.

[0098] (2) Construction of industry target dictionary: The dictionary uses the target category as the key and the value contains three parts: parent class name, a list of synonyms, and a list of random descriptive phrases.

[0099] Example:

[0100] {

[0101] "car":{

[0102] Parent class: "Motor Vehicle"

[0103] "Alternative names": ["car", "sedan", "family car"],

[0104] Random phrases: ["This vehicle", "The vehicle on the roadside", "A personal vehicle", "A parked vehicle"]

[0105] }

[0106] }

[0107] (3) Label replacement strategy: During training, random phrases corresponding to the industry target dictionary are randomly selected, and then labels are randomly selected from "this category name, synonym, parent category name, parent category name" to replace xx in the phrases, generating diverse Chinese labels and enhancing the model's ability to generalize industry terms.

[0108] (4) Parameter configuration: Freeze the parameters of Chinese text encoder, visual encoder, and feature fusion layer, and train only the projection layer, type feature representation head, and target box regression head to adapt to industry data distribution.

[0109] Part Three: Cross-Dataset Batch Processing Fusion Training Strategy.

[0110] To address the issues of difficulty in directly integrating historical datasets with different category systems and annotation standards for training, and low data utilization in related technologies, this invention adopts a cross-dataset batch processing fusion strategy. This eliminates the need for manual standardization of annotation standards and directly integrates historical datasets from different category systems, thereby reducing training costs. Figure 5 This is an example diagram of processing an optional model batch dataset according to an embodiment of the present invention, such as... Figure 5 As shown, it includes three types of dataset target types ( Figure 5 The diagram illustrates target types for dataset A (including A_1, A_2, A_3, etc.), dataset B (including B_1, B_2, B_3, B_4, etc.), and dataset C (including C_1, C_2, C_3…C_10, etc.), addressing the issue of inconsistent category systems across different datasets that prevent direct fusion. Further processing of the target dataset's category system is then performed.

[0111] When building a model batch dataset, the following steps may be included:

[0112] a. Batch construction: Randomly select N training samples from different datasets to form a training batch.

[0113] b. Negative sample sampling: Set a maximum number of negative samples M. For each sample, randomly select M_min=min(M, number of categories in the dataset-1) categories from the non-target categories of the corresponding dataset as negative samples to avoid category conflicts.

[0114] c. Feature encoding: For each sample, the positive sample category + M_min negative sample categories are used to generate a dim-dimensional feature vector through a Chinese text encoder, forming a (M_min+1)×dim-dimensional category feature matrix.

[0115] d. Dimensional alignment padding: Calculate the maximum value S of all samples (M_min+1) in the batch. For samples that are less than S, pad them to the S×dim dimension with random vectors. Finally, output the batch feature data in N×S×dim dimensions to achieve unified format across multiple datasets.

[0116] The above implementation methods enable efficient fusion training of multi-source heterogeneous datasets. The proposed cross-dataset batch processing fusion strategy addresses the inconsistencies in category systems and annotation specifications across different historical datasets. By employing negative sample sampling and dimension alignment padding, multi-source data can be directly integrated without the need for manual standardization of annotation formats. This significantly reduces the costs of data cleaning and format conversion, improves the utilization rate of historical data resources, and accelerates the model convergence process.

[0117] This invention utilizes an industry-specific target dictionary and label replacement strategy to rapidly adapt a general visual model to vertical industries such as industrial quality inspection, safety inspection, and autonomous driving. By introducing parent class names, synonyms, and random descriptive phrases, the semantic diversity of the training data is enriched, enabling the model to have stronger robustness and recognition accuracy for industry-specific terms and targets in complex contexts.

[0118] This invention improves detection accuracy and efficiency in complex scenarios. It combines the rapid localization advantage of discriminative models with the strong semantic understanding capabilities of generative models (such as multimodal large models). A general foundation is established during the pre-training phase, and during the fine-tuning phase, the backbone network is frozen, and only the projection layer and regression head are trained. This preserves pre-training knowledge while achieving efficient optimization for specific tasks. This design effectively improves detection accuracy and localization error control in scenarios with dense targets, small targets, and multiple targets, while maintaining inference speed.

[0119] The embodiments of the present invention can reduce the engineering deployment cost and threshold. The model architecture based on the backbone network balances detection accuracy and inference speed, making it suitable for deployment in actual hardware environments. The two-stage training strategy allows the model to achieve high performance without having to be trained from scratch, requiring only a small amount of industry data for fine-tuning. This greatly reduces the application threshold and computing cost of the model in industry implementation.

[0120] The following is a detailed description with reference to another embodiment.

[0121] Example 2

[0122] The detection device based on a multimodal target detection model provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above. The specific implementation method and its beneficial effects can be referred to the aforementioned method embodiment, and will not be repeated here.

[0123] Figure 6 This is a schematic diagram of an optional detection device based on a multimodal target detection model according to an embodiment of the present invention, such as... Figure 6As shown, the detection device based on the multimodal target detection model may include: a data acquisition unit 61, a detection model acquisition unit 62, and a target detection unit 63.

[0124] The data acquisition unit 61 is used to respond to regional security monitoring instructions, acquire real-time video data of the target area, and extract image frames from the real-time video data.

[0125] The detection model acquisition unit 62 is used to acquire a pre-built multimodal object detection model. The multimodal object detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between state text and video images is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the bimodal architecture of the backbone and the feature fusion layer are frozen, and only the projection layer, type feature representation head and target box regression head are trained to adapt to the specified industry distribution.

[0126] The target detection unit 63 is used to input the pre-set industry prompt text and the extracted image frame into the multimodal target detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object position.

[0127] The aforementioned detection device based on a multimodal target detection model can, through the data acquisition unit 61, respond to regional security monitoring commands, acquire real-time video data of the target area, and extract image frames from the real-time video data; through the detection model acquisition unit 62, it acquires a pre-constructed multimodal target detection model, wherein the multimodal target detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and the video image is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution; through the target detection unit 63, the pre-set industry prompt text and the extracted image frames are input into the multimodal target detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object position. In this embodiment, multi-scale feature extraction can be performed using a bimodal architecture (visual encoder and text encoder) in a multimodal object detection model. By combining a feature fusion layer and a type feature representation head, text semantic features and image multi-scale features can be fused, improving the ability to understand dense targets. In terms of training strategy, a two-stage training process is adopted. The pre-training stage ensures the model's understanding of general scenarios. In the fine-tuning stage, by freezing the parameters of the Chinese text encoder and visual encoder, only the projection layer, type feature representation head, and target box regression head are trained. This allows the model to focus on optimizing the bounding box regression accuracy of dense small targets, avoiding the destruction of pre-trained knowledge, reducing training costs, significantly reducing localization errors, and improving detection accuracy. This solves the technical problem of low accuracy in locating dense target scenes in related technologies.

[0128] Optionally, the multimodal object detection model includes: a text encoder, which performs text segmentation on the input diverse Chinese tag list and generates semantic vectors based on the text segmentation, and generates a text semantic feature matrix by combining the semantic vector generation results; a visual encoder, which extracts multi-scale features from video images to obtain multi-level image features; a feature fusion layer, which performs cross-modal fusion on the text semantic feature matrix and image features to output multi-level image features; a projection layer, which uses a multi-layer perceptron structure connected to the text encoder to map the text semantic feature matrix to the object detection space, wherein the multi-layer perceptron structure uses two fully connected layers; a type feature representation layer, which connects to the feature fusion layer and uses a three-layer convolutional network to replace the category prediction branch, which performs convolution processing on each layer of image features output by the feature fusion layer to output a multi-dimensional feature matrix; and a target bounding box regression layer, which outputs the target bounding box coordinates based on the multi-dimensional feature matrix and the text feature matrix output from the object detection space.

[0129] Optionally, the detection device based on the multimodal target detection model, when generating a diversified Chinese label list, includes: an industry target dictionary acquisition unit, used to acquire the original labeled category names and the industry target dictionary, wherein the industry target dictionary includes parent category names, a list of synonyms, and a list of random descriptive phrases; a similar name search unit, used to search for the parent category names, a list of synonyms, and a list of random descriptive phrases corresponding to each original labeled category name from the industry target dictionary; and a phrase replacement unit, used to randomly select a random descriptive phrase and randomly select a placeholder from the original labeled category names, synonyms, parent category names, and parent category names to generate a diversified Chinese label list, wherein the diversified Chinese label list includes positive sample labels and negative sample labels.

[0130] Optionally, the feature fusion layer includes an image enhancement module, which performs cross-modal fusion of the text semantic feature matrix with multi-scale image features at various levels, enhances the image feature expression through text semantics, and retains the multi-layer image features after fusion enhancement.

[0131] Optionally, the pre-training stage of the multimodal object detection model includes: an image-text detection dataset acquisition module, used to acquire open-source image-text detection datasets, batch-annotate the open-source image-text detection datasets with Chinese characters to obtain a Chinese image-text pre-training dataset, and execute a cross-dataset batch processing fusion strategy on the Chinese image-text pre-training dataset; a first freezing module, used to freeze the parameters of the text encoder and open the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation layer, and target box regression layer head for training during the pre-training stage; and a first training module, used to train the initialized multimodal object detection model using the Chinese image-text pre-training data, so that the initialized multimodal object detection model learns the cross-modal alignment relationship between text and image, establishes text semantic representation capabilities, and obtains the parameters of the pre-trained multimodal object detection model.

[0132] Optionally, the fine-tuning stage of the multimodal object detection model includes: an industry object dictionary retrieval module, used to retrieve an open-source image and text detection dataset and construct an industry object dictionary based on the open-source image and text detection dataset; a second freezing module, used to freeze the parameters of the text encoder, visual encoder, and feature fusion layer during the fine-tuning stage, and only open the parameters of the projection layer, type feature representation layer, and target box regression layer for training; and a second training module, used to train the multimodal object detection model using an industry-specific dataset and an industry object dictionary, wherein the multimodal object detection model performs cross-dataset batch fusion processing on different historical detection datasets in the industry object dictionary, and controls the multimodal object detection model to complete industry data distribution and industry terminology recognition operations, and outputs the parameters of the industry-adapted multimodal object detection model, wherein the industry-specific dataset includes a dataset that has been collected from datasets related to a specified industry and then fused.

[0133] Optionally, the detection device based on the multimodal object detection model includes, during generation: a training sample random selection unit, used to randomly select N training samples from different datasets to form a training batch, where N is a positive integer; a sample category determination unit, used to set a maximum negative sampling number M, and for each training sample, randomly select a specified number I categories from the non-target categories of the corresponding dataset as negative sample categories, and other categories as positive sample categories; and a category feature matrix generation unit, used to generate a D-dimensional feature vector from the positive and negative sample categories of each training sample through a text encoder, forming an I-dimensional feature matrix. A D-dimensional category feature matrix; an industry-specific dataset generation unit used to calculate the maximum value S of all training samples within a batch. For training samples that are less than the maximum value S, random vectors are used to fill in the gaps to S. D dimensions, ultimately outputting N S D-dimensional batch feature data yields an industry-specific dataset.

[0134] The detection device based on the multimodal target detection model may also include a processor and a memory. The data acquisition unit 61, the detection model acquisition unit 62, the target detection unit 63, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0135] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters enables the training of a multimodal object detection model and the detection of target objects based on that model.

[0136] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0137] Example 3

[0138] Embodiments of this application may provide an electronic device. Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 (Only one is shown) Processor 702, memory 707, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0139] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the detection method and apparatus based on the multimodal target detection model in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned detection method based on the multimodal target detection model. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The processor can access information and applications stored in memory via a transmission device to execute the following steps: responding to regional security monitoring commands, acquiring real-time video data of the target area, and extracting image frames from the real-time video data; acquiring a pre-built multimodal object detection model, wherein the multimodal object detection model adopts a two-stage training strategy: in the pre-training stage, the parameters of the text encoder are frozen, and the cross-modal alignment relationship between the state text and video images is learned using other model architectures to establish basic semantic alignment; in the fine-tuning stage, the parameters of the backbone bimodal architecture and feature fusion layer are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution; the pre-set industry prompt text and the extracted image frames are input into the multimodal object detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object position.

[0141] Those skilled in the art will understand that Figure 7The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0142] Those skilled in the art will understand that all or part of the steps in the detection methods based on the various multimodal target detection models in the above embodiments can be implemented by a program instructing the hardware of the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0143] Example 4

[0144] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the detection method based on the multimodal target detection model provided in Embodiment 1.

[0145] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the detection method based on any one of the above embodiments of the first embodiment.

[0146] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the detection method based on a multimodal target detection model described in various embodiments of this application.

[0148] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the detection method based on a multimodal target detection model described in various embodiments of this application.

[0149] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0155] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A detection method based on a multimodal target detection model, characterized in that, include: In response to regional security monitoring instructions, real-time video data of the target area is collected, and image frames are extracted from the real-time video data; A pre-constructed multimodal object detection model is obtained, wherein the multimodal object detection model adopts a two-stage training strategy: the parameters of the text encoder are frozen in the pre-training stage, and the cross-modal alignment relationship between state text and video image is learned by using other model architectures to establish basic semantic alignment; During the fine-tuning phase, the parameters of the bimodal architecture and feature fusion layer of the backbone are frozen, and only the projection layer, type feature representation head, and target box regression head are trained to adapt to the specified industry distribution. The pre-defined industry-specific prompt text and the extracted image frame are input into the multimodal target detection model, which extracts the target object features and outputs the target bounding box coordinates to locate the target object.

2. The detection method based on a multimodal target detection model according to claim 1, characterized in that, The multimodal target detection model includes: The text encoder performs text segmentation on the input list of diverse Chinese tags, generates semantic vectors based on the text segmentation, and combines the semantic vector generation results to generate a text semantic feature matrix. A visual encoder is used to extract multi-scale features from video images to obtain multi-level image features. The feature fusion layer performs cross-modal fusion of the text semantic feature matrix and the image features, and outputs multi-layer image features; The projection layer, which uses a multilayer perceptron structure to connect to the text encoder, maps the text semantic feature matrix to the target detection space. The multilayer perceptron structure uses two fully connected layers. The type feature representation layer, connected to the feature fusion layer, uses a three-layer convolutional network to replace the category prediction branch, and is used to perform convolution processing on each layer of image features output by the feature fusion layer to output a multi-dimensional feature matrix. The target bounding box regression layer outputs the coordinates of the target bounding box based on the multi-dimensional feature matrix and the text feature matrix output from the target detection space.

3. The detection method based on a multimodal target detection model according to claim 2, characterized in that, The diverse list of Chinese tags was obtained through the following methods: Obtain the original labeled category name and industry target dictionary, wherein the industry target dictionary includes the parent category name, a list of synonyms, and a list of random descriptive phrases; Search the industry target dictionary for the parent class name, the list of synonyms, and the list of random descriptive phrases corresponding to each original labeled category name; A random descriptive phrase is randomly selected, and a placeholder in the phrase is randomly selected from the original labeled category name, synonym, parent category name, and parent category name to generate the diversified Chinese label list, wherein the diversified Chinese label list includes positive sample labels and negative sample labels.

4. The detection method based on a multimodal target detection model according to claim 2, characterized in that, The steps of performing cross-modal fusion of the text semantic feature matrix and the image features to output multi-layer image features include: The text semantic feature matrix is ​​cross-modal fused with multi-scale image features at various levels to enhance the image feature expression through text semantics, while retaining the multi-layer image features after fusion enhancement.

5. The detection method based on a multimodal target detection model according to claim 1, characterized in that, The pre-training phase of the multimodal target detection model includes: An open-source image and text detection dataset is collected, and Chinese characters are annotated in batches on the open-source image and text detection dataset to obtain a Chinese image and text pre-training dataset. A cross-dataset batch processing and fusion strategy is then performed on the Chinese image and text pre-training dataset. During the pre-training phase, the parameters of the text encoder are frozen, while the parameters of the visual encoder, feature fusion layer, projection layer, type feature representation layer, and target box regression layer are opened for training. The initialized multimodal object detection model is trained using the Chinese image and text pre-training data. The initialized multimodal object detection model learns the cross-modal alignment relationship between text and image, establishes text semantic representation ability, and obtains the parameters of the pre-trained multimodal object detection model.

6. The detection method based on a multimodal target detection model according to claim 5, characterized in that, The fine-tuning phase of the multimodal target detection model includes: Retrieve an open-source image and text detection dataset, and construct an industry target dictionary based on the open-source image and text detection dataset; During the fine-tuning phase, the parameters of the text encoder, visual encoder, and feature fusion layer are frozen, and only the parameters of the projection layer, type feature representation layer, and bounding box regression layer are made available for training. The multimodal object detection model is trained using an industry-specific dataset and the industry target dictionary. The multimodal object detection model performs cross-dataset batch fusion processing on different historical detection datasets in the industry target dictionary. The multimodal object detection model is also controlled to complete industry data distribution and industry terminology recognition operations, and outputs the parameters of the industry-adapted multimodal object detection model. The industry-specific dataset includes a dataset that is a collection of datasets related to a specified industry and then fused.

7. The detection method based on a multimodal target detection model according to claim 6, characterized in that, When performing batch fusion processing across datasets, the following are included: N training samples from different datasets are randomly selected to form a training batch, where N is a positive integer; Set a maximum number of negative samples M. For each training sample, randomly select a specified number I of categories from the non-target categories in the corresponding dataset as negative sample categories, and the other categories as positive sample categories. For each training sample, the positive and negative sample categories are used to generate a D-dimensional feature vector through a text encoder, forming I. D-dimensional category feature matrix; Calculate the maximum value S of all training samples within the batch. For training samples that are less than the maximum value S, fill the gap to S using a random vector. D dimensions, ultimately outputting N S The industry-specific dataset is obtained by analyzing D-dimensional batch feature data.

8. A detection device based on a multimodal target detection model, characterized in that, include: The data acquisition unit is used to respond to regional security monitoring instructions, acquire real-time video data of the target area, and extract image frames from the real-time video data; The detection model acquisition unit is used to acquire a pre-constructed multimodal object detection model, wherein the multimodal object detection model adopts a two-stage training strategy: the pre-training stage freezes the parameters of the text encoder and uses other model architectures to learn the cross-modal alignment relationship between state text and video images to establish basic semantic alignment; the fine-tuning stage freezes the parameters of the backbone bimodal architecture and feature fusion layer, and only trains the projection layer, type feature representation head and target box regression head to adapt to the specified industry distribution. The target detection unit is used to input the pre-set industry prompt text and the extracted image frame into the multimodal target detection model, and the multimodal target detection model extracts the target object features and outputs the target bounding box coordinates to locate the target object position.

9. An electronic device, characterized in that, The device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the detection method based on the multimodal target detection model as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the detection method based on the multimodal target detection model as described in any one of claims 1 to 7.

Images