Target detection method and system based on multi-modal model, electronic device and medium

Abstract

The application discloses a target detection method and system based on a multi-modal model, an electronic device and a medium. The method comprises the following steps: acquiring a to-be-detected image, a depth image and millimeter wave signal data; performing feature mapping on the to-be-detected image and the depth image through a modal mapping module of the multi-modal model to obtain first feature data, and performing a matrix transposition operation on the millimeter wave signal data to obtain second feature data; performing feature fusion operation on the feature data through an adaptive fusion module to obtain fused features; performing feature encoding on the fused features through a global encoder to obtain a first feature map and a second feature map; inputting the first feature map and the second feature map into a target decoder to perform fusion encoding operation, and outputting a category vector and a target box vector; and labeling target objects on the to-be-detected image according to the category vector and the target box vector to obtain a detection result. The application can improve the target recognition accuracy and realize accurate multi-target detection.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a target detection method, system, electronic device and medium based on a multimodal model. Background Technology

[0002] In the field of computer vision, the development of multi-target human detection technology has shifted from a single-modal to a multi-modal approach. This shift stems from the inherent limitations of single-modal human detection in complex application scenarios. Multi-target human detection is a key technology in computer vision, primarily aimed at identifying and accurately locating all human bodies from images or video streams. This technology plays a crucial role in numerous application areas, such as public safety monitoring, crowd behavior analysis, interactive media, and advanced driver assistance systems. However, with technological advancements, single-modal human detection systems, due to their reliance on a single data source, have gradually become unable to meet the growing application demands when facing complex environments and dynamic changes, further complicating the requirements for high accuracy and robustness in target recognition. Summary of the Invention

[0003] This application provides a target detection method, system, electronic device, and storage medium based on a multimodal model, which can improve target recognition accuracy and achieve accurate multi-target detection.

[0004] In a first aspect, embodiments of this application provide a target detection method based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the method includes:

[0005] The system acquires an image to be detected, a depth image, and millimeter-wave signal data. The image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is acquired by a sensor.

[0006] The modality mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image to obtain first feature data, and performs matrix transpose operation on the millimeter wave signal data through the modality mapping module to obtain second feature data;

[0007] The adaptive fusion module performs a feature fusion operation on the first feature data and the second feature data to obtain fused features.

[0008] The fused features are encoded using the global encoder to obtain a first feature map and a second feature map.

[0009] The first feature map and the second feature map are input into the target decoder for fusion encoding, and the class vector and the target bounding box vector are output.

[0010] The target objects in the image to be detected are labeled according to the category vector and the target bounding box vector to obtain the detection result.

[0011] In some embodiments, the step of performing feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model to obtain first feature data includes:

[0012] The image to be detected and the depth image are input into the modality mapping module, so that the modality mapping module performs dimensional transformation operations on the image to be detected and the depth image according to a preset number of operations to obtain the first feature data.

[0013] In some embodiments, the step of performing a matrix transpose operation on the millimeter-wave signal data through the modal mapping module to obtain the second feature data includes:

[0014] The millimeter-wave signal data is processed to obtain the number of point clouds and the length of the feature vectors, wherein the length of the point feature vectors is used to characterize the information dimension contained in each point cloud point;

[0015] The millimeter-wave signal data is input into the modal mapping module, so that the modal mapping module performs a matrix transpose operation on the millimeter-wave signal to continuously increase the dimensionality of the point cloud and the feature vector length, and outputs the second feature data.

[0016] In some embodiments, the adaptive fusion module includes a first branch and a second branch; the step of performing a feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain fused features includes:

[0017] The first feature data and the feature data are input into the adaptive fusion module, so that the first branch performs a weighting operation on the first feature data based on a first preset weight to obtain the first weighted data, and the second branch performs a weighting operation on the second feature data based on a second preset weight to obtain the second weighted data;

[0018] Perform dimensional transformation on the first weighted data and the second weighted data;

[0019] Matrix addition is performed on the first and second weighted data after dimensionality transformation to obtain the fused features.

[0020] In some embodiments, the global encoder includes a third branch and a fourth branch; the step of performing feature encoding on the fused features through the global encoder to obtain a first feature map and a second feature map includes:

[0021] The fused features are input into the global encoder, so that the third branch performs feature encoding on the fused features and outputs a column feature vector, and the fourth branch performs feature encoding on the fused features and outputs a row feature vector;

[0022] The fused features are weighted according to the column feature vector and the row feature vector to obtain a first feature map and a second feature map.

[0023] In some embodiments, the target decoder includes a first fully connected layer and a second fully connected layer; the step of inputting the first feature map and the second feature map into the target decoder for fusion encoding operation, and outputting a class vector and a target bounding box vector, includes:

[0024] The first feature map and the second feature map are input into the target decoder, so that the target decoder performs a feature fusion operation on the first feature map and the second feature map and outputs the fused feature.

[0025] The fused features are then subjected to feature separation to obtain the separated features;

[0026] The separation features are input into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer encode the separation features in sequence and output the category vector and the target box vector.

[0027] In some embodiments, the step of labeling the target objects on the image to be detected based on the category vector and the bounding box vector to obtain the detection result includes:

[0028] The category probability of all objects in the image to be detected is determined based on the category vector, and the image to be detected is labeled based on the target bounding box vector to obtain multiple prediction boxes;

[0029] The target prediction box of the target object is determined based on the category probability, and the detection result is obtained.

[0030] Secondly, embodiments of this application also provide a target detection system based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the system includes:

[0031] The data acquisition module is used to acquire an image to be detected, a depth image, and millimeter-wave signal data. The image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is acquired by a sensor.

[0032] The feature mapping module is used to perform feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model to obtain first feature data, and to perform matrix transpose operation on the millimeter wave signal data through the modality mapping module to obtain second feature data;

[0033] The feature fusion module is used to perform feature fusion operations on the first feature data and the second feature data through the adaptive fusion module to obtain fused features;

[0034] The feature encoding module is used to encode the fused features through the global encoder to obtain a first feature map and a second feature map;

[0035] The fusion encoding module is used to input the first feature map and the second feature map into the target decoder for fusion encoding operation, and output a category vector and a target bounding box vector.

[0036] The result annotation module is used to annotate the target objects on the image to be detected according to the category vector and the target bounding box vector, so as to obtain the detection result.

[0037] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the target detection method based on a multimodal model as described in the first aspect.

[0038] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the target detection method based on a multimodal model as described in the first aspect.

[0039] The target detection method based on a multimodal model provided in this application has at least the following beneficial effects: First, it acquires an image to be detected, a depth image, and millimeter-wave signal data captured by a sensor. The image to be detected includes at least one target object. By acquiring the depth image, the geometry of the visible surface of the target object can be directly determined, and the distance between the target object and the image acquisition device can be determined. By acquiring the millimeter-wave signal data, accurate detection and positioning of the target object can be achieved. Acquiring different types of data facilitates subsequent detection of the target object. Then, the modality mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image to obtain first feature data. The modality mapping module also performs matrix transpose operation on the millimeter-wave signal data to obtain second feature data, realizing the conversion of different data, which facilitates subsequent feature selection and fusion processing, and enhances the feature representation capabilities of the first and second feature data. Since different feature data may contain complementary information, this embodiment uses an adaptive fusion module to perform feature fusion on the first and second feature data, fusing features of different dimensions into a unified scale output feature to obtain fused features. This reduces the number of features the model needs to process, lowers the dimensionality, and thus reduces computational complexity. Then, a global encoder encodes the fused features, which retains important information while reducing feature dimensionality, improving the model's prediction accuracy. This yields a first feature map and a second feature map. The first and second feature maps are then input into a target decoder for fusion encoding, enabling the target decoder to perform a weighted sum based on the importance of the first and second feature maps, outputting a category vector and a target bounding box vector. This achieves accurate detection of item categories and locations, improving detection accuracy. Finally, the target objects on the image to be detected are labeled based on the category vector and target bounding box vector, achieving labeling of different categories of items and obtaining detection results. This results in accurate item detection, labeling the category and location of items and improving the accuracy of item detection.

[0040] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description and the accompanying drawings. Attached Figure Description

[0041] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and form part of the specification. They are used together with the examples of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0042] Figure 1 This is a flowchart illustrating the specific method of the target detection method based on a multimodal model provided in the embodiments of this application;

[0043] Figure 2 This is a flowchart illustrating the specific process of feature mapping of the image to be detected and the depth image using the modality mapping module of the multimodal model provided in this application embodiment.

[0044] Figure 3 This is a flowchart illustrating the specific process of performing matrix transpose on millimeter-wave signal data using a modal mapping module, as provided in this embodiment of the application.

[0045] Figure 4 This is a detailed flowchart of step S103 provided in the embodiments of this application;

[0046] Figure 5 This is a flowchart illustrating step S104 provided in the embodiments of this application;

[0047] Figure 6 This is a detailed flowchart of step S105 provided in the embodiments of this application;

[0048] Figure 7 This is a flowchart illustrating step S106 provided in the embodiments of this application;

[0049] Figure 8 This is a schematic diagram of the structure of the target detection system based on a multimodal model provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the overall framework provided in one example of this application;

[0051] Figure 10 This is a schematic diagram of the modal mapping module provided in an example of this application;

[0052] Figure 11 This is a schematic diagram of the adaptive fusion module provided in an example of this application;

[0053] Figure 12 This is a schematic diagram of a global encoder provided as an example in this application;

[0054] Figure 13 This is a schematic diagram of a target decoder provided as an example in this application;

[0055] Figure 14 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] The target detection method based on a multimodal model provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the above method, but is not limited to the above forms.

[0060] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0061] In the field of computer vision, the development of multi-target human detection technology has shifted from a single-modal to a multi-modal approach. This shift stems from the inherent limitations of single-modal human detection in complex application scenarios. Multi-target human detection is a key technology in computer vision, primarily aimed at identifying and accurately locating all human bodies from images or video streams. This technology plays a crucial role in numerous application areas, such as public safety monitoring, crowd behavior analysis, interactive media, and advanced driver assistance systems. However, with technological advancements, single-modal human detection systems, due to their reliance on a single data source, have gradually become unable to meet the growing application demands when facing complex environments and dynamic changes, further complicating the requirements for high accuracy and robustness in target recognition.

[0062] To address the aforementioned issues, this application provides a target detection method, system, electronic device, and storage medium based on a multimodal model. First, it acquires an image to be detected, a depth image, and millimeter-wave signal data captured by a sensor. The image to be detected includes at least one target object. Acquiring the depth image allows direct determination of the geometry of the visible surface of the target object and the distance between the target object and the image acquisition device. Acquiring the millimeter-wave signal data enables precise detection and positioning of the target object. Acquiring different types of data facilitates subsequent target object detection. Next, the modality mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image to obtain first feature data. Then, the modality mapping module performs a matrix transpose operation on the millimeter-wave signal data to obtain second feature data. This conversion between different data types facilitates subsequent feature selection and fusion processing, while enhancing the feature representation capabilities of both the first and second feature data. Since different feature data may contain complementary information, this embodiment uses an adaptive fusion module to perform feature fusion on the first and second feature data, fusing features of different dimensions into a unified scale output feature to obtain fused features. This reduces the number of features the model needs to process, lowers the dimensionality, and thus reduces computational complexity. Then, a global encoder encodes the fused features, which retains important information while reducing feature dimensionality, improving the model's prediction accuracy. This yields a first feature map and a second feature map. The first and second feature maps are then input into a target decoder for fusion encoding, enabling the target decoder to perform a weighted sum based on the importance of the first and second feature maps, outputting a category vector and a target bounding box vector. This achieves accurate detection of item categories and locations, improving detection accuracy. Finally, the target objects on the image to be detected are labeled based on the category vector and target bounding box vector, achieving labeling of different categories of items and obtaining detection results. This results in accurate item detection, labeling the category and location of items and improving the accuracy of item detection.

[0063] Reference Figure 1 , Figure 1 This is a flowchart illustrating a specific method of the target detection method based on a multimodal model provided in this application. In some embodiments, the target detection method based on a multimodal model includes, but is not limited to, steps S101 to S106.

[0064] It should be noted that the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder.

[0065] Step S101: Acquire the image to be detected, the depth image, and the millimeter-wave signal data.

[0066] It should be noted that the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is obtained by the sensor.

[0067] In step S101 of some embodiments, the image to be detected, the depth image, and the millimeter-wave signal data are acquired. The image to be detected is acquired by an image acquisition device. The depth image can be obtained by a depth sensor, laser scanning technology, etc., and is used to reflect the distance between the target object and the image acquisition device. Specifically, the depth image is an image of the distance (depth) values ​​of each point in the scene acquired by the image acquisition device as pixel values. The millimeter-wave signal data is obtained by a sensor using high-frequency millimeter-wave signals for ranging. This facilitates the subsequent detection of the target object using different images and data, and can simultaneously support the detection of multiple data types, which is convenient for subsequent multimodal target detection.

[0068] It should be noted that the embodiments of this application use millimeter-wave radar as an example for illustration. The millimeter-wave radar emits a series of frequency-modulated signals. These signals are reflected back to the radar after they come into contact with environmental objects. The radar receives the reflected signals through a receiving antenna and measures parameters such as the time delay, frequency, and amplitude of the signals. The received millimeter-wave signal data is then processed.

[0069] It is understood that the target detection method based on the multimodal model in this application embodiment can be applied to public safety monitoring, crowd behavior analysis, interactive media and advanced driver assistance systems, etc., where the target object can be a vehicle, a human body, etc., and this application embodiment does not impose specific limitations.

[0070] In step S102, the modal mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image to obtain the first feature data, and performs matrix transpose operation on the millimeter wave signal data through the modal mapping module to obtain the second feature data.

[0071] In step S102 of some embodiments, the modality mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image, thereby converting the image to be detected and the depth image into feature data to obtain first feature data. This allows the image to be detected and the depth image to be mapped into a more compact feature map. Furthermore, the modality mapping module performs a matrix transpose operation on the millimeter-wave signal data, thereby enabling continuous dimensionality upscaling operations on different dimensions of the millimeter-wave signal data to obtain second feature data, thus enhancing the feature representation capability of the millimeter-wave signal data.

[0072] Step S103: Perform feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain fused features.

[0073] In step S103 of some embodiments, the first feature data and the second feature data are fused by the adaptive fusion module. Different feature data are merged and output to obtain fused features, thereby reducing the impact of outliers and noise, improving the prediction accuracy of the model, and reducing the number of features that the model needs to process, reducing dimensionality, and thus reducing computational complexity.

[0074] Step S104: The fused features are encoded using a global encoder to obtain a first feature map and a second feature map.

[0075] In step S104 of some embodiments, the fused features are feature-encoded by a global encoder to measure the importance of different branch feature maps and obtain a first feature map and a second feature map. By focusing on the most influential features, the prediction accuracy of the model can be improved, and weighted feature selection helps to improve the interpretability of the model and improve the effect of feature fusion.

[0076] Step S105: Input the first feature map and the second feature map into the target decoder for fusion encoding operation, and output the category vector and the target bounding box vector.

[0077] In step S105 of some embodiments, the first feature map and the second feature map are input into the target decoder for fusion encoding operation, which can encode the target feature vector one by one. Finally, after two parallel fully connected operations, the category vector and the target box vector are output respectively, thereby determining the object category and the number of objects of the target object, which is convenient for subsequent annotation of the target objects on the image to be detected.

[0078] Step S106: Label the target objects on the image to be detected according to the category vector and the target bounding box vector to obtain the detection result.

[0079] In step S106 of some embodiments, the target object on the image to be detected is labeled according to the category vector and the target box vector, thereby labeling the position of the target object and obtaining multiple prediction boxes, realizing the accurate standard of the target object, accurately labeling the type and position of the target object, and improving the labeling accuracy.

[0080] In some embodiments of this application, a multimodal model is also trained. During the training process, the target classification task is learned using the cross-entropy loss function, as follows:

[0081]

[0082] Among them, y i and These represent the true label and the predicted probability, respectively.

[0083] In the bounding box regression task, this application embodiment uses the Mean Squared Error loss function, as shown below:

[0084]

[0085] Among them, L i and Let x and y represent the vectors of the i-th coordinate points in the ground truth bounding box and the predicted bounding box, respectively. i ,y i ).

[0086] Reference Figure 2 , Figure 2 This is a flowchart illustrating the specific process of feature mapping of the image to be detected and the depth image using a modality mapping module based on a multimodal model, as provided in this application embodiment. In some embodiments, the method includes, but is not limited to, step S201.

[0087] It is understood that the modality mapping module in the embodiments of this application includes a 1*1 convolutional layer and a 3*3 convolutional layer.

[0088] Step S201: Input the image to be detected and the depth image into the modality mapping module so that the modality mapping module performs dimensional transformation operations on the image to be detected and the depth image according to a preset number of operations to obtain the first feature data.

[0089] In step S201 of some embodiments, during the feature mapping process of the modal mapping module of the multimodal model for the image to be detected and the depth image, the modal mapping module is mainly responsible for converting the original data into feature data to facilitate subsequent feature selection and fusion processing. In this embodiment, the image to be detected and the depth image are input into the modal mapping module, so that the modal mapping module performs dimensionality transformation operations on the image to be detected and the depth image according to a preset number of operations. Specifically, the modal mapping module first performs downsampling operations on the image to be detected and the depth image, and then sequentially performs convolution operations on the downsampled image to be detected and the depth image through 1*1 convolutional layers and 3*3 convolutional layers to continuously reduce and increase the dimensionality of the row and column lengths and the number of channels of the image to be detected and the depth image, obtaining the first feature data, thereby enabling the image to be detected and the depth image to be mapped into a more compact feature map.

[0090] It should be noted that the preset number of operations in the embodiments of this application can be set according to the user's needs. For example, five sets of repeated operations can be set, that is, the downsampling operation and the convolution operation can be repeated five times.

[0091] It is understandable that during the dimensional transformation operation on the image to be detected and the depth image, the dimensions of the image to be detected and the depth image are taken as W. i ×H i Taking ×C as an example, the modality mapping module includes downsampling operations (multiplied by 2), 1*1 convolutional layers (containing 1*1 convolution operations, batch normalization, and ReLU activation), and 3*3 convolutional layers (containing 3*3 convolution operations, batch normalization, and ReLU activation). Through five sets of repeated operations, the row and column lengths and number of channels of the original data can be continuously reduced and increased in dimensionality, mapping to a more compact feature map. The output dimension is: (W i / 32×H i / 32)×32C.

[0092] Reference Figure 3 , Figure 3 This is a flowchart illustrating the matrix transpose operation on millimeter-wave signal data using a modal mapping module, as provided in this application embodiment. In some embodiments, the method includes, but is not limited to, steps S301 to S302.

[0093] Step S301: Perform signal processing on the millimeter-wave signal data to obtain the number of point clouds and the length of the feature vectors. The length of the point feature vectors is used to characterize the information dimension contained in each point cloud point.

[0094] Step S302: Input the millimeter-wave signal data into the modal mapping module so that the modal mapping module performs matrix transpose operation on the millimeter-wave signal to continuously increase the dimensionality of the point cloud and the feature vector length, and outputs the second feature data.

[0095] In steps S301 to S302 of some embodiments, during the matrix transpose operation of the millimeter-wave signal data through the modal mapping module, this embodiment first performs signal processing on the millimeter-wave signal data. By processing the received millimeter-wave signal data, information such as the distance, velocity, and angle of the object is calculated. Then, this information is converted into point cloud data to obtain the number of point clouds and the feature vector length. The number of point clouds refers to the total number of target points detected in the point cloud data. Each target point is composed of information such as distance, velocity, and angle, and these points together constitute the three-dimensional model of the target object. The feature vector length is used to characterize the information dimension contained in each point cloud point. Specifically, the longer the feature vector length, the richer the information contained in each point cloud point, and the more accurately the characteristics of the target object can be described. The millimeter-wave signal data is then input into the modal mapping module so that the modal mapping module performs a matrix transpose operation on the millimeter-wave signal, thereby enabling continuous dimensionality increase operations on the two dimensions of the number of point clouds and the feature vector length, outputting second feature data, and realizing the enhancement of the feature representation capability of the millimeter-wave signal data.

[0096] Understandably, the modality mapping module also includes three sets of alternating 1x1 convolutional layers and matrix transpose operations to process millimeter-wave signal data.

[0097] Reference Figure 4 , Figure 4 This is a flowchart illustrating step S103 provided in an embodiment of this application. In some embodiments, step S103 may include, but is not limited to, steps S401 and S403.

[0098] It should be noted that the adaptive fusion module includes a first branch and a second branch, and the adaptive fusion module assigns a selective weight to the input features of each branch to measure the importance of the features.

[0099] Step S401: Input the first feature data and feature data into the adaptive fusion module, so that the first branch performs a weighting operation on the first feature data based on the first preset weight to obtain the first weighted data, and the second branch performs a weighting operation on the second feature data based on the second preset weight to obtain the second weighted data.

[0100] Step S402: Perform dimensional transformation on the first weighted data and the second weighted data.

[0101] Step S403: Perform matrix addition on the first weighted data and the second weighted data after dimensional transformation to obtain the fused features.

[0102] In steps S401 to S403 of some embodiments, during the feature fusion operation of the first feature data and the second feature data by the adaptive fusion module, the adaptive fusion module is mainly responsible for receiving input features from multiple branches and fusing features of different dimensions into output features of a unified scale. In this embodiment, the first feature data and the second feature data are input into the adaptive fusion module. The first branch performs a weighted operation on the first feature data based on a first preset weight, and the second branch performs a weighted operation on the second feature data based on a second preset weight, resulting in second weighted data. This allows for the measurement of the importance of the first and second feature data. The first and second weighted data are then subjected to dimensionality transformation. Specifically, a set of simplified 1*1 convolutional layers and matrix transpose operations convert the feature dimensions of the first and second weighted data into a unified feature dimension. Since different features may behave differently under different conditions, this embodiment performs matrix addition on the dimensionally transformed first and second weighted data. Specifically, matrix addition merges and outputs different feature data to obtain fused features, thereby reducing the impact of outliers and noise, improving the model's prediction accuracy, and reducing the number of features the model needs to process, thus reducing dimensionality and computational complexity.

[0103] Reference Figure 5 , Figure 5 This is a flowchart illustrating step S104 as provided in an embodiment of this application. In some embodiments, step S104 includes, but is not limited to, steps S501 and S502.

[0104] It should be noted that the global encoder includes a third branch and a fourth branch. Specifically, the first branch mainly contains W. a ×1 convolutional layer (i.e., W) a The second branch mainly contains 1×H convolution operations, batch normalization operations, and ReLU activation operations) and fully connected layers (fully connected operations, batch normalization operations, and Sigmoid activation operations). a Convolutional layer (i.e., 1×H) a Convolutional operations, batch normalization operations, and ReLU activation operations) and fully connected layers (fully connected operations, batch normalization operations, and Sigmoid activation operations).

[0105] Step S501: Input the fused features into the global encoder so that the third branch encodes the fused features and outputs column feature vectors, and the fourth branch encodes the fused features and outputs row feature vectors.

[0106] Step S502: Perform a weighted operation on the fused features based on the column feature vector and the row feature vector to obtain the first feature map and the second feature map.

[0107] In steps S501 to S502 of some embodiments, during the feature encoding process of the fused features by the global encoder, the global encoder is mainly responsible for encoding the fused features, further improving the feature representation capability. In this embodiment, the fused features are input into the global encoder so that the third branch of the global encoder performs feature encoding on the fused features. Specifically, W is encoded through the third branch. a The process involves a 1×1 convolution operation, batch normalization, and ReLU activation, followed by fully connected layer operations, batch normalization, and Sigmoid activation to output a column feature vector. This facilitates subsequent column feature weighting operations, and the fourth branch encodes the fused features. Specifically, the fourth branch performs a 1×H... a Convolutional operations, batch normalization operations, and ReLU activation operations are performed, followed by fully connected layer operations, batch normalization operations, and Sigmoid activation operations to output row feature vectors, facilitating subsequent row feature weighting operations. Then, the fused features are weighted based on the column feature vectors and row feature vectors. Specifically, in this embodiment, the fused features are weighted using preset selective weights to measure the importance of different branch feature maps, resulting in a first feature map and a second feature map. By focusing on the most influential features, the prediction accuracy of the model can be improved, and the weighted feature selection helps to improve the interpretability of the model and enhances the effect of feature fusion.

[0108] Reference Figure 6 , Figure 6 This is a flowchart illustrating step S105 provided in an embodiment of this application. In some embodiments, step S105 includes, but is not limited to, steps S601 and S603.

[0109] It should be noted that the target decoder includes a first fully connected layer and a second fully connected layer, which are connected in series.

[0110] Step S601: Input the first feature map and the second feature map into the target decoder so that the target decoder performs feature fusion operation on the first feature map and the second feature map and outputs fused features.

[0111] Step S602: Perform feature separation on the fused features to obtain the separated features.

[0112] Step S603: Input the separation features into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer encode the separation features in sequence and output the category vector and the target box vector.

[0113] In steps S601 to S603 of some embodiments, during the process of inputting the first feature map and the second feature map into the target decoder for fusion encoding, the target decoder is mainly responsible for performing the final fusion and feature encoding operation on the feature maps of different branches. In this embodiment, the first feature map and the second feature map are input into the target decoder so that the target decoder performs feature fusion operation on the first feature map and the second feature map and outputs fused features. Specifically, the feature fusion operation on the first feature map and the second feature map in this embodiment is consistent with the feature fusion operation in step S103, and will not be repeated here. Then, the fused features are separated to obtain separated features. Afterwards, the separated features are sequentially input into the first fully connected layer and the second fully connected layer so that the first and second fully connected layers sequentially encode the separated features, outputting a category vector and a target bounding box vector. This allows the target feature vector to be encoded one by one. Finally, after two parallel fully connected operations, the category vector and the target bounding box vector are output respectively, thereby determining the object category and the number of objects of the target object, facilitating subsequent annotation of the target object on the image to be detected.

[0114] Reference Figure 7 , Figure 7 This is a flowchart illustrating step S106 provided in an embodiment of this application. In some embodiments, step S106 includes, but is not limited to, steps S701 and S702.

[0115] Step S701: Determine the class probability of all objects in the image to be detected based on the class vector, and label the image to be detected based on the target box vector to obtain multiple prediction boxes.

[0116] Step S702: Determine the target prediction box of the target object based on the category probability to obtain the detection result.

[0117] In steps S701 to S702 of some embodiments, during the process of labeling target objects on the image to be detected according to the category vector and the target bounding box vector, the embodiments of this application first determine the category probability of all objects on the image to be detected according to the category vector. Taking the target object as a human body as an example, the category vector at this time represents the probability vector of human targets and non-human targets. The image to be detected is then labeled according to the target bounding box vector, thereby labeling the position of the target object and obtaining multiple prediction boxes. This achieves a precise standard for the target object, accurately labels the type and position of the target object, and improves the labeling accuracy.

[0118] Please see Figure 8 This application also provides a target detection system based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the system can implement the above-mentioned target detection method based on a multimodal model, and includes:

[0119] The data acquisition module 801 is used to acquire the image to be detected, the depth image, and the millimeter-wave signal data. The image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is acquired by a sensor.

[0120] The feature mapping module 802 is used to perform feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain the first feature data, and to perform matrix transpose operation on the millimeter wave signal data through the modal mapping module to obtain the second feature data;

[0121] The feature fusion module 803 is used to perform feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain fused features;

[0122] The feature encoding module 804 is used to encode the fused features through a global encoder to obtain a first feature map and a second feature map.

[0123] The fusion encoding module 805 is used to input the first feature map and the second feature map into the target decoder for fusion encoding operation, and output the category vector and the target bounding box vector.

[0124] The result annotation module 806 is used to annotate the target objects on the image to be detected based on the category vector and the target bounding box vector, so as to obtain the detection result.

[0125] The target detection system based on a multimodal model in this application is used to execute the target detection method based on a multimodal model in the above embodiments. Its specific processing procedure is the same as that of the target detection method based on a multimodal model in the above embodiments, and will not be described in detail here.

[0126] To more clearly illustrate the target detection method based on a multimodal model in the embodiments of this application, specific examples are provided below.

[0127] Example 1:

[0128] Please see Figure 9 , Figure 9 This is a schematic diagram of the overall framework provided in one example of this application.

[0129] Example 1 uses the human body as an example to illustrate the concept.

[0130] like Figure 9 As shown, this embodiment is designed as an end-to-end multimodal neural network model, composed of various types of network modules stacked together. Functionally, these modules can be categorized into a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder. This model can receive data from different modalities, including RGB images (W). i ×H i ×3), depth image (W) i ×H i ×1) and millimeter-wave signal data (W m ×H m ), where W i and H i W represents the row and column length of the image. m and H m This represents the number of point clouds and the length of feature vectors in the millimeter-wave signal data. After a series of calculations, the model can output the bounding box positions (represented by dashed boxes) of multiple target human bodies. Each functional module will be described in detail below.

[0131] Please see Figure 10 , Figure 10 This is a schematic diagram of the modal mapping module provided in an example of this application.

[0132] In some embodiments, the modality mapping module includes two mapping sub-modules, denoted as modality mapping module a and modality mapping module b. The modality mapping module is primarily responsible for converting raw data into feature data to facilitate subsequent feature selection and fusion processing. Modality mapping module a is mainly used to process image data (W... i ×H i ×C), including RGB images and depth images. This module includes downsampling operations (multiplied by 2), 11 convolutional layers (containing 1*1 convolution operations, batch normalization operations, and ReLU activation operations), and 3*3 convolutional layers (containing 3*3 convolution operations, batch normalization operations, and ReLU activation operations). Through 5 sets of repeated operations, the row and column lengths and number of channels of the original data can be continuously reduced and increased in dimensionality, mapped to a more compact feature map, with an output dimension of: (W i / 32×H i / 32)×32C. The modal mapping module b is primarily responsible for processing millimeter-wave signal data (W... m ×H m This module contains three sets of alternating 1x1 convolutional layers and matrix transpose operations. Its purpose is to continuously increase the dimensionality of the original data in terms of both the number of point clouds and the length of feature vectors, thereby enhancing the feature representation capability and ultimately outputting 8W. m ×8H m Feature data.

[0133] Please see Figure 11 , Figure 11 This is a schematic diagram of the adaptive fusion module provided in an example of this application.

[0134] In some embodiments, the adaptive fusion module is primarily responsible for receiving input features from multiple branches and fusing features of different dimensions into a single output feature with a unified scale. Taking two branches as an example, the input features of each branch are assigned a selective weight α to measure the importance of that feature. The weighted features are then transformed into a unified feature dimension W through a set of alternating 1x1 convolutional layers and matrix transpose operations. a ×H a Finally, matrix addition is used to merge the different feature data and output the result. Our method includes a total of four branches of fusion operations (such as...). Figure 9 As shown, each branch outputs a feature map at a different scale: W a ×H a W a / 2×H a / 2、W a / 4×H a / 4、W a / 8×H a / 8).

[0135] Please see Figure 12 , Figure 12 This is a schematic diagram of a global encoder provided in an example of this application.

[0136] In some embodiments, the global encoder is primarily responsible for encoding the fused features, further improving the feature representation capability. The global encoder mainly comprises two branches of feature encoding operations. The first branch primarily includes W... a ×1 convolutional layer (i.e., W) a A 1×1 convolutional operation, batch normalization operation, and ReLU activation operation) and a fully connected layer (fully connected operation, batch normalization operation, and Sigmoid activation operation) can generate a column feature vector (1×H). a The first branch is used to perform column feature weighting on the fused feature map. The second branch mainly contains 1×H. a Convolutional layer (i.e., 1×H) a Convolutional operations, batch normalization, and ReLU activation operations, along with fully connected layers (fully connected operations, batch normalization, and Sigmoid activation operations), can generate a row feature vector (W). a ×1), used to weight the row features of the fused feature map. Finally, embodiments of this application introduce selective weights to measure the importance of different branch feature maps and sum them in a weighted manner.

[0137] Please see Figure 13 , Figure 13 This is a schematic diagram of a target decoder provided in an example of this application.

[0138] In some embodiments, the target decoder is primarily responsible for performing a final fusion and feature encoding operation on the feature maps of different branches, generating a class vector and a bounding box vector. The fusion operation here is consistent with the adaptive fusion module. First, an N×C fusion feature is generated, where N represents the maximum number of human targets that can be detected, and C represents the feature vector length of each human target. Then, through a separation operation and two cascaded fully connected layers, the human target feature vectors are encoded one by one. Finally, after two parallel fully connected operations, the class vector (1×2) and the bounding box vector (1×4) are output respectively. The class vector represents the probability vector between human targets and non-human targets, and the bounding box vector represents the position of the predicted target box, including the top-left corner coordinates (x1, y1) and the bottom-right corner coordinates (x2, y2).

[0139] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, is used by the processor to perform the target detection method based on a multimodal model in the above embodiments of this application.

[0140] Please see Figure 14 , Figure 14 The illustration shows the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0141] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0142] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 using the multimodal model-based target detection method of the embodiments of this application.

[0143] The input / output interface 903 is used to implement information input and output;

[0144] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0145] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0146] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0147] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target detection method based on a multimodal model.

[0148] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0150] It will be understood by those skilled in the art that Figure 1-14 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0151] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0153] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0154] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0155] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A target detection method based on a multimodal model, characterized in that, The multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the method includes: The system acquires an image to be detected, a depth image, and millimeter-wave signal data. The image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is acquired by a sensor. The modality mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image to obtain first feature data, and performs matrix transpose operation on the millimeter wave signal data through the modality mapping module to obtain second feature data; The adaptive fusion module performs a feature fusion operation on the first feature data and the second feature data to obtain fused features. The fused features are encoded using the global encoder to obtain a first feature map and a second feature map. The first feature map and the second feature map are input into the target decoder for fusion encoding, and the class vector and the target bounding box vector are output. The target objects in the image to be detected are labeled according to the category vector and the target bounding box vector to obtain the detection result; The adaptive fusion module includes a first branch and a second branch; the step of performing feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain fused features includes: The first feature data and the feature data are input into the adaptive fusion module, so that the first branch performs a weighting operation on the first feature data based on a first preset weight to obtain the first weighted data, and the second branch performs a weighting operation on the second feature data based on a second preset weight to obtain the second weighted data; Perform dimensional transformation on the first weighted data and the second weighted data; Perform matrix addition on the first and second weighted data after dimensionality transformation to obtain the fused features; The global encoder includes a third branch and a fourth branch; the step of performing feature encoding on the fused features through the global encoder to obtain a first feature map and a second feature map includes: The fused features are input into the global encoder, so that the third branch performs feature encoding on the fused features and outputs a column feature vector, and the fourth branch performs feature encoding on the fused features and outputs a row feature vector; The first feature map and the second feature map are obtained by weighting the column features of the fused feature map according to the column feature vector and by weighting the row features of the fused feature map according to the row feature vector.

2. The target detection method based on a multimodal model according to claim 1, characterized in that, The first feature data is obtained by performing feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model, including: The image to be detected and the depth image are input into the modality mapping module, so that the modality mapping module performs dimensional transformation operations on the image to be detected and the depth image according to a preset number of operations to obtain the first feature data.

3. The target detection method based on a multimodal model according to claim 1, characterized in that, The step of performing a matrix transpose operation on the millimeter-wave signal data through the modal mapping module to obtain the second feature data includes: The millimeter-wave signal data is processed to obtain the number of point clouds and the length of the feature vector, wherein the length of the feature vector is used to characterize the information dimension contained in each point cloud point; The millimeter-wave signal data is input into the modal mapping module, so that the modal mapping module performs a matrix transpose operation on the millimeter-wave signal to continuously increase the dimensionality of the point cloud and the feature vector length, and outputs the second feature data.

4. The target detection method based on a multimodal model according to claim 1, characterized in that, The target decoder includes a first fully connected layer and a second fully connected layer; the step of inputting the first feature map and the second feature map into the target decoder for fusion encoding operation, and outputting a class vector and a target bounding box vector, includes: The first feature map and the second feature map are input into the target decoder, so that the target decoder performs a feature fusion operation on the first feature map and the second feature map and outputs the fused feature. The fused features are then subjected to feature separation to obtain the separated features; The separation features are input into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer encode the separation features in sequence and output the category vector and the target box vector.

5. The target detection method based on a multimodal model according to claim 1, characterized in that, The step of labeling the target objects in the image to be detected based on the category vector and the target bounding box vector to obtain the detection result includes: The category probability of all objects in the image to be detected is determined based on the category vector, and the image to be detected is labeled based on the target bounding box vector to obtain multiple prediction boxes; The target prediction box of the target object is determined based on the category probability, and the detection result is obtained.

6. A target detection system based on a multimodal model, characterized in that, The multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the system includes: The data acquisition module is used to acquire an image to be detected, a depth image, and millimeter-wave signal data. The image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter-wave signal data is acquired by a sensor. The feature mapping module is used to perform feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model to obtain first feature data, and to perform matrix transpose operation on the millimeter wave signal data through the modality mapping module to obtain second feature data; The feature fusion module is used to perform feature fusion operations on the first feature data and the second feature data through the adaptive fusion module to obtain fused features; The feature encoding module is used to encode the fused features through the global encoder to obtain a first feature map and a second feature map; The fusion encoding module is used to input the first feature map and the second feature map into the target decoder for fusion encoding operation, and output a category vector and a target bounding box vector. The result annotation module is used to annotate the target objects on the image to be detected according to the category vector and the target bounding box vector, so as to obtain the detection result; The adaptive fusion module includes a first branch and a second branch; the step of performing feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain fused features includes: The first feature data and the feature data are input into the adaptive fusion module, so that the first branch performs a weighting operation on the first feature data based on a first preset weight to obtain the first weighted data, and the second branch performs a weighting operation on the second feature data based on a second preset weight to obtain the second weighted data; Perform dimensional transformation on the first weighted data and the second weighted data; Perform matrix addition on the first and second weighted data after dimensionality transformation to obtain the fused features; The global encoder includes a third branch and a fourth branch; the step of performing feature encoding on the fused features through the global encoder to obtain a first feature map and a second feature map includes: The fused features are input into the global encoder, so that the third branch performs feature encoding on the fused features and outputs a column feature vector, and the fourth branch performs feature encoding on the fused features and outputs a row feature vector; The fused features are weighted according to the column feature vector and the row feature vector to obtain a first feature map and a second feature map.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the target detection method based on a multimodal model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the target detection method based on a multimodal model as described in any one of claims 1 to 5.

Images