Target detection method and system based on multi-modal model, electronic equipment and medium

Abstract

The invention discloses a multi-modal model-based target detection method and system, electronic equipment and a medium. The method comprises the steps of obtaining a to-be-detected image, a depth image and millimeter wave signal data; performing feature mapping on the to-be-detected image and the depth image through a modal mapping module of a multi-modal model to obtain first feature data, and performing matrix transposition operation on the millimeter wave signal data to obtain second feature data; performing feature fusion operation on the feature data through an adaptive fusion module to obtain fused features; performing feature coding on the fusion features through a global encoder to obtain a first feature map and a second feature map; inputting the first feature map and the second feature map into a target decoder for fusion coding operation, and outputting a category vector and a target frame vector; and marking the target object on the to-be-detected image according to the category vector and the target frame vector to obtain a detection result. According to the embodiment of the invention, the target recognition precision can be improved, and accurate multi-target detection is realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a target detection method, system, electronic device and medium based on a multimodal model. Background Art

[0002] In the field of computer vision, the development of multi-target human detection technology has shifted from single modality to multi-modality. This shift is due to some inherent deficiencies of single-modality human detection in the face of complex application scenarios. Multi-target human detection is a key technology in the field of computer vision. Its main purpose is to identify and accurately locate the positions of all human bodies from images or video streams. This technology plays an important role in many application fields, such as public safety monitoring, crowd behavior analysis, interactive media, and advanced driver assistance systems. However, with the development of technology, single-modality human detection systems have gradually failed to meet the growing application needs when facing complex environments and dynamic changes due to their dependence on a single data source. It is further difficult to meet the requirements for high accuracy and high robustness of target recognition. Summary of the invention

[0003] The embodiments of the present application provide a target detection method, system, electronic device and storage medium based on a multimodal model, which can improve target recognition accuracy and achieve accurate multi-target detection.

[0004] In a first aspect, an embodiment of the present application provides a target detection method based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the method includes:

[0005] Acquire an image to be detected, a depth image, and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and an image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0006] Performing feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and performing a matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data;

[0007] Performing a feature fusion operation on the first feature data and the second feature data by the adaptive fusion module to obtain a fusion feature;

[0008] Performing feature encoding on the fused features through the global encoder to obtain a first feature map and a second feature map;

[0009] Inputting the first feature map and the second feature map into the target decoder for fusion encoding operation, and outputting a category vector and a target box vector;

[0010] The target object on the image to be detected is marked according to the category vector and the target frame vector to obtain a detection result.

[0011] In some embodiments, the performing feature mapping on the image to be detected and the depth image by the modality mapping module of the multimodal model to obtain first feature data includes:

[0012] The image to be detected and the depth image are input into the modality mapping module, so that the modality mapping module performs a dimension transformation operation on the image to be detected and the depth image according to a preset number of operations to obtain first feature data.

[0013] In some embodiments, performing a matrix transposition operation on the millimeter wave signal data by the modal mapping module to obtain the second feature data includes:

[0014] Performing signal processing on the millimeter wave signal data to obtain the number of point clouds and the length of feature vectors, wherein the length of the point feature vector is used to characterize the information dimension contained in each point cloud point;

[0015] The millimeter wave signal data is input into the modal mapping module, so that the modal mapping module performs a matrix transposition operation on the millimeter wave signal to continuously increase the dimension of the point cloud quantity and the feature vector length, and outputs second feature data.

[0016] In some embodiments, the adaptive fusion module includes a first branch and a second branch; the step of performing a feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain a fusion feature includes:

[0017] Inputting the first feature data and the feature data into an adaptive fusion module, so that the first branch performs a weighted operation on the first feature data based on a first preset weight to obtain first weighted data, and the second branch performs a weighted operation on the second feature data based on a second preset weight to obtain second weighted data;

[0018] Performing dimension conversion on the first weighted data and the second weighted data;

[0019] A matrix addition operation is performed on the first weighted data and the second weighted data after dimension conversion to obtain a fusion feature.

[0020] In some embodiments, the global encoder includes a third branch and a fourth branch; and the feature encoding of the fused feature by the global encoder to obtain a first feature map and a second feature map includes:

[0021] Input the fused feature into the global encoder, so that the third branch performs feature encoding on the fused feature and outputs a column feature vector, and the fourth branch performs feature encoding on the fused feature and outputs a row feature vector;

[0022] A weighted operation is performed on the fused features according to the column feature vector and the row feature vector to obtain a first feature map and a second feature map.

[0023] In some embodiments, the target decoder includes a first fully connected layer and a second fully connected layer; the step of inputting the first feature map and the second feature map into the target decoder for a fusion encoding operation and outputting a category vector and a target box vector includes:

[0024] Inputting the first feature map and the second feature map into the target decoder, so that the target decoder performs a feature fusion operation on the first feature map and the second feature map, and outputs a fusion feature;

[0025] Performing feature separation on the fused features to obtain separated features;

[0026] The separation feature is input into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer sequentially perform encoding operations on the separation feature and output a category vector and a target box vector.

[0027] In some embodiments, the step of labeling the target object on the image to be detected according to the category vector and the target frame vector to obtain a detection result includes:

[0028] Determine the category probabilities of all objects on the image to be detected according to the category vector, and annotate the image to be detected according to the target frame vector to obtain multiple prediction frames;

[0029] A target prediction box of the target object is determined according to the category probability to obtain a detection result.

[0030] In a second aspect, an embodiment of the present application further provides a target detection system based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder and a target decoder; the system includes:

[0031] A data acquisition module, used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0032] A feature mapping module, configured to perform feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and perform a matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data;

[0033] A feature fusion module, used for performing a feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain a fusion feature;

[0034] A feature encoding module, used for encoding the fused features through the global encoder to obtain a first feature map and a second feature map;

[0035] A fusion coding module, used for inputting the first feature map and the second feature map into the target decoder for fusion coding operation, and outputting a category vector and a target box vector;

[0036] The result labeling module is used to label the target object on the image to be detected according to the category vector and the target frame vector to obtain a detection result.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multimodal model-based target detection method as described in the first aspect is implemented.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the target detection method based on the multimodal model as described in the first aspect.

[0039] The target detection method based on a multimodal model provided in the embodiment of the present application has at least the following beneficial effects: first, the image to be detected, the depth image and the millimeter wave signal data captured by the sensor are obtained, and the image to be detected includes at least one target object. By obtaining the depth image, the geometric shape of the visible surface of the target object can be directly obtained, and the distance between the target object and the image acquisition device can be determined. The target object is accurately detected and positioned by obtaining the millimeter wave signal data, and the subsequent detection of the target object is facilitated by obtaining different types of data. Then, the modal mapping module of the multimodal model is used to perform feature mapping on the image to be detected and the depth image to obtain the first feature data, and the modal mapping module is used to perform a matrix transposition operation on the millimeter wave signal data to obtain the second feature data, thereby realizing the conversion of different data, facilitating the subsequent feature selection and fusion processing, and enhancing the feature expression capabilities of the first feature data and the second feature data. Since different feature data may contain complementary information, the embodiment of the present application performs a feature fusion operation on the first feature data and the second feature data through an adaptive fusion module, fuses features of different dimensions into an output feature of a unified scale, obtains a fused feature, reduces the number of features that the model needs to process, reduces the dimension, and thus reduces the computational complexity. Afterwards, the fused feature is feature encoded through a global encoder, which can retain important information while reducing the feature dimension, improve the prediction accuracy of the model, and obtain a first feature map and a second feature map. The first feature map and the second feature map are then input into a target decoder for a fusion encoding operation, so that the target decoder can perform a weighted sum according to the importance of the first feature map and the second feature map, output a category vector and a target frame vector, and achieve accurate detection of the category and position of the object, thereby improving the detection accuracy. Finally, the target object on the image to be detected is labeled according to the category vector and the target frame vector, and the labeling of objects of different categories is achieved to obtain a detection result, thereby achieving accurate detection of the object, being able to label the category and position of the object, and improving the accuracy of object detection.

[0040] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the examples of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0042] Figure 1 is a flowchart of a specific method of a target detection method based on a multimodal model provided in an embodiment of the present application;

[0043] Figure 2 It is a specific flow chart of performing feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model provided in the embodiment of the present application;

[0044] Figure 3 It is a specific flow chart of performing a matrix transposition operation on millimeter wave signal data through a modal mapping module provided in an embodiment of the present application;

[0045] Figure 4 is a specific flow chart of step S103 provided in an embodiment of the present application;

[0046] Figure 5 is a specific flow chart of step S104 provided in an embodiment of the present application;

[0047] Figure 6 is a specific flow chart of step S105 provided in an embodiment of the present application;

[0048] Figure 7 is a specific flow chart of step S106 provided in an embodiment of the present application;

[0049] Figure 8 is a schematic diagram of the structure of a target detection system based on a multimodal model provided in an embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application;

[0051] Fig.10 is a module schematic diagram of a modality mapping module provided in an example of the present application;

[0052] Fig.11 is a module schematic diagram of an adaptive fusion module provided in an example of the present application;

[0053] Fig.12 is a schematic diagram of a global encoder provided by an example of the present application;

[0054] Fig.13 is a schematic diagram of a target decoder provided in an example of the present application;

[0055] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] The target detection method based on a multimodal model provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0060] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0061] In the field of computer vision, the development of multi-target human detection technology has shifted from single modality to multi-modality. This shift is due to some inherent deficiencies of single-modality human detection in the face of complex application scenarios. Multi-target human detection is a key technology in the field of computer vision. Its main purpose is to identify and accurately locate the positions of all human bodies from images or video streams. This technology plays an important role in many application fields, such as public safety monitoring, crowd behavior analysis, interactive media, and advanced driver assistance systems. However, with the development of technology, single-modality human detection systems have gradually failed to meet the growing application needs when facing complex environments and dynamic changes due to their dependence on a single data source. It is further difficult to meet the requirements for high accuracy and high robustness of target recognition.

[0062] In order to solve the above problems, the embodiments of the present application provide a target detection method, system, electronic device and storage medium based on a multimodal model. First, the image to be detected, the depth image and the millimeter wave signal data captured by the sensor are obtained, and the image to be detected includes at least one target object. By obtaining the depth image, the geometric shape of the visible surface of the target object can be directly obtained, and the distance between the target object and the image acquisition device can be determined. By obtaining the millimeter wave signal data, the target object can be accurately detected and positioned. By obtaining different types of data, the subsequent detection of the target object is facilitated. Then, the modal mapping module of the multimodal model is used to perform feature mapping on the image to be detected and the depth image to obtain the first feature data, and the modal mapping module is used to perform a matrix transposition operation on the millimeter wave signal data to obtain the second feature data, so as to realize the conversion of different data, facilitate the subsequent feature selection and fusion processing, and enhance the feature expression ability of the first feature data and the second feature data. Since different feature data may contain complementary information, the embodiment of the present application performs a feature fusion operation on the first feature data and the second feature data through an adaptive fusion module, fuses features of different dimensions into an output feature of a unified scale, obtains a fused feature, reduces the number of features that the model needs to process, reduces the dimension, and thus reduces the computational complexity. Afterwards, the fused feature is feature encoded through a global encoder, which can retain important information while reducing the feature dimension, improve the prediction accuracy of the model, and obtain a first feature map and a second feature map. The first feature map and the second feature map are then input into a target decoder for a fusion encoding operation, so that the target decoder can perform a weighted sum according to the importance of the first feature map and the second feature map, output a category vector and a target frame vector, and achieve accurate detection of the category and position of the object, thereby improving the detection accuracy. Finally, the target object on the image to be detected is labeled according to the category vector and the target frame vector, and the labeling of objects of different categories is achieved to obtain a detection result, thereby achieving accurate detection of the object, being able to label the category and position of the object, and improving the accuracy of object detection.

[0063] Reference Figure 1 , Figure 1 4 is a flowchart of a specific method of a target detection method based on a multimodal model provided in an embodiment of the present application. In some embodiments, the target detection method based on a multimodal model includes but is not limited to steps S101 to S106.

[0064] It should be noted that the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder and a target decoder.

[0065] Step S101, obtaining an image to be detected, a depth image and millimeter wave signal data.

[0066] It should be noted that the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by the sensor.

[0067] In step S101 of some embodiments, an image to be detected, a depth image and millimeter wave signal data are obtained, wherein the image to be detected is obtained by performing image acquisition by an image acquisition device, and the depth image can be obtained by a depth sensor, laser scanning technology, etc., and is used to reflect the distance between the target object and the image acquisition device. Specifically, the depth image uses the distance (depth) value of each point in the scene acquired by the image acquirer as the pixel value of the image, and the millimeter wave signal data is obtained by a sensor using a high-frequency millimeter wave signal for ranging, which facilitates the subsequent detection of the target object through different images and data, and can simultaneously support the detection of multiple data types, facilitating the subsequent multi-modal target detection.

[0068] It should be noted that the embodiment of the present application is described using a millimeter-wave radar as an example. The millimeter-wave radar transmits a series of frequency-modulated signals, which are reflected back to the radar after contacting environmental objects. The radar receives the reflected signals through the receiving antenna and measures the signal's time delay, frequency, amplitude and other parameters, and processes the received millimeter-wave signal data.

[0069] It can be understood that the target detection method based on the multimodal model in the embodiment of the present application can be applied to public safety monitoring, crowd behavior analysis, interactive media and advanced driver assistance systems, etc., wherein the target object can be a vehicle, a human body, etc., and the embodiment of the present application does not make specific restrictions.

[0070] Step S102: Perform feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and perform matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data.

[0071] In step S102 of some embodiments, the modal mapping module of the multimodal model performs feature mapping on the image to be detected and the depth image, thereby converting the image to be detected and the depth image into feature data to obtain first feature data, which can map the image to be detected and the depth image into a more compact feature map. The modal mapping module performs a matrix transposition operation on the millimeter wave signal data, so that different dimensions in the millimeter wave signal data can be continuously upgraded to obtain second feature data, thereby enhancing the feature expression capability of the millimeter wave signal data.

[0072] Step S103: performing a feature fusion operation on the first feature data and the second feature data through an adaptive fusion module to obtain a fusion feature.

[0073] In step S103 of some embodiments, a feature fusion operation is performed on the first feature data and the second feature data through an adaptive fusion module, and different feature data are merged and output to obtain fused features, thereby reducing the impact of outliers and noise, improving the prediction accuracy of the model, and at the same time reducing the number of features that the model needs to process, reducing the dimension, and thus reducing the computational complexity.

[0074] Step S104, feature encoding the fused features through a global encoder to obtain a first feature map and a second feature map.

[0075] In step S104 of some embodiments, the fused features are feature encoded by a global encoder to measure the importance of feature maps of different branches, and a first feature map and a second feature map are obtained. By focusing on the most influential features, the prediction accuracy of the model can be improved, and weighted feature selection helps to improve the interpretability of the model and improve the effect of feature fusion.

[0076] Step S105, input the first feature map and the second feature map into the target decoder for fusion encoding operation, and output a category vector and a target box vector.

[0077] In step S105 of some embodiments, the first feature map and the second feature map are input into the target decoder for fusion coding operation, so that the target feature vectors can be encoded one by one. Finally, after two parallel fully connected operations, the category vector and the target box vector are output respectively, so that the object category and the number of objects of the target object can be determined, which is convenient for the subsequent labeling of the target objects on the image to be detected.

[0078] Step S106, labeling the target object on the image to be detected according to the category vector and the target frame vector to obtain a detection result.

[0079] In step S106 of some embodiments, the target object on the image to be detected is labeled according to the category vector and the target frame vector, so that the position of the target object is marked, and multiple prediction frames are obtained to achieve precise standards for the target object, accurately mark the type and position of the target object, and improve the labeling accuracy.

[0080] In some embodiments, the embodiments of the present application also train the multimodal model. During the training process, the learning of the target classification task adopts the cross entropy loss function, which is expressed as follows:

[0081]

[0082] Among them, y i and represent the true label and predicted probability respectively.

[0083] In the target box regression task, the embodiment of the present application uses the mean squared error (MSE) loss function, which is expressed as follows:

[0084]

[0085] Among them, L i and Represents the i-th coordinate point vector (x i ,y i ).

[0086] Reference Figure 2 , Figure 2 It is a specific flow chart of performing feature mapping of a to-be-detected image and a depth image by a modality mapping module of a multimodal model provided in an embodiment of the present application. In some embodiments, the method includes but is not limited to step S201.

[0087] It can be understood that the modality mapping module in the embodiment of the present application includes a 1*1 convolution layer and a 3*3 convolution layer.

[0088] Step S201: input the image to be detected and the depth image into the modality mapping module, so that the modality mapping module performs a dimensional transformation operation on the image to be detected and the depth image according to a preset number of operations to obtain first feature data.

[0089] In step S201 of some embodiments, in the process of feature mapping the image to be detected and the depth image through the modal mapping module of the multimodal model, the modal mapping module is mainly responsible for converting the original data into feature data to facilitate subsequent feature selection and fusion processing. In the embodiment of the present application, the image to be detected and the depth image are input into the modal mapping module, so that the modal mapping module performs dimensional transformation operations on the image to be detected and the depth image according to a preset number of operations. Specifically, the modal mapping module first performs a downsampling operation on the image to be detected and the depth image, and then sequentially performs a convolution operation on the downsampled image to be detected and the depth image through a 1*1 convolution layer and a 3*3 convolution layer, so as to continuously reduce and increase the row and column length and the number of channels of the image to be detected and the depth image, and obtain the first feature data, so that the image to be detected and the depth image can be mapped into a more compact feature map.

[0090] It should be noted that the preset number of operations in the embodiment of the present application can be set according to the needs of the user. For example, five sets of repeated operations are set, that is, the downsampling operation and the convolution operation are repeated five times.

[0091] It can be understood that, in the process of performing the dimension transformation operation on the image to be detected and the depth image, the dimension of the image to be detected and the depth image is W. i ×H i ×C is used as an example to illustrate that the modal mapping module includes downsampling operations (multiple is 2), 1*1 convolution layer (including 1*1 convolution operation, batch normalization operation and Relu activation operation), 3*3 convolution layer (including 3*3 convolution operation, batch normalization operation and Relu activation operation). Through 5 sets of repeated operations, the row and column length and number of channels of the original data can be continuously reduced and increased in dimension, mapped into a more compact feature map, and the output dimension is: (W i / 32×H i / 32)×32C.

[0092] Reference Figure 3 , Figure 3 It is a specific flow chart of performing matrix transposition operation on millimeter wave signal data through the modal mapping module provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to step S301 to step S302.

[0093] Step S301, performing signal processing on the millimeter wave signal data to obtain the number of point clouds and the length of feature vectors, wherein the length of the point feature vector is used to characterize the information dimension contained in each point cloud point.

[0094] Step S302: input the millimeter wave signal data into the modal mapping module, so that the modal mapping module performs a matrix transposition operation on the millimeter wave signal to continuously increase the dimension of the point cloud quantity and the feature vector length, and outputs the second feature data.

[0095] In steps S301 to S302 of some embodiments, in the process of performing a matrix transposition operation on the millimeter wave signal data through the modal mapping module, the embodiment of the present application first performs signal processing on the millimeter wave signal data, calculates the distance, speed, angle and other information of the object by processing the received millimeter wave signal data, and then converts this information into point cloud data to obtain the number of point clouds and the length of feature vectors, wherein the number of point clouds refers to the total number of target points detected in the point cloud data. Each target point is composed of information such as distance, speed, angle, etc., and these points together constitute a three-dimensional model of the target object; the length of the feature vector is used to characterize the information dimension contained in each point cloud point. Specifically, the longer the length of the feature vector, the richer the information contained in each point cloud point, and the more accurately the characteristics of the target object can be described. Then the millimeter wave signal data is input into the modal mapping module so that the modal mapping module performs a matrix transposition operation on the millimeter wave signal, so that the two dimensions of the number of point clouds and the length of the feature vector can be continuously upgraded, and the second feature data can be output to enhance the feature expression capability of the millimeter wave signal data.

[0096] It can be understood that the modal mapping module also includes 3 sets of alternating 1*1 convolutional layers and matrix transposition operations to realize the processing of millimeter wave signal data.

[0097] Reference Figure 4 , Figure 4 401 and 403. FIG. 402 is a specific flow chart of step S103 provided in an embodiment of the present application. In some embodiments, step S103 specifically includes but is not limited to step S401 and step S403.

[0098] It should be noted that the adaptive fusion module includes a first branch and a second branch, and the adaptive fusion module assigns a selective weight to the input feature of each branch to measure the importance of the feature.

[0099] Step S401, input the first feature data and the feature data into the adaptive fusion module, so that the first branch performs a weighted operation on the first feature data based on the first preset weight to obtain the first weighted data, and the second branch performs a weighted operation on the second feature data based on the second preset weight to obtain the second weighted data.

[0100] Step S402: performing dimension conversion on the first weighted data and the second weighted data.

[0101] Step S403, performing a matrix addition operation on the first weighted data and the second weighted data after dimension conversion to obtain a fusion feature.

[0102] In steps S401 to S403 of some embodiments, in the process of performing feature fusion operation on the first feature data and the second feature data through the adaptive fusion module, the adaptive fusion module is mainly responsible for receiving input features of multiple branches and fusing features of different dimensions into an output feature of a unified scale. The embodiment of the present application inputs the first feature data and the feature data into an adaptive fusion module, so that the first branch performs a weighted operation on the first feature data based on the first preset weight, and the second branch performs a weighted operation on the second feature data based on the second preset weight, to obtain the second weighted data, thereby achieving a measure of the importance of the first feature data and the second feature data, and then performs a dimension conversion on the first weighted data and the second weighted data. Specifically, the feature dimensions of the first weighted data and the second weighted data are converted into a unified feature dimension through a set of simplified 1*1 convolutional layers and matrix transposition operations. Since different features may perform differently under different conditions, the embodiment of the present application performs a matrix addition operation on the first weighted data and the second weighted data after the dimension conversion. Specifically, different feature data are merged and output through a matrix addition operation to obtain a fused feature, thereby reducing the impact of outliers and noise, improving the prediction accuracy of the model, and at the same time being able to reduce the number of features that the model needs to process, reduce the dimension, and thus reduce the computational complexity.

[0103] Reference Figure 5 , Figure 5 4 is a specific flow chart of step S104 provided in an embodiment of the present application. In some embodiments, step S104 specifically includes but is not limited to step S501 and step S502.

[0104] It should be noted that the global encoder includes a third branch and a fourth branch. Specifically, the first branch mainly includes W a ×1 convolutional layer (i.e. W a ×1 convolution operation, batch normalization operation and Relu activation operation) and fully connected layer (fully connected operation, batch normalization operation and Sigmoid activation operation). The second branch mainly contains 1×H a Convolutional layer (i.e. 1×H a Convolution operation, batch normalization operation and Relu activation operation) and fully connected layer (fully connected operation, batch normalization operation and Sigmoid activation operation).

[0105] Step S501, input the fused features into the global encoder, so that the third branch performs feature encoding on the fused features and outputs a column feature vector, and the fourth branch performs feature encoding on the fused features and outputs a row feature vector.

[0106] Step S502: performing a weighted operation on the fused features according to the column feature vectors and the row feature vectors to obtain a first feature map and a second feature map.

[0107] In steps S501 to S502 of some embodiments, in the process of encoding the fused features by the global encoder, the global encoder is mainly responsible for encoding the fused features to further improve the feature expression capability. In the embodiment of the present application, the fused features are input into the global encoder so that the third branch of the global encoder performs feature encoding on the fused features. Specifically, the third branch performs W a ×1 convolution operation, batch standardization operation and Relu activation operation, and then through the full connection operation of the fully connected layer, batch standardization operation and Sigmoid activation operation, the column feature vector is output to facilitate the subsequent column feature weighting operation, and the fourth branch performs feature encoding on the fusion feature. Specifically, the fourth branch performs 1×H a Convolution operation, batch standardization operation and Relu activation operation, and then the fully connected layer's fully connected operation, batch standardization operation and Sigmoid activation operation, output a row feature vector, which is convenient for subsequent row feature weighted operation. After that, the fused features are weighted according to the column feature vector and the row feature vector. Specifically, the embodiment of the present application performs a weighted operation on the fused features through a preset selective weight, so as to measure the importance of different branch feature maps, and obtain a first feature map and a second feature map. By focusing on the most influential features, the prediction accuracy of the model can be improved, and the weighted feature selection helps to improve the interpretability of the model and improve the effect of feature fusion.

[0108] Reference Figure 6 , Figure 6 It is a specific flow chart of step S105 provided in the embodiment of the present application. In some embodiments, step S105 specifically includes but is not limited to step S601 and step S603.

[0109] It should be noted that the target decoder includes a first fully connected layer and a second fully connected layer, and the first fully connected layer and the second fully connected layer are connected in series.

[0110] Step S601: input the first feature map and the second feature map into a target decoder, so that the target decoder performs a feature fusion operation on the first feature map and the second feature map, and outputs a fused feature.

[0111] Step S602, performing feature separation on the fused features to obtain separated features.

[0112] Step S603: Input the separation features into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer sequentially perform encoding operations on the separation features and output a category vector and a target box vector.

[0113] In steps S601 to S603 of some embodiments, in the process of inputting the first feature map and the second feature map into the target decoder for fusion coding operation, the target decoder is mainly responsible for performing the last fusion and feature coding operation on the feature maps of different branches. In the embodiment of the present application, the first feature map and the second feature map are input into the target decoder so that the target decoder performs a feature fusion operation on the first feature map and the second feature map, and outputs a fusion feature. Specifically, the feature fusion operation of the first feature map and the second feature map in the embodiment of the present application is consistent with the feature fusion operation in step S103, and the embodiment of the present application is not repeated here. The fusion feature is then feature separated to obtain a separated feature. After that, the separated feature is sequentially input into the first fully connected layer and the second fully connected layer so that the first fully connected layer and the second fully connected layer sequentially perform coding operations on the separated features, and output a category vector and a target box vector, so that the target feature vector can be encoded one by one, and finally, after two parallel fully connected operations, the category vector and the target box vector are respectively output, so that the object category and the number of objects of the target object can be determined, which is convenient for the subsequent annotation of the target object on the image to be detected.

[0114] Reference Figure 7 , Figure 7 1 is a specific flow chart of step S106 provided in an embodiment of the present application. In some embodiments, step S106 specifically includes but is not limited to step S701 and step S702.

[0115] Step S701, determining the category probabilities of all objects on the image to be detected according to the category vector, and annotating the image to be detected according to the target frame vector to obtain multiple prediction frames.

[0116] Step S702, determining a target prediction box of the target object according to the category probability to obtain a detection result.

[0117] In steps S701 to S702 of some embodiments, in the process of labeling the target object on the image to be detected according to the category vector and the target frame vector, the embodiment of the present application first determines the category probability of all objects on the image to be detected according to the category vector. Taking the target object as a human body as an example, the category vector at this time represents the probability vector of human targets and non-human targets, and labels the image to be detected according to the target frame vector, thereby marking the position of the target object, obtaining multiple prediction frames, realizing the precise standard for the target object, being able to accurately label the type and position of the target object, and improving the labeling accuracy.

[0118] See also Figure 8 The embodiment of the present application also provides a target detection system based on a multimodal model, wherein the multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder; the target detection method based on the multimodal model can be implemented, and the system includes:

[0119] The data acquisition module 801 is used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0120] A feature mapping module 802 is used to perform feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and perform a matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data;

[0121] A feature fusion module 803 is used to perform a feature fusion operation on the first feature data and the second feature data through an adaptive fusion module to obtain a fusion feature;

[0122] A feature encoding module 804 is used to perform feature encoding on the fused features through a global encoder to obtain a first feature map and a second feature map;

[0123] A fusion coding module 805 is used to input the first feature map and the second feature map into a target decoder for a fusion coding operation, and output a category vector and a target frame vector;

[0124] The result labeling module 806 is used to label the target object on the image to be detected according to the category vector and the target frame vector to obtain the detection result.

[0125] The multimodal model-based target detection system of the embodiment of the present application is used to execute the multimodal model-based target detection method in the above embodiment. Its specific processing process is the same as the multimodal model-based target detection method in the above embodiment, and will not be repeated here.

[0126] In order to more clearly explain the target detection method based on the multimodal model of the embodiment of the present application, a specific example is given below for illustration.

[0127] Example 1:

[0128] See also Fig. 9 , Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application.

[0129] Example 1 is explained by taking the target object as a human body.

[0130] like Fig. 9 As shown in FIG. 1 , the embodiment of the present application is designed as an end-to-end multimodal neural network model, which is composed of various types of network modules stacked together. According to the module function, it can be divided into a modality mapping module, an adaptive fusion module, a global encoder, and a target decoder. The model can receive data of different modalities, including RGB images (W i ×H i ×3), depth image (W i ×H i ×1) and millimeter wave signal data (W m ×H m ), where W i and H i Represents the row and column length of the image, W m and H m The number of point clouds and feature vector lengths representing millimeter wave signal data. After a series of module calculations, the model can output the rectangular box positions of multiple target human bodies (indicated by dotted boxes). Each functional module will be described in detail below.

[0131] See also Fig.10 , Fig.10 It is a module diagram of a modal mapping module provided in an example of the present application.

[0132] In some embodiments, the modality mapping module includes two mapping submodules, denoted as modality mapping module a and modality mapping module b. The modality mapping module is mainly responsible for converting raw data into feature data to facilitate subsequent feature selection and fusion processing. Among them, the modality mapping module a is mainly used to process image data (W i ×H i ×C), including RGB images and depth images. This module includes downsampling operations (multiple of 2), 11 convolutional layers (including 1*1 convolution operations, batch normalization operations and Relu activation operations), and 3*3 convolutional layers (including 33 convolution operations, batch normalization operations and Relu activation operations). Through 5 sets of repeated operations, the row and column lengths and the number of channels of the original data can be continuously reduced and increased in dimension, mapped to a more compact feature map, and the output dimension is: (W i / 32×H i / 32)×32C. The modal mapping module b is mainly responsible for processing the millimeter wave signal data (W m ×H m ). This module contains 3 sets of alternating 1*1 convolutional layers and matrix transposition operations. Its purpose is to continuously increase the dimension of the original data point cloud number and feature vector length, enhance the feature expression ability, and finally output 8W m ×8H m feature data.

[0133] See also Fig.11 , Fig.11 It is a module diagram of an adaptive fusion module provided in an example of the present application.

[0134] In some embodiments, the adaptive fusion module is mainly responsible for receiving input features of multiple branches and fusing features of different dimensions into an output feature of a unified scale. Here, two branches are taken as an example. The input features of each branch are assigned a selective weight α to measure the importance of the feature. The weighted features are converted to a uniform feature dimension W through a set of alternating 1*1 convolutional layers and matrix transposition operations. a ×H a Finally, different feature data are merged and output through matrix addition operation. In our method, a total of 4 branch fusion operations are included (such as Fig. 9 As shown, each branch outputs feature maps of different scales: W a ×H a , W a / 2×H a / 2, W a / 4×H a / 4, W a / 8×H a / 8).

[0135] See also Fig.12 , Fig.12 is a schematic diagram of a global encoder provided in an example of the present application.

[0136] In some embodiments, the global encoder is mainly responsible for encoding the fused features to further improve the feature expression capability. The global encoder mainly includes two branches of feature encoding operations. The first branch mainly includes W a ×1 convolutional layer (i.e. W a ×1 convolution operation, batch normalization operation, and Relu activation operation) and a fully connected layer (fully connected operation, batch normalization operation, and Sigmoid activation operation) can generate a column feature vector (1×H a ), which is used to weight the column features of the fused feature map. The second branch mainly includes 1×H a Convolutional layer (i.e. 1×H a Convolution operation, batch normalization operation and Relu activation operation) and fully connected layer (fully connected operation, batch normalization operation and Sigmoid activation operation) can generate a row feature vector (W a ×1), which is used to weight the row features of the fused feature map. Finally, the embodiment of the present application introduces selective weights to measure the importance of different branch feature maps and perform weighted summation on them.

[0137] See also Fig.13 , Fig.13 It is a schematic diagram of a target decoder provided in an example of the present application.

[0138] In some embodiments, the target decoder is mainly responsible for performing the final fusion and feature encoding operations on the feature maps of different branches to generate a category vector and a target box vector. The fusion operation here is consistent with the adaptive fusion module. First, N×C fusion features are generated, where N represents the maximum number of human target detections that can be supported, and C represents the length of the feature vector of each human target. Then, through the separation operation and two series-connected fully connected layers, the human target feature vectors are encoded one by one. Finally, after two parallel fully connected operations, the category vector (1×2) and the target box vector (1×4) are output respectively, where the category vector is represented as a probability vector of human targets and non-human targets, and the target box vector represents the position of the target prediction box, including the upper left corner coordinates (x1, y1) and the lower right corner (x2, y2).

[0139] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the multimodal model-based target detection method in the above embodiment of the present application.

[0140] See also Fig.14 , Fig.14 The hardware structure of the electronic device provided in the embodiment of the present application is illustrated, and the electronic device includes:

[0141] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0142] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the target detection method based on the multimodal model in the embodiment of this application;

[0143] Input / output interface 903, used to implement information input and output;

[0144] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0145] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0146] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0147] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned target detection method based on the multimodal model.

[0148] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0150] It can be understood by those skilled in the art that Figure 1-14 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0151] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0159] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection method based on a multimodal model, characterized in that: The multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder and a target decoder; the method includes: Acquire an image to be detected, a depth image, and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and an image acquisition device, and the millimeter wave signal data is captured by a sensor; Performing feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and performing a matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data; Performing a feature fusion operation on the first feature data and the second feature data by the adaptive fusion module to obtain a fusion feature; Performing feature encoding on the fused features through the global encoder to obtain a first feature map and a second feature map; Inputting the first feature map and the second feature map into the target decoder for fusion encoding operation, and outputting a category vector and a target box vector; The target object on the image to be detected is marked according to the category vector and the target frame vector to obtain a detection result.

2. The target detection method based on a multimodal model according to claim 1, characterized in that: The method of performing feature mapping on the image to be detected and the depth image through the modality mapping module of the multimodal model to obtain first feature data includes: The image to be detected and the depth image are input into the modality mapping module, so that the modality mapping module performs a dimension transformation operation on the image to be detected and the depth image according to a preset number of operations to obtain first feature data.

3. The target detection method based on a multimodal model according to claim 1, characterized in that: The performing a matrix transposition operation on the millimeter wave signal data by the modal mapping module to obtain second feature data includes: Performing signal processing on the millimeter wave signal data to obtain the number of point clouds and the length of feature vectors, wherein the length of the point feature vector is used to characterize the information dimension contained in each point cloud point; The millimeter wave signal data is input into the modal mapping module, so that the modal mapping module performs a matrix transposition operation on the millimeter wave signal to continuously increase the dimension of the point cloud quantity and the feature vector length, and outputs second feature data.

4. The target detection method based on a multimodal model according to claim 1, characterized in that: The adaptive fusion module includes a first branch and a second branch; the adaptive fusion module performs a feature fusion operation on the first feature data and the second feature data to obtain a fusion feature, including: Inputting the first feature data and the feature data into an adaptive fusion module, so that the first branch performs a weighted operation on the first feature data based on a first preset weight to obtain first weighted data, and the second branch performs a weighted operation on the second feature data based on a second preset weight to obtain second weighted data; Performing dimension conversion on the first weighted data and the second weighted data; A matrix addition operation is performed on the first weighted data and the second weighted data after dimension conversion to obtain a fusion feature.

5. The target detection method based on a multimodal model according to claim 1, characterized in that: The global encoder includes a third branch and a fourth branch; the feature encoding of the fusion feature by the global encoder to obtain a first feature map and a second feature map includes: Input the fused feature into the global encoder, so that the third branch performs feature encoding on the fused feature and outputs a column feature vector, and the fourth branch performs feature encoding on the fused feature and outputs a row feature vector; A weighted operation is performed on the fused features according to the column feature vector and the row feature vector to obtain a first feature map and a second feature map.

6. The target detection method based on a multimodal model according to claim 1, characterized in that: The target decoder includes a first fully connected layer and a second fully connected layer; the first feature map and the second feature map are input into the target decoder for fusion encoding operation, and the category vector and the target frame vector are output, including: Inputting the first feature map and the second feature map into the target decoder, so that the target decoder performs a feature fusion operation on the first feature map and the second feature map, and outputs a fusion feature; Performing feature separation on the fused features to obtain separated features; The separation feature is input into the first fully connected layer and the second fully connected layer, so that the first fully connected layer and the second fully connected layer sequentially perform encoding operations on the separation feature and output a category vector and a target box vector.

7. The target detection method based on a multimodal model according to claim 1, characterized in that: The step of labeling the target object on the image to be detected according to the category vector and the target frame vector to obtain a detection result includes: Determine the category probabilities of all objects on the image to be detected according to the category vector, and annotate the image to be detected according to the target frame vector to obtain multiple prediction frames; A target prediction box of the target object is determined according to the category probability to obtain a detection result.

8. A target detection system based on a multimodal model, characterized in that: The multimodal model includes a modality mapping module, an adaptive fusion module, a global encoder and a target decoder; the system includes: A data acquisition module, used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor; A feature mapping module, configured to perform feature mapping on the image to be detected and the depth image through the modal mapping module of the multimodal model to obtain first feature data, and perform a matrix transposition operation on the millimeter wave signal data through the modal mapping module to obtain second feature data; A feature fusion module, used for performing a feature fusion operation on the first feature data and the second feature data through the adaptive fusion module to obtain a fusion feature; A feature encoding module, used for encoding the fused features through the global encoder to obtain a first feature map and a second feature map; A fusion coding module, used for inputting the first feature map and the second feature map into the target decoder for a fusion coding operation, and outputting a category vector and a target box vector; The result labeling module is used to label the target object on the image to be detected according to the category vector and the target frame vector to obtain a detection result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the target detection method based on a multimodal model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the target detection method based on a multimodal model as described in any one of claims 1 to 7.

Images