An ai target detection and recognition method and system for multi-modal data fusion

Abstract

The application discloses a kind of multimodal data fusion AI target detection identification method and system, it is related to computer vision technical field, including, acquisition multimodal original data, to multimodal original data is space-time alignment preprocessing, generates space-time alignment multimodal data cube;With space-time alignment multimodal data cube input multiple lightweight feature extraction backbone network, extracts multimodal primary feature map set;With multimodal primary feature map set input task perception router, generates dynamic routing instruction;According to dynamic routing instruction, from pre-constructed network library, activate corresponding feature extraction subnetwork, and input relevant modal data in space-time alignment multimodal data cube into activated feature extraction subnetwork.The application generates target detection result and decision trace report based on the concept and attribute information after fusion, while improving detection accuracy and stability, enhance the explainability of system and resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an AI target detection and recognition method and system based on multimodal data fusion. Background Technology

[0002] In the fields of artificial intelligence and computer vision, multimodal data fusion for object detection, by integrating complementary information from multiple sensors, has become a key direction for improving the environmental perception capabilities of systems such as autonomous driving and intelligent monitoring. Current mainstream technologies are based on deep learning frameworks and continue to develop along the path of feature-level fusion. Advanced methods, represented by BEVFusion, achieve deeper information interaction by aligning and integrating data from different modalities in a unified bird's-eye view feature space, demonstrating excellent detection accuracy in standard testing environments. The core of these methods lies in designing more precise coordinate transformations, more efficient feature encoding networks, and more powerful detection heads, driving the continuous improvement of multimodal perception performance.

[0003] Existing technologies generally employ static, fixed fusion architectures, where the fusion strategy is determined after model training and cannot be dynamically adjusted according to real-time scenarios. In real-world complex environments, the perception challenges vary across different regions and targets, and the quality and effectiveness of data from each modality constantly change. For example, in nighttime environments, the quality of visible light images degrades while infrared data becomes more reliable; radar data is superior to visual information in detecting small targets at long distances; and traffic sign recognition requires high-resolution RGB texture features. Static fusion architectures struggle to allocate appropriate modal resources for different perception tasks and cannot intelligently adjust dependencies when some modalities fail. This limits the system's robustness and adaptability in dynamic and complex scenarios and may lead to inefficient use of computational resources. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an AI target detection and recognition method based on multimodal data fusion, which solves the problem of lack of scene adaptability in the fusion strategy in the prior art, and realizes the optimal dynamic allocation and fusion of multimodal resources for different perception tasks and real-time environmental conditions.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an AI target detection and recognition method based on multimodal data fusion, which includes: collecting multimodal raw data, performing spatiotemporal alignment preprocessing on the multimodal raw data, and generating a spatiotemporally aligned multimodal data cube; A spatiotemporally aligned multimodal data cube is input into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps; The multimodal primary feature map set is input into the task-aware router to generate dynamic routing instructions; According to the dynamic routing instructions, the corresponding feature extraction subnetwork is activated from the pre-built network library, and the relevant modal data in the spatiotemporally aligned multimodal data cube is input into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set. Within the feature extraction subnetwork, concept and attribute decoupling representation is performed on the modal data to obtain concept vectors and attribute vectors. Concept vectors from different modalities are aligned and fused to obtain fused concept vectors, and attribute vectors from different modalities are associated and organized to obtain a set of associated attributes. The fused concept vector and the associated attribute set are input into the detection head network to obtain the target detection results and decision tracing report.

[0007] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the generation of the spatiotemporally aligned multimodal data cube includes, Timestamp alignment and motion compensation are performed on synchronously acquired RGB visible light images, long-wave infrared thermal imaging images, 3D lidar point cloud and millimeter-wave radar point cloud data; By using a pre-calibrated inter-sensor extrinsic matrix, all modal data are uniformly transformed into the same coordinate system, generating a spatiotemporally aligned multimodal data cube that corresponds strictly to pixel-level and three-dimensional spatial positions.

[0008] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the extracted multimodal primary feature map set includes, The RGB image, infrared image, lidar point cloud projection map, and millimeter-wave radar point cloud projection map in the spatiotemporally aligned multimodal data cube are respectively input into the corresponding lightweight convolutional neural network to extract RGB primary feature maps, infrared primary feature maps, lidar primary feature maps, and millimeter-wave radar primary feature maps of multiple scales containing texture, edge, basic shape, and coarse motion information, thus forming a multimodal primary feature map set.

[0009] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the generation of dynamic routing instructions includes: The multimodal primary feature map set is input into the task-aware router, which analyzes the features through its internal scene parsing subnetwork to generate scene understanding maps that identify different types of regions of interest. The routing decision subnetwork assigns specific detection task labels and corresponding modality combination strategies to each spatial region based on the scenario understanding graph, and outputs dynamic routing instructions for task labels and modality combination strategies.

[0010] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the generated task-oriented deep feature representation set includes, Based on the detection task label in the dynamic routing instruction, retrieve and activate the matching feature extraction subnetwork from the pre-built network library; The modal data specified by the modal combination strategy of the dynamic routing instruction in the spatiotemporally aligned multimodal data cube are input into the activated feature extraction subnetwork for task-related feature extraction, resulting in a task-oriented deep feature representation set.

[0011] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the associated attribute set includes: In the attribute encoding path of the feature extraction subnetwork, feature vectors are extracted from different modal data to describe one or more attributes among target size, three-dimensional shape, color, surface material, temperature, and speed. The associated attribute set is formed by concatenating attribute vectors from different modalities or establishing an attribute-modality mapping table to create a structured attribute description set.

[0012] As a preferred embodiment of the AI ​​target detection and recognition method based on multimodal data fusion described in this invention, the step of obtaining the target detection result and decision tracing report includes: The detection head network predicts the target category based on the fused concept vector and regresses the target location based on the associated attribute set, generating target bounding box, category, and confidence score as the target detection result. By integrating the modality combination strategy and the fusion weight of the concept vector in the dynamic routing instruction, a decision tracing report on the contribution of the detection results is generated.

[0013] Secondly, the present invention provides an AI target detection and recognition system based on multimodal data fusion, comprising a preprocessing module for collecting multimodal raw data, performing spatiotemporal alignment preprocessing on the multimodal raw data, and generating a spatiotemporally aligned multimodal data cube; The extraction module inputs a spatiotemporally aligned multimodal data cube into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps. The instruction module inputs the multimodal primary feature map set into the task-aware router to generate dynamic routing instructions; The construction module activates the corresponding feature extraction subnetwork from the pre-built network library according to the dynamic routing instructions, and inputs the relevant modal data in the spatiotemporally aligned multimodal data cube into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set. The association module, within the feature extraction sub-network, performs concept and attribute decoupling representation on the modal data to obtain concept vectors and attribute vectors. It also aligns and fuses concept vectors from different modalities to obtain fused concept vectors and associates and organizes attribute vectors from different modalities to obtain an association attribute set. The detection module inputs the fused concept vector and the associated attribute set into the detection head network to obtain the target detection results and decision tracing report.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the AI ​​target detection and recognition method for multimodal data fusion as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the AI ​​target detection and recognition method for multimodal data fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By generating dynamic routing instructions through a task-aware router, analyzing scene and target characteristics in real time, and intelligently allocating detection tasks and optimal modal combinations for different regions, adaptive scheduling of computing resources is achieved, improving the system's task execution efficiency and adaptability in complex environments. Concept and attribute decoupling representation and fusion are performed within a dedicated feature extraction sub-network, separating the semantic and physical attributes of target categories for processing. Cross-modal alignment of concept vectors enhances recognition robustness, and structured association of attribute vectors enriches state description. Based on the fused concept and attribute information, target detection results and decision tracing reports are generated, which improves detection accuracy and stability while enhancing the system's interpretability and resource utilization efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an AI-based object detection and recognition method that integrates multimodal data fusion.

[0019] Figure 2 This is a schematic diagram of an AI target detection and recognition system based on multimodal data fusion. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The appearance of an embodiment in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0023] Reference Figures 1-2 This is one embodiment of the present invention, which provides an AI target detection and recognition method based on multimodal data fusion, comprising the following steps: S1. Collect multimodal raw data, perform spatiotemporal alignment preprocessing on the multimodal raw data, and generate a spatiotemporally aligned multimodal data cube.

[0024] S1.1. Perform timestamp alignment and motion compensation on the synchronously acquired RGB visible light images, long-wave infrared thermal imaging images, 3D lidar point cloud and millimeter-wave radar point cloud data.

[0025] Furthermore, at the hardware level, the physical trigger signals of the RGB visible light camera, long-wave infrared thermal imaging camera, 3D LiDAR, and millimeter-wave radar are connected to the same synchronous controller to ensure that all sensors start acquiring data at the same physical moment. Each frame of acquired RGB visible light image, long-wave infrared thermal imaging image, 3D LiDAR point cloud, and millimeter-wave radar point cloud data is stamped with a unified timestamp obtained from the same master clock source with microsecond-level accuracy. Then, for the RGB visible light image and long-wave infrared thermal imaging image, based on the camera's motion trajectory recorded by the inertial measurement unit during exposure, inverse convolution is used. The polar coordinate transformation algorithm performs pixel-level motion compensation line by line on the image to eliminate motion blur caused by carrier movement. For 3D LiDAR point clouds, the three-dimensional coordinates of each LiDAR point are back-interpolated based on the LiDAR's own rotation scanning model and carrier motion information to compensate for point cloud distortion caused by carrier movement. For millimeter-wave radar point clouds, the Doppler phase compensation algorithm is used to correct the velocity measurement deviation caused by carrier movement. Finally, the output RGB visible light image frames, long-wave infrared thermal imaging image frames, 3D LiDAR point cloud frames, and millimeter-wave radar point cloud frames with strictly aligned timestamps and eliminated motion effects are output.

[0026] S1.2 Using the pre-calibrated inter-sensor extrinsic parameter matrix, all modal data are uniformly transformed into the same coordinate system to generate a spatiotemporally aligned multimodal data cube with pixel-level and three-dimensional spatial positions that strictly correspond to each other.

[0027] Furthermore, in the offline phase, a polyhedral calibration plate is simultaneously observed by all sensors at different positions and orientations. The rigid transformation matrices from the RGB visible light camera coordinate system to the 3D LiDAR coordinate system, from the RGB visible light camera coordinate system to the long-wave infrared thermal imaging camera coordinate system, and from the RGB visible light camera coordinate system to the millimeter-wave radar coordinate system are calculated and stored. These matrices are collectively referred to as the inter-sensor extrinsic parameter matrices. During online processing, the long-wave infrared thermal imaging image, the 3D LiDAR point cloud, and the millimeter-wave radar point cloud, processed by S1.1, are multiplied by their respective inter-sensor extrinsic parameter matrices, transforming them from their respective sensor coordinate systems to the RGB visible light camera coordinate system. Under a unified coordinate system based on the RGB visible light camera coordinate system, for 3D LiDAR point clouds and millimeter-wave radar point clouds, their 3D coordinates correspond one-to-one with the physical spatial points in the RGB visible light camera coordinate system after transformation. For long-wave infrared thermal imaging images, through coordinate system transformation and reprojection, an infrared aligned image is generated that is strictly aligned with the RGB visible light image in terms of pixel position. Finally, the aligned RGB visible light image, infrared aligned image, transformed 3D LiDAR point cloud, and transformed millimeter-wave radar point cloud are organized in memory into a multimodal data cube with a unified spatiotemporal reference, where each spatial location index is associated with data from different modalities describing the same physical point.

[0028] S2. Input the spatiotemporally aligned multimodal data cube into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps.

[0029] S2.1. Input the RGB image, infrared image, lidar point cloud projection map and millimeter-wave radar point cloud projection map in the spatiotemporally aligned multimodal data cube into the corresponding lightweight convolutional neural network to extract RGB primary feature maps, infrared primary feature maps, lidar primary feature maps and millimeter-wave radar primary feature maps of multiple scales containing texture, edge, basic shape and coarse motion information, and form a multimodal primary feature map set.

[0030] Furthermore, from the spatiotemporally aligned multimodal data cube, RGB image components, infrared-aligned image components, 3D LiDAR point cloud components, and millimeter-wave radar point cloud components are separated. The RGB image components are fed into a pre-trained residual network containing only the first three convolutional stages with reduced channel count. After inputting the RGB image components, this residual network sequentially performs convolution, batch normalization, ReLU activation, and max pooling operations, and samples the output feature maps of the last three stages of the network to obtain three RGB primary feature maps with different spatial resolutions. The infrared-aligned image components are fed into another residual network with identical structure but independent weights, pre-trained with only the first three convolutional stages and reduced channel count. This residual network processes the infrared-aligned image components in the same way, outputting three infrared primary feature maps corresponding to the spatial resolutions. The 3D LiDAR point cloud components are first converted into a top-view representation based on a unified coordinate system, i.e., the 3D point cloud is discretized in the height direction. The code is a multi-channel 2D raster image, which is then fed into a pre-trained, simplified 2D convolutional neural network (CNN) consisting of several stacked convolutional and pooling layers, outputting a primary feature map of the LiDAR with a fixed number of channels. The millimeter-wave radar point cloud component is processed in the same way as the 3D LiDAR point cloud component. It is first converted into a top-view 2D raster image, and then input into a simplified 2D CNN with the same structure as the LiDAR data but independent weights, outputting a primary feature map of the millimeter-wave radar with a fixed number of channels. Finally, the three-scale RGB primary feature maps extracted from the RGB residual network, the three-scale infrared primary feature maps extracted from the infrared residual network, the LiDAR primary feature map output from the 2D CNN processing the 3D LiDAR point cloud, and the millimeter-wave radar primary feature map output from the 2D CNN processing the millimeter-wave radar point cloud are collectively organized into a multimodal primary feature map set.

[0031] S3. Input the multimodal primary feature map set into the task-aware router to generate dynamic routing instructions.

[0032] S3.1 Input the multimodal primary feature map set into the task-aware router. The task-aware router analyzes the features through its internal scene parsing subnetwork and generates scene understanding maps that identify different types of regions of interest.

[0033] Furthermore, the RGB primary feature maps, infrared primary feature maps, LiDAR primary feature maps, and millimeter-wave radar primary feature maps from the multimodal primary feature map set are concatenated along the channel dimension to form a comprehensive feature map that integrates multi-source information. This comprehensive feature map is input into the scene parsing subnetwork inside the task perception router. The scene parsing subnetwork consists of a convolutional encoder with downsampling capability and a deconvolutional decoder with upsampling capability. The convolutional encoder performs multiple convolutions and pooling operations on the comprehensive feature map to extract high-level semantics, while the deconvolutional decoder gradually restores the spatial resolution and outputs a category score vector at each pixel position. Each element in the category score vector corresponds to a preset region of interest type, such as a road region, sky region, building region, vegetation region, small target enrichment region, or moving object region. By applying the Softmax function to the category score vector at each pixel position and selecting the category with the highest score, a scene understanding map with the same spatial resolution as the input image is generated. Each pixel value in the scene understanding map clearly identifies the region of interest type to which that position belongs.

[0034] S3.2 The routing decision sub-network assigns specific detection task labels and corresponding modal combination strategies to each spatial region based on the scenario understanding graph, and outputs dynamic routing instructions of task labels and modal combination strategies.

[0035] Furthermore, the routing decision subnetwork inside the task-aware router receives the scene understanding map as input. This subnetwork consists of a fully convolutional network that processes the scene understanding map using a sliding window approach, outputting a task decision vector for each spatial location or each image patch defined by the sliding window. The task decision vector is a multi-dimensional vector; its first few dimensions, after being normalized by the Softmax function, represent the probability distribution of the region being assigned to different detection task labels. These detection task labels include long-range general object detection, close-range fine object detection, small object retrieval, traffic sign recognition, and motion trajectory prediction. The last few dimensions of the task decision vector are normalized by the Sigmoid function. After activation, it represents the modality combination strategy corresponding to the aforementioned detection task label. The modality combination strategy is represented by a set of weight values ​​between zero and one. These weight values ​​correspond to the suggested usage intensity or initial weight of RGB mode, infrared mode, LiDAR mode and millimeter-wave radar mode in the current task. For each spatial unit in the scene understanding map, the detection task label with the highest probability in the task decision vector is selected and bound to the corresponding modality combination strategy weight in the vector. Finally, a structured dynamic routing instruction is output. The dynamic routing instruction records the detection task label assigned to different spatial units in the image plane and the modality combination strategy recommended for the detection task label.

[0036] S4. Activate the corresponding feature extraction subnetwork from the pre-built network library according to the dynamic routing instructions, and input the relevant modal data in the spatiotemporally aligned multimodal data cube into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set.

[0037] S4.1. Based on the detection task label in the dynamic routing instruction, retrieve and activate the matching feature extraction sub-network from the pre-built network library.

[0038] Furthermore, in the offline phase, for each possible detection task label, such as long-range general object detection, short-range fine object detection, small object retrieval, traffic sign recognition, and motion trajectory prediction, an independent and complete deep convolutional neural network is trained as a feature extraction sub-network. The structure of each feature extraction sub-network is optimized according to the characteristics of its corresponding detection task. For example, the feature extraction sub-network trained for the traffic sign recognition task may contain more shallow convolutional kernels to capture fine textures, while the feature extraction sub-network trained for the long-range general object detection task may focus more on expanding the receptive field. All these trained feature extraction sub-networks are stored together to form a pre-built network library, with each feature extraction sub-network using its corresponding detection task label as a unique index in the library. During online processing, dynamic routing instructions are parsed, and the detection task labels assigned to different image regions are extracted. Based on these detection task labels as retrieval keys, the corresponding weight parameters of the pre-built feature extraction sub-networks are searched from the pre-built network library and loaded into the memory of the computing device to complete the activation preparation of the feature extraction sub-networks.

[0039] S4.2. Input the modal data specified by the modal combination strategy of the dynamic routing instruction in the spatiotemporally aligned multimodal data cube into the activated feature extraction subnetwork to extract task-related features and obtain a task-oriented deep feature representation set.

[0040] Furthermore, based on the modal combination strategy recorded for each image region in the dynamic routing instructions, specified modal data components are selected from the spatiotemporally aligned multimodal data cube. For example, for regions assigned the long-range general target detection task label and whose modal combination strategy recommends high weighting of the LiDAR and millimeter-wave radar modes, the 3D LiDAR point cloud components and millimeter-wave radar point cloud components in the spatiotemporally aligned multimodal data cube are mainly selected as input. For regions assigned the traffic sign recognition task label and whose modal combination strategy recommends high weighting of the RGB mode, the RGB image components in the spatiotemporally aligned multimodal data cube are mainly selected as input. The selected regions, which are relevant to the region task, are then processed. Modal data components are cropped or masked to the corresponding image region space to form modal data blocks for that region. These modal data blocks are then input into the corresponding feature extraction subnetwork activated in that region for forward propagation computation. The feature extraction subnetwork performs a series of convolution, normalization, nonlinear activation, and pooling operations on the input modal data blocks, ultimately outputting a high-dimensional feature tensor at the deep layer of the network. This high-dimensional feature tensor is the task-oriented deep feature representation of that region. The above process of modal data filtering, cropping, and forward propagation of the feature extraction subnetwork is repeated for all image regions involved in the dynamic routing instructions. The set of high-dimensional feature tensors of all regions obtained constitutes the task-oriented deep feature representation set.

[0041] The modal data expression is: ; in, For modal data, The total number of modes, For modal indexing, For adaptive fusion weights, For normalization function, This is the modal concept vector.

[0042] S5. Within the feature extraction sub-network, concept and attribute decoupling representation is performed on the modal data to obtain concept vectors and attribute vectors. Concept vectors from different modalities are aligned and fused to obtain fused concept vectors. Attribute vectors from different modalities are associated and organized to obtain a set of associated attributes.

[0043] S5.1 In the attribute encoding path of the feature extraction subnetwork, feature vectors are extracted from different modal data to describe one or more attributes of the target, such as size, three-dimensional shape, color, surface material, temperature, and speed.

[0044] Furthermore, the feature extraction subnetwork contains two parallel information processing pathways. One pathway is the attribute encoding pathway, which receives intermediate layer features from different modal data as input. The intermediate layer features of RGB image data are fed into one branch of the attribute encoding pathway, which processes them through convolutional and fully connected layers, ultimately outputting a feature vector describing color and surface texture attributes. The intermediate layer features of long-wave infrared thermal imaging image data are fed into another branch of the attribute encoding pathway, which processes them through a similar structure, outputting a feature vector describing temperature distribution attributes. The features converted from 3D LiDAR point cloud data are fed into a third branch of the attribute encoding pathway, outputting a feature vector describing the target's 3D shape and size attributes. The features converted from millimeter-wave radar point cloud data are fed into a fourth branch of the attribute encoding pathway, outputting a feature vector describing the target's radial velocity and motion trend attributes. These feature vectors extracted from different modal data each focus on characterizing one or more physical attributes of the target, such as size, 3D shape, color, surface material, temperature, and velocity, collectively constituting the output of the attribute encoding pathway.

[0045] S5.2 The associated attribute set is formed by concatenating attribute vectors from different modalities or establishing an attribute-modal mapping table to form a structured attribute description set.

[0046] Furthermore, the color texture attribute feature vectors output from the RGB modal attribute encoding path, the temperature attribute feature vectors output from the infrared modal attribute encoding path, the three-dimensional shape and size attribute feature vectors output from the lidar modal attribute encoding path, and the velocity attribute feature vectors output from the millimeter-wave radar modal attribute encoding path are directly concatenated along the dimension of the feature channels to form a longer-dimensional composite attribute feature vector. This operation is called channel splicing, and the composite attribute feature vector maintains the order and independence of attribute features from different sources. Alternatively, channel splicing can be avoided, and a corresponding entry can be created for each attribute feature vector. The entry records the content of the attribute feature vector, its source modality, and the attribute category it represents, such as size, color, and velocity. The set of all entries constitutes an attribute-modality mapping table. Whether it is a long-dimensional composite attribute feature vector formed by channel splicing or an attribute-modality mapping table formed by recording entries, they both organize and save attribute information from various modalities that describe different aspects of the target in a structured way. This structured whole is called a structured attribute description set, or associated attribute set.

[0047] S6. Input the fused concept vector and the associated attribute set into the detection head network to obtain the target detection results and decision tracing report.

[0048] S6.1 The detection head network predicts the target category based on the fused concept vector and regresses the target location based on the associated attribute set to generate the target bounding box, category, and confidence score of the target detection result. Furthermore, the detection head network includes classification and regression branches. The classification branch receives a fused concept vector as input. This vector undergoes linear transformations and nonlinear activation functions through multiple fully connected layers in the classification branch, ultimately being mapped into a score vector with a dimension equal to the total number of preset target categories. After normalizing this score vector using the Softmax function, the predicted probability for each category is obtained. The category with the highest probability is selected as the predicted target category, and this highest probability value is recorded as the confidence level of the target detection result. The regression branch receives a set of associated attributes as input. If the set of associated attributes is a composite attribute feature vector in channel concatenation form, this vector is directly fed into the regression branch. If the set of associated attributes is an attribute-modality mapping table, then... The table retrieves the attribute feature vectors describing the 3D shape and size and feeds them into the regression branch. The regression branch, also composed of fully connected layers, calculates the input attribute feature vectors and outputs a multi-dimensional vector. This multi-dimensional vector encodes the geometric parameters of the target bounding box. For 2D detection tasks, the geometric parameters include the x and y coordinates of the bounding box center point, width, and height. For 3D detection tasks, the geometric parameters may also include the depth of the center point, the length, width, height, and orientation angle of the bounding box. Based on these regressed geometric parameters, the bounding box of the target can be drawn in the image or 3D space. Combining the predicted target category output by the classification branch, the bounding box geometric parameters output by the regression branch, and the confidence score obtained from the classification probability, a complete target detection result is formed.

[0049] S6.2 Integrate the modal combination strategy and the fusion weight of the concept vector in the dynamic routing instruction to generate a decision tracing report on the contribution of the detection results.

[0050] Furthermore, the generation of the decision tracing report requires integrating information from two different stages. The first is the modality combination strategy pre-assigned to the image region where the current target is located in the dynamic routing instruction. This strategy indicates the recommended usage intensity of the RGB mode, infrared mode, LiDAR mode, and millimeter-wave radar mode in the initial stage in the form of a set of weight values. The second is the fusion weight of each modality concept vector actually calculated and recorded in the concept vector alignment and fusion step, i.e., the adaptive fusion weight coefficient multiplied by each modality concept vector when forming the final fused concept vector. The modality combination strategy weights in the dynamic routing instruction are then aligned and compared with the fusion weights of the concept vectors. Analysis can be performed, for example, by calculating the correlation or difference between two sets of weight vectors, or by weighting and fusing them according to a preset ratio, to form a comprehensive contribution evaluation vector that reflects the relative importance of each modality throughout the entire process from task pre-setting to actual fusion. Based on this comprehensive contribution evaluation vector, one or more dominant modalities that contribute the most to the final detection result can be identified, and a decision tracing report can be generated in the form of natural language description or structured data fields. The decision tracing report clearly indicates which modal data played a key role in the category judgment and location regression of the target detection result, thus providing users with a traceable explanation of the detection decision process.

[0051] This embodiment also provides an AI target detection and recognition system based on multimodal data fusion, including: a preprocessing module that collects multimodal raw data, performs spatiotemporal alignment preprocessing on the multimodal raw data, and generates a spatiotemporally aligned multimodal data cube; The extraction module inputs a spatiotemporally aligned multimodal data cube into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps. The instruction module inputs the multimodal primary feature map set into the task-aware router to generate dynamic routing instructions; The construction module activates the corresponding feature extraction subnetwork from the pre-built network library according to the dynamic routing instructions, and inputs the relevant modal data in the spatiotemporally aligned multimodal data cube into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set. The association module, within the feature extraction sub-network, performs concept and attribute decoupling representation on the modal data to obtain concept vectors and attribute vectors. It also aligns and fuses concept vectors from different modalities to obtain fused concept vectors and associates and organizes attribute vectors from different modalities to obtain an association attribute set. The detection module inputs the fused concept vector and the associated attribute set into the detection head network to obtain the target detection results and decision tracing report.

[0052] This embodiment also provides a computer device applicable to the AI ​​target detection and recognition method of multimodal data fusion, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the AI ​​target detection and recognition method of multimodal data fusion as proposed in the above embodiment.

[0053] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0054] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the AI ​​target detection and recognition method for multimodal data fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0055] In summary, this invention generates dynamic routing instructions through a task-aware router, analyzes scene and target characteristics in real time, and intelligently allocates detection tasks and optimal modal combinations for different regions. This achieves adaptive scheduling of computing resources, improves the system's task execution efficiency and adaptability in complex environments, performs concept and attribute decoupling representation and fusion within a dedicated feature extraction sub-network, separates the semantic and physical attributes of target categories, enhances recognition robustness through cross-modal alignment of concept vectors, enriches state description through structured association of attribute vectors, and generates target detection results and decision tracing reports based on the fused concept and attribute information. This improves detection accuracy and stability while enhancing the system's interpretability and resource utilization efficiency.

[0056] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An AI target detection and recognition method based on multimodal data fusion, characterized in that: include, Collect multimodal raw data, perform spatiotemporal alignment preprocessing on the multimodal raw data, and generate a spatiotemporally aligned multimodal data cube; A spatiotemporally aligned multimodal data cube is input into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps; The multimodal primary feature map set is input into the task-aware router to generate dynamic routing instructions; According to the dynamic routing instructions, the corresponding feature extraction subnetwork is activated from the pre-built network library, and the relevant modal data in the spatiotemporally aligned multimodal data cube is input into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set. Within the feature extraction subnetwork, concept and attribute decoupling representation is performed on the modal data to obtain concept vectors and attribute vectors. Concept vectors from different modalities are aligned and fused to obtain fused concept vectors, and attribute vectors from different modalities are associated and organized to obtain a set of associated attributes. The fused concept vector and the associated attribute set are input into the detection head network to obtain the target detection results and decision tracing report.

2. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 1, characterized in that: The generation of the spatiotemporally aligned multimodal data cube includes, Timestamp alignment and motion compensation are performed on synchronously acquired RGB visible light images, long-wave infrared thermal imaging images, 3D lidar point cloud and millimeter-wave radar point cloud data; By using a pre-calibrated inter-sensor extrinsic matrix, all modal data are uniformly transformed into the same coordinate system, generating a spatiotemporally aligned multimodal data cube that corresponds strictly to pixel-level and three-dimensional spatial positions.

3. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 2, characterized in that: The extracted multimodal primary feature map set includes, The RGB image, infrared image, lidar point cloud projection map, and millimeter-wave radar point cloud projection map in the spatiotemporally aligned multimodal data cube are respectively input into the corresponding lightweight convolutional neural network to extract RGB primary feature maps, infrared primary feature maps, lidar primary feature maps, and millimeter-wave radar primary feature maps of multiple scales containing texture, edge, basic shape, and coarse motion information, thus forming a multimodal primary feature map set.

4. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 3, characterized in that: The generated dynamic routing instructions include, The multimodal primary feature map set is input into the task-aware router, which analyzes the features through its internal scene parsing subnetwork to generate scene understanding maps that identify different types of regions of interest. The routing decision subnetwork assigns specific detection task labels and corresponding modality combination strategies to each spatial region based on the scenario understanding graph, and outputs dynamic routing instructions for task labels and modality combination strategies.

5. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 4, characterized in that: The set of deep feature representations for generating task-oriented features includes, Based on the detection task label in the dynamic routing instruction, retrieve and activate the matching feature extraction subnetwork from the pre-built network library; The modal data specified by the modal combination strategy of the dynamic routing instruction in the spatiotemporally aligned multimodal data cube are input into the activated feature extraction subnetwork for task-related feature extraction, resulting in a task-oriented deep feature representation set.

6. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 5, characterized in that: The set of associated attributes includes, In the attribute encoding path of the feature extraction subnetwork, feature vectors are extracted from different modal data to describe one or more attributes among target size, three-dimensional shape, color, surface material, temperature, and speed. The associated attribute set is formed by concatenating attribute vectors from different modalities or establishing an attribute-modality mapping table to create a structured attribute description set.

7. The AI ​​target detection and recognition method based on multimodal data fusion as described in claim 6, characterized in that: The obtained target detection results and decision tracing report include, The detection head network predicts the target category based on the fused concept vector and regresses the target location based on the associated attribute set, generating target bounding box, category, and confidence score as the target detection result. By integrating the modality combination strategy and the fusion weight of the concept vector in the dynamic routing instruction, a decision tracing report on the contribution of the detection results is generated.

8. A multimodal data fusion AI target detection and recognition system, based on the multimodal data fusion AI target detection and recognition method according to any one of claims 1 to 7, characterized in that: include, The preprocessing module collects multimodal raw data, performs spatiotemporal alignment preprocessing on the multimodal raw data, and generates a spatiotemporally aligned multimodal data cube. The extraction module inputs a spatiotemporally aligned multimodal data cube into multiple lightweight feature extraction backbone networks to extract a set of multimodal primary feature maps. The instruction module inputs the multimodal primary feature map set into the task-aware router to generate dynamic routing instructions; The construction module activates the corresponding feature extraction subnetwork from the pre-built network library according to the dynamic routing instructions, and inputs the relevant modal data in the spatiotemporally aligned multimodal data cube into the activated feature extraction subnetwork to generate a task-oriented deep feature representation set. The association module, within the feature extraction sub-network, performs concept and attribute decoupling representation on the modal data to obtain concept vectors and attribute vectors. It also aligns and fuses concept vectors from different modalities to obtain fused concept vectors and associates and organizes attribute vectors from different modalities to obtain an association attribute set. The detection module inputs the fused concept vector and the associated attribute set into the detection head network to obtain the target detection results and decision tracing report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the AI ​​target detection and recognition method based on multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the AI ​​target detection and recognition method of multimodal data fusion as described in any one of claims 1 to 7.

Images