Target detection method, system, device and medium
By constructing a target detection model with multimodal fusion and single-modal detection branches, and utilizing a cross-attention mechanism to handle common and specific features, the performance limitations of target detection in single-modal and multimodal scenarios are solved, achieving efficient and accurate target detection.
Patent Information
- Application Number
- CN202511415556.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-29
- Publication Date
- 2025-12-23
AI Technical Summary
Existing technologies have limited performance in target detection in both single-modal and multi-modal scenarios, making it difficult to fully utilize multi-modal complementary information, and their detection performance is poor under complex lighting conditions.
A training set is constructed and the multimodal fusion detection branch and single-modal detection branch of the object detection model are trained. Basic features are obtained through the feature extraction module, and common features and specific features are processed in single-modal and multimodal scenarios respectively. Feature fusion is performed using a cross-attention mechanism.
It improves the generalization ability and accuracy of the target detection model, ensuring that it can output high-precision detection results in both single-modal and multi-modal scenarios, and adapts to complex data input scenarios.
Smart Images
Figure CN121190907A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection, and more specifically to a target detection method, system, device, and medium. Background Technology
[0002] In the field of target detection, infrared imaging and visible light imaging are two mainstream sensing methods, each with its unique advantages and limitations. Infrared imaging, based on the thermal radiation characteristics of a target, can image without relying on visible light, thus enabling stable operation under complex lighting conditions such as nighttime, fog, smoke, or strong backlight. By capturing temperature differences, infrared imaging can effectively identify targets including humans, vehicles, and equipment, even those camouflaged (e.g., with camouflage or vegetation cover). Furthermore, because infrared images focus on thermal radiation, they can, to some extent, avoid interference from complex backgrounds (e.g., swaying leaves or shadows). However, infrared detection also has drawbacks such as lower resolution, edge blurring due to thermal diffusion, and potential missed detection of cold targets (e.g., metal).
[0003] In contrast, visible light images provide rich details in color, texture, and shape, making them suitable for fine-grained classification tasks, such as distinguishing vehicle models or pedestrian clothing. Mainstream visible light cameras (such as 4K resolution cameras) offer significantly higher resolution than infrared imaging devices, making them more advantageous for detecting small targets such as distant license plates and faces. Furthermore, visible light detection results are consistent with human visual observation, facilitating manual verification and annotation. Existing visible light datasets (such as COCO and PascalVOC) are large-scale, and annotation tools are mature, making model training more efficient.
[0004] Existing technologies typically employ image-level fusion or decision-level fusion for multimodal target detection. Image-level fusion combines infrared and visible light images into a single image for detection, but it requires both modalities to be present simultaneously, making it difficult to fully utilize existing massive visible light datasets, and may also result in information loss during the fusion process. Decision-level fusion detects the two modalities separately and then fuses the results. While this method can decouple and utilize single-modal data, it fails to fully leverage the complementary information of multiple modalities during detection, leading to performance limitations. Furthermore, real-world scenarios do not always allow for the simultaneous acquisition of multiple sensory data, further restricting the versatility and adaptability of existing multimodal detection methods.
[0005] Therefore, how to achieve efficient and accurate target detection in both single-modal and multi-modal scenarios has become an urgent problem to be solved. Summary of the Invention
[0006] In view of this, in order to overcome at least one aspect of the above problems, embodiments of the present invention propose a target detection method, comprising the following steps: A training set is constructed based on multiple unimodal data samples and multiple multimodal data samples, and the multimodal fusion detection branch and multiple unimodal detection branches in the target detection model are trained based on the training set; Acquire sensory data for at least one modality; Based on the trained feature extraction module, corresponding basic features are calculated for the perceptual data of at least one modality, wherein the basic features include common features and specific features; The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch.
[0007] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of basic features being a first preset value, the basic features are processed based on the corresponding single-modal detection branch to obtain aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The target detection result is obtained based on the alignment feature.
[0008] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of the basic features being a second preset value, each of the basic features is processed based on a multimodal fusion detection branch to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The modality fusion detection result is obtained based on the extracted features.
[0009] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of the basic features being a second preset value, each basic feature is processed based on the corresponding single-modal detection branch to obtain multiple aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The multimodal fusion detection branch processes each of the basic features to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The target detection result is obtained based on the alignment features and the extraction features.
[0010] In some embodiments, obtaining a target detection result based on a plurality of alignment features and a plurality of extracted features further includes: Multiple single-modal detection results are obtained based on the alignment features described above; The modality fusion detection result is obtained based on multiple extracted features; The final target detection result is obtained based on the multiple single-modal detection results and the modal fusion detection results.
[0011] In some embodiments, obtaining a modality fusion detection result based on multiple extracted features further includes: A fused feature is obtained based on the multiple extracted features; The modality fusion detection result is obtained based on the fusion features.
[0012] In some embodiments, obtaining fused features based on the plurality of extracted features further includes: The extracted features are fused based on a cross-attention mechanism to obtain the fused features.
[0013] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a target detection system, comprising: The training module is configured to construct a training set based on multiple single-modal data samples and multiple multimodal data samples, and to train the feature extraction module, multimodal fusion detection branch and multiple single-modal detection branches in the target detection model based on the training set; The acquisition module is configured to acquire sensing data of at least one modality; The first processing module is configured to calculate corresponding basic features based on the trained feature extraction module for the perception data of at least one modality, wherein the basic features include common features and specific features. The second processing module is configured to process at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result.
[0014] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising: At least one processor; and The memory stores a computer program that can run on the processor, which, when executing the program, performs the steps of any of the target detection methods described above.
[0015] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the target detection methods described above.
[0016] This invention offers one of the following beneficial technical effects: The proposed solution fully utilizes the complementarity of single-modal and multi-modal data to train the target detection model, thereby improving the generalization ability of the target detection model. This ensures that the model can stably output high-precision detection results regardless of whether the input scenario is single-modal or multi-modal, achieving perfect adaptation to complex data input scenarios in practical applications. During actual inference, basic features, including common and specific features, are obtained from the perceptual data. Target detection is then performed based on these basic features. When the perceptual data includes only one modality, the common components of the basic features provide good generalization ability, while the specific features retain the unique advantages of that modality, thus ensuring detection performance. When the perceptual data includes multiple modalities, the common features and specific features from multiple basic features can be interacted and enhanced, thereby fully utilizing the complementarity of multiple modalities and improving the accuracy of target detection. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.
[0018] Figure 1 A schematic flowchart of a target detection method provided for an embodiment of the present invention; Figure 2 A flowchart illustrating a target detection method provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the fusion of multiple target detection results provided for embodiments of the present invention; Figure 4 A flowchart illustrating the fusion of multiple extracted features provided for embodiments of the present invention; Figure 5 A schematic diagram of the target detection system provided for an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention; Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.
[0020] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.
[0021] According to one aspect of the present invention, embodiments of the present invention provide a target detection method, such as... Figure 1 As shown, it may include the following steps: S1, a training set is constructed based on multiple single-modal data samples and multiple multimodal data samples, and the feature extraction module, multimodal fusion detection branch and multiple single-modal detection branches in the target detection model are trained based on the training set; S2, acquire sensory data for at least one modality; S3, based on the trained feature extraction module, calculate the corresponding basic features of the perceptual data of at least one modality, wherein the basic features include common features and specific features; S4. Based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch, at least one basic feature is processed to obtain the target detection result.
[0022] Specifically, a dataset is first constructed using various data samples, which can include multiple unimodal and multimodal data, such as visible light image samples, infrared image samples, and visible light and infrared image samples. Then, the target detection model is trained based on the above dataset, making full use of the complementarity of unimodal and multimodal data, thereby improving the generalization ability of the target detection model and ensuring that the model can stably output high-precision detection results in both unimodal and multimodal data input scenarios, achieving perfect adaptation to complex data input scenarios in practical applications.
[0023] After training the target detection module, during actual inference, at least one modality of data from the same scene is first acquired. This can include visible light images, infrared images, LiDAR point clouds, and millimeter-wave radar data. Visible light images provide rich information on color, texture, and shape; infrared images are unaffected by lighting conditions and can highlight thermally radiating targets in low light, nighttime, or smoky environments; LiDAR point clouds provide precise 3D spatial position and depth information, making distance and geometric structure measurements more accurate; and millimeter-wave radar can directly acquire dynamic information such as target velocity, distance, and angle. Time synchronization and spatial registration must be ensured during acquisition to facilitate subsequent feature alignment and fusion. Then, the same feature extraction module can be used to process each modality of data separately, obtaining basic features including common and specific features. Common features ensure feature consistency across different modalities, while specific features retain the unique advantages of each modality, enhancing detection capabilities in complex scenes and thus improving the accuracy of target detection results based on these basic features.
[0024] When the perceived data includes only one modality, the common components of the basic features provide good generalization ability, while specific features retain the unique advantages of that modality. This ensures detection performance when using the corresponding single-modality detection branch. When the perceived data includes multiple modalities, using a multi-modality detection branch allows for interaction and enhancement of common features and specific features from multiple basic features, thereby fully utilizing the complementarity of multiple modalities and improving the accuracy of target detection.
[0025] In some embodiments, although the data acquisition methods and detailed information of the targets differ across modalities, the macroscopic features of the targets, especially edges, shapes, and contextual information, are similar. Therefore, a modality-independent feature extraction module can be used to extract the basic features of each modality, making the extracted features modality-independent. However, in order to preserve the uniqueness of each modality, the basic features obtained by the feature extraction module will not be completely modality-independent; that is, the basic features include a certain proportion of common features and specific features.
[0026] For example, such as Figure 2 As shown, if the collected sensory data consists of visible light and infrared images, these images can be input into the modality-independent feature extraction module for basic feature extraction. During training, to ensure that the modality-independent feature extraction module can effectively extract common features, a feature consistency loss function L can be introduced. Consistent For example, by using common L1 and L2 metrics, visible light images and infrared images can be made similar in features after passing through a mode-independent feature extraction module. L Consistent =Loss(F uni-modal(I Visible ), F uni-modal (I Infrared )) Among them, F uni-modal I represents the modality-independent feature extraction module. Visible and I Infrared This represents a pair of visible light and infrared images.
[0027] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of basic features being a first preset value, the basic features are processed based on the corresponding single-modal detection branch to obtain aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The target detection result is obtained based on the alignment feature.
[0028] Specifically, when the collected sensory data is only from a single modality, the basic features of that single modality (such as visible light only or infrared only) already contain both common and specific features. Furthermore, the specific features within the basic features are generally relatively small, potentially leading to the loss of some modality-specific information. Therefore, to ensure the detection performance of a single modality, the extracted basic features in the corresponding single-modality detection branch can be processed through a modality feature alignment module. This allows the basic features to be better aligned with the original modality, i.e., enhancing specific features and weakening common features. For example, this can be achieved by increasing the proportion of specific features or using other methods to enhance them; or by reducing the proportion of common features or using other methods to weaken them. This strengthens the inherent advantages of single-modality detection, achieving high-precision and high-robustness target detection. Finally, the target detection head is used to obtain the detection results for the corresponding modality.
[0029] For example, such as Figure 2As shown, if the collected sensory data is only visible light images, after obtaining the basic features of the visible light images using a modality-independent feature extraction module, target detection is subsequently performed using a single-modality detection branch for visible light. This single-modality detection branch can include a visible light feature alignment module and a target detection head. In the single-modality detection branch for visible light, the basic features obtained based on the visible light images can be further processed by the visible light feature alignment module to obtain aligned features, thereby supplementing some information unique to visible light images. That is, specific features in the basic features are enhanced while common features are weakened. In other words, the aligned features include enhanced specific features and weakened common features. Finally, the visible light target detection head is used to obtain the visible light detection result. When training the visible light feature alignment module and the visible light target detection head, a target detection loss based on visible light images can be constructed: L Visible =Loss(GT, Det) visible (F visible-align (F uni-modal (I Visible )))) F visible-align Det represents the visible light feature alignment module. visible This represents a target detection head for visible light images. Common target detection loss and optimization methods can be used for model training. GT represents the ground truth.
[0030] Similarly, such as Figure 2 As shown, if the collected sensing data is only infrared images, after obtaining the basic features of the infrared images using a modality-independent feature extraction module, target detection is subsequently performed using a single-modality detection branch of the infrared images. This single-modality detection branch can include an infrared feature alignment module and a target detection head. In the single-modality detection branch of the infrared images, the basic features obtained based on the infrared images can be further processed by the infrared feature alignment module to obtain aligned features, thereby supplementing some information unique to infrared images. That is, specific features in the basic features are enhanced while common features are weakened. In other words, the aligned features include enhanced specific features and weakened common features. Finally, the infrared target detection head is used to obtain the detection result of the infrared images. When training the infrared feature alignment module and the infrared target detection head, a target detection loss based on infrared images can be constructed: L Infrared =Loss(GT, Det) infrared (F infrared-align (F uni-modal (I Infrared )))) F infrared-align Det represents the infrared feature alignment module. infraredThis represents the target detection head for infrared images. Common target detection loss and optimization methods can be used for model training. GT represents the ground truth.
[0031] It should be noted that, Figure 2 Only visible light image detection branch and infrared image detection branch are shown as examples. It is understood that the single-modal detection branches in the target detection model can be set according to actual needs. Each single-modal detection branch includes a feature alignment module and a target detection head for the corresponding modality.
[0032] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of the basic features being a second preset value, each of the basic features is processed based on a multimodal fusion detection branch to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The modality fusion detection result is obtained based on the extracted features.
[0033] Specifically, when the collected perceptual data is multimodal, extracted features can be generated for each basic feature. Since each modality's basic features already contain common and specific features, and the proportion of specific features within the basic features is generally relatively small, to ensure the detection effect of multimodal fusion, targeted adjustments can be made to the common and specific features. This involves enhancing the specific features within each basic feature set and weakening the common features. For example, the proportion of specific features can be increased to enhance them, or other methods can be used to enhance them; conversely, the proportion of common features can be reduced to weaken them, or other methods can be used to weaken them. This process yields extracted features that highlight the core features of each modality and reduce redundant information. Thus, when multiple basic features exist, the extracted features, by optimizing both common and specific features, provide high-quality input for multimodal fusion, resulting in more accurate modal fusion detection results.
[0034] For example, such as Figure 2As shown, if the collected sensing data consists of visible light and infrared images, after obtaining the basic features of the visible light image and the infrared image using a modality-independent feature extraction module, target detection is subsequently performed using a multimodal detection branch. This multimodal detection branch can include a visible light feature extraction module, an infrared feature extraction module, a multimodal feature fusion module, and a target detection head. In the multimodal detection branch, the basic features obtained from the visible light image can be further processed by the visible light feature extraction module to obtain extracted features, and the basic features obtained from the infrared image can be further processed by the infrared feature extraction module to obtain extracted features. That is, specific features in the basic features are enhanced while common features are weakened; in other words, the extracted features include both enhanced specific features and weakened common features. The two extracted features are fused in the multimodal feature fusion module, and finally, the multimodal fusion detection result is obtained using the target detection head. When training the visible light feature extraction module, the infrared feature extraction module, the multimodal feature fusion module, and the target detection head, the following loss function can be constructed: L Fusion =Loss(GT,Det fusion (F fusion [F infrared (F uni-modal (I Infrared )),F visible (F uni-modal (I Visible ))])) Among them, F visible and F infrared This represents the visible light extraction module and the infrared feature extraction module, F fusion Det represents the multimodal feature fusion module. fusion This represents a fused feature object detection head, which can be trained using common object detection loss and optimization methods. GT represents the ground truth.
[0035] It should be noted that, Figure 2 The example multimodal detection branch only includes visible light feature extraction module and infrared feature extraction module. Understandably, the multimodal detection branch can be configured with feature extraction modules for the corresponding modes according to actual needs.
[0036] The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch, and further includes: In response to the number of the basic features being a second preset value, each basic feature is processed based on the corresponding single-modal detection branch to obtain multiple aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The multimodal fusion detection branch processes each of the basic features to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The target detection result is obtained based on the alignment features and the extraction features.
[0037] Specifically, when the collected perceptual data is multimodal, separate extracted features and alignment features can be generated simultaneously for each basic feature. Based on the same process described above, specific features are enhanced while common features are weakened in the alignment and extracted features, thus ensuring the detection performance of both the single-modality branch and the fusion-modality branch. Finally, the detection performance of the single-modality branch and the fusion-modality branch are fused to obtain the final target detection result.
[0038] In some embodiments, obtaining a target detection result based on a plurality of alignment features and a plurality of extracted features further includes: Multiple single-modal detection results are obtained based on the alignment features described above; The modality fusion detection result is obtained based on multiple extracted features; The final target detection result is obtained based on the multiple single-modal detection results and the modal fusion detection results.
[0039] Specifically, multiple single-modal detection results and the modal fusion detection results can be fused based on the cross-union ratio (CUNR) method, that is, each single-modal detection result and the modal fusion detection result are weighted and calculated to obtain the final target detection result.
[0040] For example, if the collected sensory data includes visible light and infrared images, and BOX1 represents the visible light image detection result, where X1, Y1, X1', and Y1' are the bounding box coordinates and S1 is the confidence score; BOX2 represents the infrared image prediction result, where X2, Y2, X2', and Y2' are the bounding box coordinates and S2 is the confidence score; and BOX3 represents the fused prediction result, where X3, Y3, X3', and Y3' are the bounding box coordinates and S3 is the confidence score, then the bounding box coordinates and confidence score of the final target detection result can be obtained based on the following formula:
[0041]
[0042]
[0043]
[0044]
[0045] Alternatively, model-based methods can be used to perform object detection and obtain the final object detection results.
[0046] For example, such as Figure 3 As shown, if the acquired sensory data includes visible light images and infrared images, after obtaining the detection results of the visible light image, the infrared image, and the multimodal fusion detection results, these three detection results are treated as proposals. Then, all proposals are mapped onto the stitched image of the visible light image and the infrared image. Finally, the target detection head is used to perform a second-stage classification and bounding box regression on these proposals to obtain the final detection result.
[0047] In some embodiments, obtaining a modality fusion detection result based on multiple extracted features further includes: A fused feature is obtained based on the multiple extracted features; The modality fusion detection result is obtained based on the fusion features.
[0048] Specifically, the multimodal feature fusion module may include a cross-modal fusion module and a feature aggregation module. In the cross-modal fusion module, multiple extracted features are enhanced to obtain multiple enhanced features, thereby enabling information interaction and mutual complementarity among multiple modalities. In the feature aggregation module, the multiple enhanced features are concatenated and then subjected to several convolutional layers to obtain the fused features.
[0049] For example, such as Figure 4 As shown, the basic features obtained from the visible light image can be processed by the visible light feature extraction module to obtain an extracted feature, and the basic features obtained from the infrared image can also be processed by the infrared feature extraction module to obtain an extracted feature. The two extracted features are enhanced in the cross-modal fusion module to obtain the visible light image enhancement feature and the infrared image enhancement feature. Finally, the visible light image enhancement feature and the infrared image enhancement feature are concatenated in the feature aggregation module and then processed through several layers of convolution to obtain the fused feature.
[0050] In some embodiments, obtaining fused features based on the plurality of extracted features further includes: The extracted features are fused based on a cross-attention mechanism to obtain the fused features.
[0051] Specifically, the cross-attention mechanism uses one extracted feature as the query (Q) and another extracted feature as the key (K)-value (V) for attention calculation. After cross-attention calculation, the extracted features corresponding to each modality are re-weighted and combined. For example, common features are cross-validated through multimodal methods to form more stable macro-information; specific features are selectively integrated according to attention weights to form complementary specific information. In this way, the final fused feature includes the core information of each modality and eliminates feature conflicts between multiple modalities through cross-fusion.
[0052] The proposed solution fully leverages the complementarity of single-modal and multi-modal data to train the target detection model, thereby improving its generalization ability. This ensures the model consistently outputs high-precision detection results regardless of whether the input scenario is single-modal or multi-modal, perfectly adapting to complex data input scenarios in real-world applications. During actual inference, basic features, including common and specific features, are obtained from the perceived data. Target detection is then performed based on these basic features. When the perceived data includes only one modality, the common components of the basic features provide good generalization ability, while the specific features retain the unique advantages of that modality, thus ensuring detection performance. When the perceived data includes multiple modalities, the common features and specific features from multiple basic features can be interacted with and enhanced, fully utilizing the complementarity of multiple modalities to improve the accuracy of target detection.
[0053] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a target detection system 400, such as... Figure 5 As shown, it includes: The training module 401 is configured to construct a training set based on multiple single-modal data samples and multiple multimodal data samples, and to train the feature extraction module, multimodal fusion detection branch and multiple single-modal detection branches in the target detection model based on the training set; The acquisition module 402 is configured to acquire sensing data of at least one modality; The first processing module 403 is configured to calculate corresponding basic features based on the trained feature extraction module for the perception data of at least one modality, wherein the basic features include common features and specific features. The second processing module 404 is configured to process at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result.
[0054] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of basic features being a first preset value, the basic features are processed based on the corresponding single-modal detection branch to obtain aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The target detection result is obtained based on the alignment feature.
[0055] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of the basic features being a second preset value, each of the basic features is processed based on a multimodal fusion detection branch to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The modality fusion detection result is obtained based on the extracted features.
[0056] In some embodiments, processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result further includes: In response to the number of the basic features being a second preset value, each basic feature is processed based on the corresponding single-modal detection branch to obtain multiple aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The multimodal fusion detection branch processes each of the basic features to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The target detection result is obtained based on the alignment features and the extraction features.
[0057] In some embodiments, obtaining a target detection result based on a plurality of alignment features and a plurality of extracted features further includes: Multiple single-modal detection results are obtained based on the alignment features described above; The modality fusion detection result is obtained based on multiple extracted features; The final target detection result is obtained based on the multiple single-modal detection results and the modal fusion detection results.
[0058] In some embodiments, obtaining a modality fusion detection result based on multiple extracted features further includes: A fused feature is obtained based on the multiple extracted features; The modality fusion detection result is obtained based on the fusion features.
[0059] In some embodiments, obtaining fused features based on the plurality of extracted features further includes: The extracted features are fused based on a cross-attention mechanism to obtain the fused features.
[0060] The proposed solution fully leverages the complementarity of single-modal and multi-modal data to train the target detection model, thereby improving its generalization ability. This ensures the model consistently outputs high-precision detection results regardless of whether the input scenario is single-modal or multi-modal, perfectly adapting to complex data input scenarios in real-world applications. During actual inference, basic features, including common and specific features, are obtained from the perceived data. Target detection is then performed based on these basic features. When the perceived data includes only one modality, the common components of the basic features provide good generalization ability, while the specific features retain the unique advantages of that modality, thus ensuring detection performance. When the perceived data includes multiple modalities, the common features and specific features from multiple basic features can be interacted with and enhanced, fully utilizing the complementarity of multiple modalities to improve the accuracy of target detection.
[0061] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer device 501, comprising: At least one processor 520; and The memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the steps of any of the target detection methods described above.
[0062] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 7 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of any of the target detection methods described above.
[0063] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.
[0064] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.
[0065] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.
[0066] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.
[0067] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.
[0068] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0069] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0070] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.
Claims
1. A target detection method, characterized in that, Includes the following steps: A training set is constructed based on multiple single-modal data samples and multiple multimodal data samples, and the feature extraction module, multimodal fusion detection branch and multiple single-modal detection branches in the target detection model are trained based on the training set; Acquire sensory data for at least one modality; Based on the trained feature extraction module, corresponding basic features are calculated for the perceptual data of at least one modality, wherein the basic features include common features and specific features; The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch.
2. The method as described in claim 1, characterized in that, The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch, and further includes: In response to the number of basic features being a first preset value, the basic features are processed based on the corresponding single-modal detection branch to obtain aligned features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The target detection result is obtained based on the alignment feature.
3. The method as described in claim 1, characterized in that, The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch, and further includes: In response to the number of the basic features being a second preset value, each of the basic features is processed based on a multimodal fusion detection branch to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The modality fusion detection result is obtained based on the extracted features.
4. The method as described in claim 1, characterized in that, The target detection result is obtained by processing at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch, and further includes: In response to the number of the basic features being a second preset value, each basic feature is processed based on the corresponding single-modal detection branch to obtain multiple alignment features, wherein the processing of the basic features includes enhancing specific features among the basic features and weakening common features; The multimodal fusion detection branch processes each of the basic features to obtain multiple extracted features. The processing of each basic feature includes enhancing specific features and weakening common features. The target detection result is obtained based on the alignment features and the extraction features.
5. The method as described in claim 4, characterized in that, The target detection result is obtained based on multiple alignment features and multiple extracted features, further including: Multiple single-modal detection results are obtained based on the alignment features described above; The modality fusion detection result is obtained based on multiple extracted features; The final target detection result is obtained based on the multiple single-modal detection results and the modal fusion detection results.
6. The method as described in claim 3 or 5, characterized in that, Based on the extracted features, the modality fusion detection result is obtained, further including: A fused feature is obtained based on the multiple extracted features; The modality fusion detection result is obtained based on the fusion features.
7. The method as described in claim 6, characterized in that, The fused features obtained based on the multiple extracted features further include: The extracted features are fused based on a cross-attention mechanism to obtain the fused features.
8. A target detection system, characterized in that, include: The training module is configured to construct a training set based on multiple single-modal data samples and multiple multimodal data samples, and to train the feature extraction module, multimodal fusion detection branch and multiple single-modal detection branches in the target detection model based on the training set; The acquisition module is configured to acquire sensing data of at least one modality; The first processing module is configured to calculate corresponding basic features based on the trained feature extraction module for the perception data of at least one modality, wherein the basic features include common features and specific features. The second processing module is configured to process at least one basic feature based on the trained multimodal fusion detection branch and / or the corresponding single-modal detection branch to obtain the target detection result.
9. A computer device, comprising: At least one processor; as well as A memory storing a computer program executable on the processor, characterized in that the processor executes the program by performing the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-7.