Scene-aware target detection methods and devices

By acquiring multimodal target detection information and scene information, calculating scene adaptation parameters and detection uncertainty, dynamically adjusting fusion weights, and using a multi-head cross-attention mechanism for cross-modal encoding, the stability and reliability issues of multimodal target detection in complex environments are solved, achieving optimal fusion performance in different scenes and perception states.

CN123116185APending Publication Date: 2026-11-17INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611623791.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing multimodal target detection schemes struggle to adapt to changes in scene and perception modality in complex environments, leading to a decline in the stability and reliability of detection results.

Method used

By acquiring target detection information and scene information from multiple perception modalities, calculating scene adaptation parameters and detection uncertainty, dynamically adjusting fusion weights, and utilizing a multi-head cross-attention mechanism for cross-modal encoding, real-time adaptation of perception modalities is achieved.

Benefits of technology

To maintain optimal fusion detection performance in complex detection environments, improve the stability and reliability of target detection results, and suppress the impact of short-term performance degradation of the sensing modality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116185A_ABST
    Figure CN123116185A_ABST
Patent Text Reader

Abstract

This invention relates to the field of target detection technology, and provides a target detection method and apparatus based on scene perception. The method includes: acquiring target detection information and scene information from multiple perception modalities; determining scene adaptation parameters for each perception modality based on the scene information and target detection information; determining detection uncertainty based on the target detection information of each perception modality; determining fusion weights for each perception modality using the detection uncertainty and scene adaptation parameters; and fusing the target detection information from each modality according to the fusion weights. Through the technical solution provided by this invention, the fusion weights combine scene information and detection uncertainty, thereby improving the stability and reliability of target detection results in complex detection environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method and apparatus based on scene perception. Background Technology

[0002] As the application scenarios for low-speed, small targets such as drones continue to expand, target detection systems need to maintain stable detection capabilities in complex environments with constantly changing lighting, weather, environmental interference, and sensor operating conditions. Since different sensing modalities differ in detection range, environmental adaptability, and detection reliability, how to fully utilize the target information provided by different sensing modalities and obtain stable and reliable fusion detection results has become a crucial technical requirement in the field of multimodal target detection.

[0003] To meet the above requirements, existing multimodal target detection schemes typically utilize multiple sensors to acquire target detection information separately and then jointly process the detection results output by different sensors. Specifically, corresponding fusion weights can be pre-configured for different sensing modalities, or the contribution of each sensing modality in the fusion process can be determined based on detection indicators such as detection confidence level. Then, the target detection results of each modality are weighted and fused according to the corresponding weights to obtain the final target detection result.

[0004] However, in actual detection processes, the detection performance of different sensing modalities changes significantly with variations in the detection scenario, such as lighting, weather, and surrounding environment. Pre-configured fusion relationships cannot accurately reflect the differences in the actual detection capabilities of each sensing modality under the current detection scenario. Furthermore, even within the same detection scenario, each sensing modality may experience varying degrees of performance fluctuations due to instantaneous noise, interference, or changes in its own detection state. Existing fusion methods struggle to balance the impact of scenario changes and the real-time detection state of the sensing modalities. When the detection performance of some sensing modalities degrades, their detection results can still significantly affect the final fusion result, leading to a decrease in the stability and reliability of multimodal target detection results in complex scenarios. Summary of the Invention

[0005] This invention provides a target detection method and apparatus based on scene awareness to address the deficiencies in the prior art.

[0006] This invention provides a scene-aware target detection method, comprising: Acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene; Based on the scene information and the target detection information, the scene adaptation parameters corresponding to each of the perception modalities are determined; Based on the target detection information corresponding to each of the aforementioned perception modes, the detection uncertainty corresponding to each of the aforementioned perception modes is determined; Based on the detection uncertainty and scene adaptation parameters corresponding to each of the aforementioned perception modes, calculate the fusion weight corresponding to each of the aforementioned perception modes; Based on the fusion weights corresponding to each of the aforementioned perception modes, the target detection information corresponding to each of the aforementioned perception modes is fused to obtain the target detection result.

[0007] According to the present invention, a scene-aware target detection method is provided, wherein calculating the fusion weight corresponding to each of the perception modes based on the detection uncertainty corresponding to each perception mode and the scene adaptation parameters includes: For each of the aforementioned perception modes, the difference between 1 and each of the aforementioned detection uncertainties is calculated, and the product of each of the aforementioned differences and the corresponding scene adaptation parameter is calculated to obtain the detection adaptation value corresponding to each of the aforementioned perception modes. The detection adaptation values ​​are normalized, and the fusion weights corresponding to each perception mode are calculated based on the normalized detection adaptation values.

[0008] According to the present invention, a scene-aware target detection method is provided, wherein determining scene adaptation parameters corresponding to each perception modality based on the scene information and the target detection information includes: The scene information is feature-encoded to obtain scene-encoded features; By using a multi-head cross-attention mechanism and the scene coding features, the modal features in each of the target detection information are interactively encoded to obtain the cross-modal coding features corresponding to each of the perception modalities; Based on the attention mechanism, the scene adaptation parameters corresponding to each of the perception modalities are calculated according to the scene encoding features and the cross-modal encoding features.

[0009] According to the present invention, a target detection method based on scene awareness is provided, wherein the scene information includes light intensity, weather conditions, time type, and environment type; the step of feature encoding the scene information to obtain scene encoded features includes: The light intensity, weather conditions, time type, and environment type are respectively subjected to feature transformation to obtain the corresponding scene features; The scene features are concatenated and mapped to a unified feature space to obtain the scene encoding features.

[0010] According to the present invention, a scene-aware target detection method is provided, wherein the modal features in each of the target detection information are interactively encoded using a multi-head cross-attention mechanism and the scene encoding features to obtain cross-modal encoding features corresponding to each of the perception modalities, including: Each modal feature is linearly mapped to obtain the query matrix and value matrix corresponding to each perceptual modality. The scene encoding features are linearly mapped to obtain the key matrix; Based on the multi-head cross-attention mechanism, multiple cross-attentions for each of the perceptual modalities are calculated based on the query matrix, the value matrix, and the key matrix. By concatenating and linearly mapping multiple cross-attentions for each of the aforementioned perceptual modalities, cross-modal coding features corresponding to each of the aforementioned perceptual modalities are obtained.

[0011] According to the present invention, a scene-aware target detection method is provided, wherein determining the detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each perception mode includes: For each of the aforementioned perception modalities, multiple forward inferences are performed on the detection model used to acquire the target detection information to obtain multiple detection outputs; The detection variances of multiple detection outputs corresponding to each of the aforementioned sensing modes are calculated separately. The detection uncertainty corresponding to each of the aforementioned perception modes is determined based on the detection variance corresponding to each of the aforementioned perception modes and the fundamental uncertainty of the aforementioned detection model.

[0012] According to the present invention, a target detection method based on scene awareness is provided, wherein the plurality of perception modes include radar perception mode, optical perception mode, and acoustic perception mode; the step of acquiring target detection information corresponding to the plurality of perception modes includes: Acquire radar echo data, visible light images, infrared images, and multi-channel audio data; The radar echo data is transformed in the range and velocity dimensions to obtain range-Doppler information. The target's range information, velocity information, and signal strength information are extracted from the range-Doppler information to obtain the target detection information corresponding to the radar sensing mode. The visible light image and the infrared image are preprocessed, and visual features and thermal features are extracted respectively to obtain the target detection information corresponding to the optical sensing mode; The multi-channel audio data is converted from time to frequency to obtain the frequency domain features of the target. The direction of arrival of the target is estimated based on the frequency domain features to obtain the target detection information corresponding to the acoustic sensing mode.

[0013] According to a scene-aware target detection method provided by the present invention, before determining the scene adaptation parameters corresponding to each of the perception modalities based on the scene information and the target detection information, the method further includes: Spatial and temporal alignment are performed on each of the target detection information; wherein, the temporal alignment uses the detection time corresponding to one of the target detection information as the base time, and matches the target detection information whose difference between the detection time and the base time meets the preset time deviation condition.

[0014] According to the present invention, a scene-aware target detection method is provided, wherein the target detection information includes detection output and detection probability; the step of fusing the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each perception mode to obtain the target detection result includes: Based on the fusion weights, the detection outputs are weighted and fused to obtain the target detection output; Based on the detection uncertainty and the fusion weight, the detection probabilities are weighted and fused to obtain the fusion confidence. The target detection output and the fusion confidence score are used as the target detection result.

[0015] The present invention also provides a scene-aware target detection device, comprising: The acquisition module is configured to acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene. The first determining module is configured to determine the scene adaptation parameters corresponding to each of the perception modes based on the scene information and the target detection information. The second determining module is configured to determine the detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each of the perception modes; The calculation module is configured to calculate the fusion weights corresponding to each of the perception modes based on the detection uncertainty and the scene adaptation parameters corresponding to each of the perception modes. The fusion module is configured to fuse the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each of the perception modes, so as to obtain the target detection result.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scene-aware target detection method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scene-aware target detection method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the scene-aware target detection method as described above.

[0019] The present invention provides a scene-aware target detection method and apparatus that senses scene information, generates scene adaptation parameters, and drives dynamic adjustment of the fusion weights of each perception modality. Compared with a fixed-weight fusion strategy, this method can maintain optimal fusion detection performance under different scene conditions, and the detection uncertainty further reflects the real-time stability of the current output information of the corresponding perception modality. By combining these two factors to determine the fusion weights, the fusion weights of different perception modalities can be adjusted in a timely manner when external detection conditions change. Simultaneously, when a perception modality experiences short-term performance degradation, the impact of corresponding low-reliability detection information on the final result is suppressed. This allows multi-modal fusion to adapt to both scene changes and perception state changes, thereby improving the stability and reliability of target detection results in complex detection environments. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the scene-aware target detection method provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the spatiotemporal alignment and coordinate transformation provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the fusion weight generation provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the multimodal target detection system provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the target detection device based on scene perception provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data that have been fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0029] Figure 1 This is a flowchart illustrating a scene-aware target detection method according to an exemplary embodiment. Figure 1 As shown in an exemplary embodiment, the scene-aware target detection method includes steps 110 to 150, which are described in detail below.

[0030] Step 110: Obtain target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene.

[0031] In this embodiment of the invention, the target detection information is information obtained by observing the target to be detected through a corresponding perception modality, which can be used for target detection or target state determination. Depending on the perception method and the detection task, the target detection information may include one or more of the following: detection probability, detection confidence, target position, motion state, target category information, or modal features.

[0032] Scene information is used to characterize external detection conditions that may affect the detection performance of each sensing modality during the current detection process. Scene information can come from environmental sensing devices, the sensing devices' own operating parameters, external data sources, or pre-set deployment environment information, or it can be obtained by analyzing the currently collected data. Scene information can be characterized by a single environmental parameter or by multiple environmental parameters jointly characterizing the current detection scene.

[0033] Step 120: Based on the scene information and the target detection information, determine the scene adaptation parameters corresponding to each of the perception modes.

[0034] In this embodiment of the invention, since different perception modalities have varying degrees of sensitivity to external detection conditions, the same scene change may have different effects on different perception modalities. Therefore, scene adaptation parameters corresponding to each perception modality are obtained based on scene information and target detection information.

[0035] Step 130: Based on the target detection information corresponding to each of the perception modes, determine the detection uncertainty corresponding to each of the perception modes.

[0036] In this embodiment of the invention, detection uncertainty is used to characterize the degree of uncertainty of the current target detection information of the corresponding sensing modality. For the same sensing modality, when the target detection information is relatively stable and the consistency between different detection results is high, its detection uncertainty can be relatively low; when the fluctuation of the detection results increases, the target features are unstable, or the differences between detection outputs increase, its detection uncertainty can be correspondingly increased.

[0037] The determination of detection uncertainty can be set according to the data type provided by each sensing modality. For example, it can be determined based on the degree of dispersion between multiple detection results, or based on the degree of fluctuation of detection results within a continuous detection period.

[0038] Step 140: Calculate the fusion weights corresponding to each of the perception modes based on the detection uncertainty and the scene adaptation parameters corresponding to each of the perception modes.

[0039] In this embodiment of the invention, the fusion weights corresponding to each sensing modality are calculated based on the detection uncertainty and scene adaptation parameters corresponding to each sensing modality. Through the above processing, the final fusion weights are not only affected by the external detection scene, but can also be adjusted according to the reliability status of the current target detection information of each sensing modality.

[0040] Step 150: Based on the fusion weights corresponding to each of the perception modes, the target detection information corresponding to each of the perception modes is fused to obtain the target detection result.

[0041] In this embodiment of the invention, after obtaining the fusion weights corresponding to each perception modality, the contribution of target detection information of different perception modalities in the fusion process can be adjusted according to the corresponding fusion weights.

[0042] For numerical target detection information, weighted processing can be performed according to the fusion weights; for target attribute or category information, the candidate detection results provided by each perception modality can be jointly judged according to the corresponding fusion weights; for feature-based target detection information, feature-level combination can be completed first according to the fusion weights, and then the final target detection result can be obtained according to the combined information.

[0043] The final target detection results can be used to characterize whether a target exists, and can also include results related to the target detection task, such as target category, target state, or target spatial information.

[0044] In this embodiment of the invention, scene information is perceived, scene adaptation parameters are generated, and the fusion weights of each perception modality are dynamically adjusted. Compared with a fixed-weight fusion strategy, this approach can maintain optimal fusion detection performance under different scene conditions, and the detection uncertainty further reflects the real-time stability of the current output information of the corresponding perception modality. By combining these two factors to determine the fusion weights, the fusion weights of different perception modalities can be adjusted in a timely manner when external detection conditions change. Simultaneously, when a perception modality experiences short-term performance degradation, the impact of corresponding low-reliability detection information on the final result is suppressed. This allows multimodal fusion to adapt to both scene changes and perception state changes, thereby improving the stability and reliability of target detection results in complex detection environments.

[0045] In an exemplary embodiment of the present invention, the plurality of sensing modes include a radar sensing mode, an optical sensing mode, and an acoustic sensing mode; the step of acquiring target detection information corresponding to the plurality of sensing modes includes: Acquire radar echo data, visible light images, infrared images, and multi-channel audio data; The radar echo data is transformed in the range and velocity dimensions to obtain range-Doppler information. The target's range information, velocity information, and signal strength information are extracted from the range-Doppler information to obtain the target detection information corresponding to the radar sensing mode. The visible light image and the infrared image are preprocessed, and visual features and thermal features are extracted respectively to obtain the target detection information corresponding to the optical sensing mode; The multi-channel audio data is converted from time to frequency to obtain the frequency domain features of the target. The direction of arrival of the target is estimated based on the frequency domain features to obtain the target detection information corresponding to the acoustic sensing mode.

[0046] In this embodiment of the invention, the multiple sensing modes may include radar sensing mode, optical sensing mode and acoustic sensing mode.

[0047] The radar sensing mode can employ millimeter-wave radar, such as 60GHz or 77GHz millimeter-wave FMCW radar. For the acquired radar echo signals, two-dimensional fast Fourier transforms (2D-FFT) can be performed on the range and velocity dimensions respectively to generate range-Doppler (RD) maps. Target detection information, such as range, velocity, and signal strength, corresponding to the target candidate points can then be extracted from the range-Doppler maps.

[0048] In one specific configuration, the radar sensing mode can employ a 77GHz FMCW radar with a transmit bandwidth of 4GHz, a range resolution of approximately 0.0375m, a velocity resolution of approximately 0.1m / s, and a data update rate of 20Hz. In addition to the above information, the detection probability can also be obtained. and radar false alarm rate This information is used as part of the target detection information corresponding to the radar sensing mode.

[0049] Optical sensing modalities can include visible light sensing and infrared sensing. Visible light devices are used to obtain visual information such as target contours, textures, and appearances, while infrared devices are used to obtain the thermal radiation differences between the target and the background. After acquiring visible light and infrared images, preprocessing such as image enhancement, denoising, and normalization can be performed, followed by target detection and feature extraction to obtain visual features. and thermal characteristics .

[0050] As a specific implementation, visible light images can be input into an object detection network to obtain target bounding boxes, category information, and corresponding detection confidence scores. For example, the YOLO series of object detection networks can be used to perform object detection on visible light images. In a specific test, the YOLOv5 object detection network was used to process the visible light image, obtaining the bounding boxes corresponding to suspected targets and a bounding box confidence score of 0.91.

[0051] For optical sensing modes that contain both visible light and infrared information, the features of both can be combined to form optical sensing mode features: .

[0052] In an optional device configuration, the visible light camera can have a resolution of 1920×1080, the infrared thermal imager can have a resolution of 640×512, and the image acquisition frame rate can be 30fps. The target detection network can output the detection confidence score for the corresponding target and further form the detection confidence score corresponding to the optical sensing modality. .

[0053] Acoustic sensing modalities can employ microphone arrays with four or more acquisition channels. In one specific embodiment, an 8-channel microphone array with a sampling rate of 48 kHz and an array aperture of 0.5 m can be used.

[0054] After acquiring multi-channel audio signals, pre-emphasis, framing, and windowing processes are performed sequentially. Then, the time-domain audio signals are converted to the frequency domain using a Fast Fourier Transform to extract the spectral and harmonic characteristics of the target sound. For low-altitude rotor-type targets, their rotor noise can form relatively obvious periodic characteristics in the frequency range of approximately 180Hz to 300Hz.

[0055] For targets with periodically rotating components, their frequency spectrum may contain corresponding higher harmonics in addition to the rotor's fundamental frequency. Therefore, when performing acoustic target detection or classification, the fundamental frequency and one or more higher harmonics can be used together as acoustic signature features. For example, in the detection of a small quadcopter UAV, a rotor fundamental frequency of approximately 195 Hz and a second harmonic of approximately 390 Hz can be detected. By simultaneously analyzing the frequency relationship between the fundamental frequency and its harmonics, the correspondence between acoustic features and rotor-type targets can be improved.

[0056] Simultaneously, spatial acoustic processing can be performed based on the time difference, phase difference, or amplitude difference between multiple microphone channels to determine the direction of arrival (DOA). Optionally, beamforming processing can be used to determine the azimuth and elevation angles of the target. The acoustic sensing modality can further output acoustic signature features, DOA estimation results, and acoustic detection confidence. .

[0057] By acquiring distance and motion information through radar sensing mode, appearance and thermal characteristics through optical sensing mode, and voiceprint and target orientation information through acoustic sensing mode, the complementarity between these three different physical sensing mechanisms can be utilized. When one sensing method is affected by external conditions, the other sensing methods can still provide independent target evidence, providing a relatively complete information source for subsequent dynamic fusion.

[0058] In an exemplary embodiment of the present invention, before determining the scene adaptation parameters corresponding to each of the perception modalities based on the scene information and the target detection information, the method further includes: Spatial and temporal alignment are performed on each of the target detection information; wherein, the temporal alignment uses the detection time corresponding to one of the target detection information as the base time, and matches the target detection information whose difference between the detection time and the base time meets the preset time deviation condition.

[0059] In this embodiment of the invention, since different sensing modalities may differ in terms of installation location, observation coordinate system, and data acquisition frequency, the target detection information of different sensing modalities can be spatially and temporally correspondingly processed before scene adaptation analysis and target information fusion.

[0060] like Figure 2 As shown, for spatial alignment, the coordinate transformation relationship between the local coordinate system of each sensing device and the unified target coordinate system can be pre-defined. Then, the target position after transformation to the unified coordinate system can be expressed by the following formula: ; in, To unify the target position vector in the coordinate system This is the target position vector in the local coordinate system of the sensing device. This is the transformation matrix from the local coordinate system to the unified coordinate system. Let be a rotation matrix. For translation vectors; as mentioned above, sensing devices include millimeter-wave radar (position). Visible light camera (location) Infrared thermal imager (location) ) and acoustic microphone array (location) ).

[0061] For time alignment, matching can be performed based on the global timestamps corresponding to each sensing modality. For any two sensing modalities, the detection timestamps can be matched accordingly. and When satisfied When both satisfy a preset time matching condition, they can be considered to meet the following conditions: This represents the maximum permissible time deviation.

[0062] In specific processing, the detection time can be selected as the reference time from the target detection information corresponding to a perception mode, and the time before and after this reference time can be... Within the corresponding time window, the detection result with the smallest time distance is searched from the target detection information of other sensing modalities, and multiple detection results that meet the time deviation condition are combined into the same group of multi-sensory modal target detection information.

[0063] For example, during a single detection process, the timestamps corresponding to the radar sensing mode, optical sensing mode, and acoustic sensing mode can be: , , r, o, and a represent radar sensing mode, optical sensing mode, and acoustic sensing mode, respectively.

[0064] By unifying spatial coordinates and matching time as described above, subsequent scene adaptation analysis, uncertainty evaluation, and target information fusion can be performed on sensing data from the same target and similar detection times, reducing fusion errors caused by asynchronous sampling or differences in installation coordinates.

[0065] In this embodiment of the invention, the target detection information for spatiotemporal alignment includes timestamps, candidate target identifiers, position or orientation observations, modal features, and detection confidence scores. Specifically, the target detection information for radar sensing modalities includes range, radial velocity, available angle observations, and RD features; the target detection information for optical sensing modalities includes target bounding boxes, visual / thermal features, and confidence scores; and the target detection information for acoustic sensing modalities includes acoustic signature features, DOA (Directivity of Assessment), and confidence scores. The false alarm rate can be provided as a radar quality indicator with the recording, but it is not the target coordinates. In an exemplary embodiment of the invention, determining the scene adaptation parameters corresponding to each sensing modality based on the scene information and the target detection information includes: The scene information is feature-encoded to obtain scene-encoded features; By using a multi-head cross-attention mechanism and the scene coding features, the modal features in each of the target detection information are interactively encoded to obtain the cross-modal coding features corresponding to each of the perception modalities; Based on the attention mechanism, the scene adaptation parameters corresponding to each of the perception modalities are calculated according to the scene encoding features and the cross-modal encoding features.

[0066] In embodiments of the present invention, such as Figure 3 As shown, the scene information corresponding to the current detection scene can be feature-encoded first to form scene-encoded features; the target detection information of each perception modality can be feature-processed respectively, and the modal features corresponding to each perception modality can interact with the scene-encoded features to form cross-modal encoding features; at the same time, the detection uncertainty can be determined based on the multiple detection outputs corresponding to each perception modality; then, the scene adaptation parameters corresponding to each perception modality can be calculated based on the scene-encoded features and cross-modal encoding features.

[0067] In this way, the inherent environmental adaptation characteristics of different perception modalities can be pre-introduced into the fusion process, so that the scene adaptation parameters have clear physical meanings. At the same time, the scene adaptation relationship and the uncertainty of real-time detection work together to avoid simply allocating fusion weights based on preset experience.

[0068] In an exemplary embodiment of the present invention, the scene information includes light intensity, weather conditions, time type, and environment type; the step of feature encoding the scene information to obtain scene encoded features includes: The light intensity, weather conditions, time type, and environment type are respectively subjected to feature transformation to obtain the corresponding scene features; The scene features are concatenated and mapped to a unified feature space to obtain the scene encoding features.

[0069] In this embodiment of the invention, when determining the scene adaptation parameters corresponding to each perception mode, the scene information and the target detection information corresponding to each perception mode can be characterized separately.

[0070] Scene information can be quantified or feature-encoded according to its data format, converting scene descriptions from different sources and data types into scene coding features that can participate in subsequent calculations. Continuous scene information can be directly formed into feature components after normalization, while discrete scene information can be encoded or feature-embedded to form corresponding vector representations. When scene information contains multiple information items, the features corresponding to each information item can be concatenated, and a unified scene coding feature can be obtained through feature mapping.

[0071] Furthermore, scene information can be composed of light intensity, weather conditions, time type, and environment type.

[0072] Among them, light intensity It can be represented as: (Unit: lux lx); Light intensity can be obtained through an independently set light sensor, or it can be estimated based on the exposure time, gain, or other automatic exposure parameters of the optical sensing device.

[0073] Weather conditions This can include conditions such as sunny, cloudy, rainy, foggy, and snowy. Weather conditions can be obtained from external weather data interfaces, determined through visibility or environmental monitoring equipment, or classified based on currently collected multi-sensor data.

[0074] Time type This can include daytime, dusk, and nighttime. Optionally, the sunrise and sunset times for the current date can be determined based on the current timestamp and the geographic information corresponding to the system deployment location, and then the time type to which the current detection time belongs can be determined accordingly.

[0075] Environment type This can include open areas, urban built-up areas, suburbs, or indoor environments. The environment type can be pre-configured during system deployment or identified based on environmental perception data in the current detection area. For example, the distribution of background clutter, multipath characteristics, or static reflective targets in the radar echo can help determine whether the current detection area belongs to a densely built-up environment.

[0076] After obtaining the aforementioned scene information, different data types can be converted into corresponding numerical features. For continuous parameters such as light intensity, normalization can be performed; for discrete parameters such as weather conditions, time type, and environment type, encoding or feature embedding can be performed. Then, the scene features are concatenated or combined and mapped to a unified feature space to form the aforementioned scene-encoded features.

[0077] The scene encoding features are calculated using the following formula: ; in, for 3D scene encoding features This represents a vector concatenation operation. , , , These are the feature embedding functions for light intensity, weather conditions, time type, and environment type, respectively. MLP stands for Multi-Layer Perceptron, used to map the concatenated scene features to a unified scene coding space, obtaining 128-dimensional scene coding features. .

[0078] By adopting the above-mentioned method of constructing scene information, the current detection conditions can be characterized from multiple aspects such as ambient brightness, meteorological conditions, time stage and surrounding spatial environment, avoiding the problem that using only a single environmental parameter cannot fully reflect the actual working environment of different sensing modes.

[0079] In an exemplary embodiment of the present invention, the step of interactively encoding the modal features in each of the target detection information through a multi-head cross-attention mechanism and the scene encoding features to obtain cross-modal encoding features corresponding to each of the perception modalities includes: Each modal feature is linearly mapped to obtain the query matrix and value matrix corresponding to each perceptual modality. The scene encoding features are linearly mapped to obtain the key matrix; Based on the multi-head cross-attention mechanism, multiple cross-attentions for each of the perceptual modalities are calculated based on the query matrix, the value matrix, and the key matrix. By concatenating and linearly mapping multiple cross-attentions for each of the aforementioned perceptual modalities, cross-modal coding features corresponding to each of the aforementioned perceptual modalities are obtained.

[0080] In this embodiment of the invention, the correlation between scene coding features and modal features corresponding to each perception modality can be determined using a multi-head cross-attention mechanism.

[0081] With the first Modal features corresponding to each perceptual modality For example, a linear mapping can be performed on the modal features, and the corresponding query matrix, key matrix, and value matrix can be calculated using the following formulas: ; ; ; in, For the first one mode ( These represent the modal characteristics of radar, optics, and acoustics, respectively. , and These are the corresponding linear mapping parameter matrices.

[0082] Simultaneously, a linear mapping is performed on the scene encoding features to obtain the key matrix generated based on the scene encoding features. , .

[0083] Then, the query matrix, the key matrix generated based on scene encoding features, and the value matrix are respectively divided into multiple attention heads. For the first... Each attention head can be used to calculate the corresponding cross-attention as follows: ; in, This is the query matrix for the m-th modality. The key matrix is ​​generated from scene-encoded features. Let m be the value matrix of the m-th mode. Indicates transpose. is the dimension of the key vector.

[0084] Multiple attention heads can compute the relationships between perceptual information in different feature subspaces. After the computation of each attention head is completed, the outputs of the multiple attention heads are concatenated and then linearly mapped to obtain the first attention head. Cross-modal encoded features corresponding to each perceptual modality: ; in, Indicates the number of attention heads. This represents the output mapping matrix. .

[0085] Subsequently, based on the attention mechanism, combined with scene encoding features Cross-modal coding features corresponding to each perceptual modality Calculate the scene adaptation parameters separately. The formula for calculating the scene adaptation parameters can be expressed as: in, Encoding features for the scene and the first Scene adaptation parameters between modal features Encode features for the scene. For the first Cross-modal coding features of each modality; The technical solution provided by this invention ensures that the features corresponding to each perception modality do not participate in scene adaptation calculation in isolation, but interact with the features of other perception modalities under the constraints of the current scene conditions. Different attention heads can focus on distance, motion, appearance, spectrum, or other feature relationships that can distinguish targets, thereby reflecting the complementary information between perception modalities in the cross-modal encoded features. The final associated response value is simultaneously affected by scene conditions, the features of the corresponding perception modality itself, and information provided by other perception modalities, thus providing more sufficient feature basis for dynamic fusion in complex detection environments.

[0086] In an exemplary embodiment of the present invention, determining the detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each of the perception modes includes: For each of the aforementioned perception modalities, multiple forward inferences are performed on the detection model used to acquire the target detection information to obtain multiple detection outputs; The detection variances of multiple detection outputs corresponding to each of the aforementioned sensing modes are calculated separately. The detection uncertainty corresponding to each of the aforementioned perception modes is determined based on the detection variance corresponding to each of the aforementioned perception modes and the fundamental uncertainty of the aforementioned detection model.

[0087] In this embodiment of the invention, the detection uncertainty corresponding to each sensing mode can be obtained by performing multiple forward inferences with random perturbations on the same data to be detected.

[0088] In one alternative implementation, the Monte Carlo Dropout method based on Bayesian deep learning performs multiple random forward propagations in the corresponding detection model.

[0089] For the Each sensing modality inputs the same data to be detected into the corresponding detection model for processing. After one random forward inference, N detection outputs can be obtained: ; in, Indicates the first The detection output is obtained through random forward inference. The detection output can be the detection probability, classification probability, target state parameters, or other output information that requires uncertainty evaluation.

[0090] First, we can calculate the average of the outputs from multiple tests: ; Furthermore, the statistical variance is obtained based on the dispersion of multiple detection outputs relative to the mean, and this is combined with the inherent fundamental uncertainty of the model. The detection uncertainty is calculated using the following formula: ; Therefore As the first The detection uncertainty corresponding to each sensing mode.

[0091] In this embodiment of the invention, the detection output includes the probability of target presence. Specifically, for the same associated candidate target, keeping the input unchanged, Dropout is enabled during the inference of the m-th modality detection network, and N independent samples are taken to obtain N predicted target presence probabilities. The mean and statistical variance of these probabilities are used to estimate the uncertainty of the modality in determining the target presence. In this embodiment of the invention, when the detection output is a multi-dimensional vector, the variances corresponding to each dimension can be calculated separately and then summed, averaged, or weighted. Alternatively, the trace of the covariance matrix and other statistical measures can be used to characterize the overall dispersion of multiple detection outputs. The dispersion in this embodiment is not limited to statistical variance; depending on the form of the detection output, standard deviation, prediction distribution entropy, or other statistical measures that can reflect the degree of difference between multiple detection outputs can also be used.

[0092] When using uncertainty variance as a basis for the reliability of subsequent detection, a relationship between detection probability and detection uncertainty can also be established: in, Indicates the first The detection probability corresponding to each sensing mode. This represents the corresponding detection probability mapping function, and can be set as a monotonically increasing function. When When it increases, This reduces the effectiveness of detection, thus lowering the reliability of the corresponding detection.

[0093] As one specific implementation method, 20 random forward inferences can be performed on the same data to be detected, i.e. In a well-lit urban detection environment during the day, after processing the three types of sensory information as described above, the detection uncertainty variances can be, for example, 0.08, 0.03, and 0.18. Smaller variances indicate that the outputs of the corresponding sensory modality are more concentrated under multiple random forward inferences, while larger variances indicate that the detection results are more sensitive to internal model perturbations, and the current target detection information has higher uncertainty.

[0094] By performing multiple random forward inferences on the same data to be detected, uncertainty assessment no longer relies solely on a single confidence level given by a single forward inference. Instead, it utilizes the dispersion among multiple prediction results to reflect the stability of the detection model for the current input. When the target features are clear and the detection model has a relatively stable judgment on the current input, the multiple outputs are relatively concentrated, corresponding to low detection uncertainty. When the perceived data is disturbed, the target features are weakened, or the detection model struggles to make stable judgments, the differences between multiple outputs increase, corresponding to a higher detection uncertainty. The detection uncertainty obtained in this way can serve as a quantitative basis for real-time evaluation of the current detection reliability of each perception modality and can be used to correct subsequent scene adaptation parameters.

[0095] In an exemplary embodiment of the present invention, calculating the fusion weights corresponding to each of the perception modalities based on the detection uncertainty and the scene adaptation parameters corresponding to each of the perception modalities includes: For each of the aforementioned perception modes, the difference between 1 and each of the aforementioned detection uncertainties is calculated, and the product of each of the aforementioned differences and the corresponding scene adaptation parameter is calculated to obtain the detection adaptation value corresponding to each of the aforementioned perception modes. The detection adaptation values ​​are normalized, and the fusion weights corresponding to each perception mode are calculated based on the normalized detection adaptation values.

[0096] In this embodiment of the invention, the fusion weight is calculated using the following formula: ; in, For the first Fusion weights for each perceptual modality To adapt parameters for the scene, To detect uncertainty, when using this implementation method, the uncertainty variance can first be transformed to a preset normalization interval, so that... Keep it as a non-negative value, while ensuring .

[0097] In this embodiment of the invention, a mapping strategy between the scene and the perception modality can also be preset to guide the fusion weights. The mapping strategies are shown in List 1 below: Table 1

[0098] The above mapping strategy is embedded into the scene perception process in the form of prior knowledge. In actual operation, the fusion weight generation combines scene coding features and detection uncertainty to dynamically adjust the fusion weight of each perception modality, so as to achieve the optimal fusion detection performance under different scene conditions.

[0099] In an exemplary embodiment of the present invention, the target detection information includes a detection output and a detection probability; the step of fusing the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each perception mode to obtain a target detection result includes: Based on the fusion weights, the detection outputs are weighted and fused to obtain the target detection output; Based on the detection uncertainty and the fusion weight, the detection probabilities are weighted and fused to obtain the fusion confidence. The target detection output and the fusion confidence score are used as the target detection result.

[0100] In this embodiment of the invention, after obtaining the fusion weights corresponding to each perception modality, the fusion weights can be used to jointly process the detection probability, target category information, and target location information.

[0101] Specifically, the calculation formula for weighted fusion of the detection output is as follows: ; in, The final output vector after fusion contains the target presence probability, target type distribution, and 3D location estimate. For the first Fusion weights for each perceptual modality For the first The detection output of each sensing modality; The formula for calculating the weighted fusion of detection probabilities is as follows: ; in, To integrate confidence levels, For the first The detection probability of each sensing modality.

[0102] As an optional system deployment method, such as Figure 4 As shown, the multimodal target detection system used to implement the above-mentioned scene-aware target detection method includes a perception layer, a fusion layer, and an output layer.

[0103] The perception layer is used to obtain target detection information under different perception modes, and may include a millimeter-wave radar subsystem, an optoelectronic perception subsystem, and an acoustic subsystem. The millimeter-wave radar subsystem can output range-Doppler information and detection probability. and radar false alarm rate The photoelectric sensing subsystem may include a visible light camera and an infrared thermal imager, and output target images, visual features, thermal features, and optical detection confidence levels. The acoustic subsystem can acquire target audio signals through a multi-channel microphone array and output acoustic signature features, direction-of-arrival estimation results, and acoustic detection confidence. .

[0104] The fusion layer may include spatiotemporal alignment processing, scene perception processing, cross-sensory modality feature encoding processing, uncertainty quantization processing, adaptive weight generation processing, and joint decision processing. The target detection information generated by each sensory modality is spatiotemporally aligned before entering the subsequent fusion process; scene perception processing obtains the scene information corresponding to the current detection scene and forms scene encoding features; cross-sensory modality feature encoding processing is used to extract the correlation information between different sensory modalities; uncertainty quantization processing is used to obtain the detection uncertainty of the current target detection information of each sensory modality; adaptive weight generation processing combines the current detection scene and detection uncertainty to determine the fusion weights corresponding to each sensory modality; and joint decision processing uses the fusion weights to form the final target detection result.

[0105] The output layer can output one or more of the following based on the actual detection task: target detection confirmation result, target classification result, 3D localization result, and fusion confidence score.

[0106] The aforementioned processing steps can be executed by the same fusion processor or distributed across multiple computing units for collaborative execution. This embodiment does not limit the specific deployment location of each processing step within the physical device.

[0107] The above process will be further explained below using three typical scenarios.

[0108] In a daytime city square scenario, with ample lighting (L>10000 lx), clear weather, and an open urban area environment, there is some building obstruction and multipath reflection. The system needs to detect and track a small drone within a range of 100-800m.

[0109] System Configuration: The millimeter-wave radar subsystem uses a 77GHz FMCW radar with a transmission bandwidth of 4GHz, a range resolution of 0.0375m, a velocity resolution of 0.1m / s, and a data update rate of 20Hz.

[0110] The optoelectronic system uses a 1920x1080 resolution visible light camera and a 640x512 resolution infrared thermal imager at a frame rate of 30fps.

[0111] The acoustic subsystem uses an 8-channel microphone array with a sampling rate of 48kHz and an array aperture of 0.5m.

[0112] Workflow: (1) Multimodal data acquisition and preprocessing The millimeter-wave radar performs 2D-FFT processing on the FMCW echo signal to generate an RD map. In the current frame, the radar detects a suspected target at a distance of 200m and an azimuth angle of 30°, and outputs the RD feature vector. = [200, 15.2, 0.85], where 200 is the target distance (m), 15.2 is the radial velocity (m / s), and 0.85 is the normalized signal strength value. Detection probability. = 0.82, false alarm rate = 0.05.

[0113] A visible light camera captures a clear image of the target from the corresponding viewpoint. After processing by the YOLOv5 target detection network, a suspected drone target is detected with a bounding box confidence score of 0.91, and a visual feature vector is output. The infrared thermal imager detected a hot spot in the corresponding area with a temperature contrast of 12℃ and output a thermal feature vector. Overall optical detection confidence level = 0.93.

[0114] The acoustic subsystem acquires ambient audio through an 8-channel microphone array. After beamforming processing, it detects an acoustic target at an azimuth angle of 32°. The rotor harmonic frequency is 195Hz, the signal-to-noise ratio (SNR) is 8dB, and the output acoustic signature vector is... Acoustic detection confidence level = 0.72.

[0115] (2) Spatiotemporal alignment Based on the global timestamp t = 1685000000.000s, the timestamps of the detection results of the three subsystems are as follows: = 1685000000.000s, = 1685000000.015s, = 1684999999.985s. Maximum permissible time deviation. = 50ms, which meets the time alignment requirements.

[0116] The detection results from each sensor are transformed into a unified coordinate system using a pre-calibrated coordinate transformation matrix. Radar target position (m), after transformation matrix After transformation: (m); The optical target position is transformed by the camera's intrinsic and extrinsic parameters. (m); Acoustic DOA estimation, azimuth 32°, elevation 12°, combined with distance estimation 200m. (m).

[0117] (3) Scene perception and encoding Illumination intensity L = 12500 lx, weather W = sunny, time T = daytime, environment E = open urban area. Scene encoding features are generated through feature embedding and MLP mapping. Among them, the feature components representing the "sunny daytime" scene have higher activation values.

[0118] (4) Cross-modal feature encoding radar characteristics Optical characteristics Harmony and acoustic features The inputs are respectively the Transformer cross-attention module and the scene encoding features. Interact and output cross-modal encoded features and .

[0119] (5) Quantification of uncertainty Using Monte Carlo Dropout (N = 20 samples), the uncertainty variance of each sensing modality is estimated as follows: = 0.08 (low radar uncertainty); = 0.03 (Low optical uncertainty, good visible light performance under clear daytime conditions); (High acoustic uncertainty, urban environmental noise interference); (6) Generation of fusion weights Calculate scene adaptation parameters: ; Calculate the fusion weights by taking into account the detection uncertainty: = (0.35 x (1-0.08)) / (0.35x0.92 + 0.48x0.97 + 0.17x0.82) = 0.322 / 0.9270 = 0.347357; = (0.48 x 0.97) / 0.9270 =0.502265; = (0.17 x 0.82) / 0.9270 =0.15037.

[0120] (7) Joint decision-making output Fusion detection probability: = 0.347357 x 0.82 + 0.502265 x 0.93 + 0.15037 x 0.72 = 0.860211; because If the threshold is >0.85, the target detection is confirmed as "existing".

[0121] Target classification: The optical modality provides the highest resolution appearance features, and the fusion classification result is "quadcopter UAV - small (wheelbase <300mm)".

[0122] 3D positioning: x [173.5, 100.2, 45.0] + x [174.0, 99.8, 45.2] + x[169.6, 105.9, 41.6] = [173.2, 100.5, 44.9](m); Fusion confidence: = 0.347357 x 0.82 x 0.92 +0.50226 x 0.93 x 0.97 +0.15037 x 0.72 x 0.82 =0.803923.

[0123] Results Analysis: In a sunny city square scene during the day, the optical sensor had the highest fusion weight (0.54), followed by radar (0.37), and acoustic assistance (0.16). The fusion detection confidence reached 0.860, which is higher than the detection performance of any single mode, verifying the effectiveness of the three-modal adaptive fusion of this invention.

[0124] In a nighttime, foggy scenario, with extremely low light levels (L < 1 lx) and low visibility (approximately 200m), the system needs to detect intruding drones within a range of 50-400m.

[0125] Workflow: (1) Multimodal data acquisition and preprocessing Millimeter-wave radar is unaffected by foggy or nighttime conditions and detected a suspected target at a distance of 150 meters. = 0.88, = 0.03.

[0126] Visible light cameras suffer severe image quality degradation under low light conditions, with target detection confidence levels dropping to only [insert value here]. =0.42. Infrared thermal imagers utilize the thermal radiation characteristics of targets, enabling them to detect target thermal features even under low visibility conditions. = 0.78. Overall optical detection confidence level = 0.65 (mainly contributed by infrared).

[0127] The acoustic subsystem operates in relatively quiet environments at night, reducing background noise to 55 dB and improving the rotor harmonic detection signal-to-noise ratio to 12 dB. = 0.85.

[0128] (2) Spatiotemporal alignment Similar to the above, trimodal data alignment is achieved through global timestamps and coordinate transformation matrices.

[0129] (3) Scene perception and encoding Light intensity L = 0.5 lx, weather W = foggy, time T = night, environment E = suburbs.

[0130] Scene encoding features are generated through feature embedding and MLP mapping. Among them, the feature components representing the "night foggy day" scene have higher activation values.

[0131] (4) Quantification of uncertainty = 0.06 (Radar performance is stable, with minimal impact from fog). = 0.22 (Optical performance degraded, visible light performance almost failed, infrared performance is acceptable); = 0.10 (Good acoustic performance, low ambient noise at night); (5) Generation of fusion weights Calculate scene adaptation parameters: ; Calculate the fusion weights by taking into account the detection uncertainty: = (0.52 x 0.94) / (0.52x0.94 + 0.20x0.78 + 0.28x0.90) = 0.4888 / 0.866 = 0.56; = (0.20 x 0.78) / 0.866 = 0.18; = (0.28 x 0.90) / 0.866 = 0.29.

[0132] (6) Joint decision-making output Fusion detection probability: = 0.56 x 0.88 + 0.18 x 0.65 + 0.29 x 0.85 = 0.493 + 0.117 + 0.247 = 0.857; Target detection confirmed as "existing"; Target classification: Radar provides micro-Doppler characteristics, and acoustics provides rotor harmonic characteristics; the overall classification is "quadcopter UAV - medium-sized"; 3D positioning: [148.5, 72.3, 38.2] (m); Fusion confidence: = 0.56 x 0.88 x 0.94 + 0.18 x 0.65 x 0.78 + 0.29 x0.85 x 0.90 = 0.463 + 0.091 + 0.222 = 0.776; Results Analysis: In foggy nighttime scenarios, the radar sensor automatically obtained the highest fusion weight (0.56), followed by the acoustic sensor (0.29), while the optical sensor's weight decreased (0.18). Through scene perception and adaptive weight adjustment, the system effectively addressed the performance degradation of optical sensors caused by foggy and nighttime conditions, maintaining a high fusion detection confidence level (0.776) and achieving all-weather operation capability.

[0133] In a building-occupied RF environment scenario, the system is deployed in an urban commercial area surrounded by multiple high-rise buildings, causing severe multipath effects and partial obstruction of millimeter-wave radar signals. The lighting conditions are cloudy (L = 5000 lx), and the environment type is an urban building area. The system needs to detect small drones flying between buildings within a range of 100-500m.

[0134] Workflow: (1) Multimodal data acquisition and preprocessing Millimeter-wave radar is severely affected by multipath effects in urban built-up environments, resulting in a large number of false targets in the radar echo. Even after constant false alarm rate (CFAR) detection, a suspected target is detected in the direction of the real target, but due to multipath interference, the detection probability drops significantly. = 0.65, the false alarm rate rose to = 0.15.

[0135] Visible light cameras can still achieve good image quality under cloudy conditions, and the target detection confidence level is high. =0.82. The infrared thermal imager is affected by background thermal radiation from buildings, resulting in a low detection confidence level. = 0.68. Overall optical detection confidence level = 0.78.

[0136] The acoustic subsystem forms a good acoustic propagation channel between the buildings, the 8-channel microphone array accurately locates the target direction through DOA estimation, and the rotor harmonic detection signal-to-noise ratio (SNR) is 10 dB. = 0.88.

[0137] (2) Spatiotemporal alignment Similar to the above, trimodal data alignment is achieved through global timestamps and coordinate transformation matrices. Since building obstruction increases radar detection angle deviation, additional angle uncertainty compensation is introduced into the radar detection results during the alignment process.

[0138] (3) Scene perception and encoding Illumination intensity L = 5000 lx, weather W = cloudy, time T = daytime, environment E = urban buildings. Scene-encoded features are generated through feature embedding and MLP mapping. Among them, the feature components representing the "urban building-RF obstruction" scenario have higher activation values.

[0139] (4) Quantification of uncertainty = 0.28 (Radar performance is significantly degraded, and multipath effects lead to a significant increase in uncertainty); = 0.15 (Optical performance is acceptable, but the viewing angle is affected by building obstruction). = 0.07 (Good acoustic performance; architectural reflections even aid in acoustic localization). (5) Generation of fusion weights Calculate scene adaptation parameters: ; Calculate the fusion weights by taking into account the detection uncertainty: = (0.25 x 0.72) / (0.25x0.72 + 0.32x0.85 + 0.43x0.93) = 0.18 / 0.8189 = 0.22; = (0.32 x 0.85) / 0.8189 = 0.33; = (0.43 x 0.93) / 0.8189 = 0.49.

[0140] (6) Joint decision-making output Fusion detection probability: = 0.22 x 0.65 + 0.33 x 0.78 + 0.49 x 0.88 = 0.143 + 0.257 + 0.431 = 0.831; Target detection confirmed as "existing".

[0141] Target classification: Acoustic modes provide rotor harmonic characteristics (195Hz, 390Hz harmonics), and optics provide fuselage profile characteristics. The overall classification is "quadcopter drone - small - consumer grade". 3D positioning: [252.0, 180.5, 65.3] (m); Fusion confidence: = 0.22 x 0.65 x 0.72 + 0.33 x 0.78 x 0.85 + 0.49 x0.88 x 0.93 = 0.103 + 0.219 + 0.401 = 0.723.

[0142] Results Analysis: In a building-obstructed RF environment, radar performance significantly degrades due to multipath effects, with the uncertainty variance increasing to 0.28. The adaptive weight generation module automatically reduced the radar fusion weight (0.22) and increased the acoustic and optical weights (acoustic 0.49, optical 0.33). The acoustic sensor utilizes the acoustic channel formed by the building complex to achieve high-precision target detection and localization, effectively compensating for the detection gap caused by radar performance degradation. This embodiment verifies the advantages of this invention in maintaining system detection capability through uncertainty quantization and adaptive weight adjustment in scenarios with partial sensor performance degradation.

[0143] The scene-aware target detection device provided by the present invention will be described below. The scene-aware target detection device described below can be referred to in correspondence with the scene-aware target detection method described above. It should be noted that the device provided in the embodiments below and the method provided in the embodiments above belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0144] In one exemplary embodiment of the present invention, please refer to Figure 5 , Figure 5 An exemplary embodiment of a scene-aware target detection device is shown, comprising the following modules: The acquisition module 510 is configured to acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene. The first determining module 520 is configured to determine the scene adaptation parameters corresponding to each of the perception modes based on the scene information and the target detection information. The second determining module 530 is configured to determine the detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each of the perception modes; The calculation module 540 is configured to calculate the fusion weights corresponding to each of the perception modes based on the detection uncertainty and the scene adaptation parameters corresponding to each of the perception modes. The fusion module 550 is configured to fuse the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each of the perception modes, so as to obtain the target detection result.

[0145] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a scene-aware target detection method, which includes: Acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene; Based on the scene information and the target detection information, the scene adaptation parameters corresponding to each of the perception modalities are determined; Based on the target detection information corresponding to each of the aforementioned perception modes, the detection uncertainty corresponding to each of the aforementioned perception modes is determined; Based on the detection uncertainty and scene adaptation parameters corresponding to each of the aforementioned perception modes, calculate the fusion weight corresponding to each of the aforementioned perception modes; Based on the fusion weights corresponding to each of the aforementioned perception modes, the target detection information corresponding to each of the aforementioned perception modes is fused to obtain the target detection result.

[0146] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the scene-aware target detection method provided by the above methods, the method comprising: Acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene; Based on the scene information and the target detection information, the scene adaptation parameters corresponding to each of the perception modalities are determined; Based on the target detection information corresponding to each of the aforementioned perception modes, the detection uncertainty corresponding to each of the aforementioned perception modes is determined; Based on the detection uncertainty and scene adaptation parameters corresponding to each of the aforementioned perception modes, calculate the fusion weight corresponding to each of the aforementioned perception modes; Based on the fusion weights corresponding to each of the aforementioned perception modes, the target detection information corresponding to each of the aforementioned perception modes is fused to obtain the target detection result.

[0148] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the scene-aware target detection method provided by the methods described above, the method comprising: Acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene; Based on the scene information and the target detection information, the scene adaptation parameters corresponding to each of the perception modalities are determined; Based on the target detection information corresponding to each of the aforementioned perception modes, the detection uncertainty corresponding to each of the aforementioned perception modes is determined; Based on the detection uncertainty and scene adaptation parameters corresponding to each of the aforementioned perception modes, calculate the fusion weight corresponding to each of the aforementioned perception modes; Based on the fusion weights corresponding to each of the aforementioned perception modes, the target detection information corresponding to each of the aforementioned perception modes is fused to obtain the target detection result.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scene-aware target detection method, characterized in that, include: Acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene; Based on the scene information and the target detection information, the scene adaptation parameters corresponding to each of the perception modalities are determined; Based on the target detection information corresponding to each of the aforementioned perception modes, the detection uncertainty corresponding to each of the aforementioned perception modes is determined; Based on the detection uncertainty and scene adaptation parameters corresponding to each of the aforementioned perception modes, calculate the fusion weight corresponding to each of the aforementioned perception modes; Based on the fusion weights corresponding to each of the aforementioned perception modes, the target detection information corresponding to each of the aforementioned perception modes is fused to obtain the target detection result.

2. The scene-aware target detection method according to claim 1, characterized in that, The calculation of the fusion weights corresponding to each of the perception modalities based on the detection uncertainty and the scene adaptation parameters includes: For each of the aforementioned perception modes, the difference between 1 and each of the aforementioned detection uncertainties is calculated, and the product of each of the aforementioned differences and the corresponding scene adaptation parameter is calculated to obtain the detection adaptation value corresponding to each of the aforementioned perception modes. The detection adaptation values ​​are normalized, and the fusion weights corresponding to each perception mode are calculated based on the normalized detection adaptation values.

3. The scene-aware target detection method according to claim 1, characterized in that, The step of determining the scene adaptation parameters corresponding to each perception modality based on the scene information and the target detection information includes: The scene information is feature-encoded to obtain scene-encoded features; By using a multi-head cross-attention mechanism and the scene coding features, the modal features in each of the target detection information are interactively encoded to obtain the cross-modal coding features corresponding to each of the perception modalities; Based on the attention mechanism, the scene adaptation parameters corresponding to each of the perception modalities are calculated according to the scene encoding features and the cross-modal encoding features.

4. The scene-aware target detection method according to claim 3, characterized in that, The scene information includes light intensity, weather conditions, time type, and environment type; the feature encoding of the scene information to obtain scene encoded features includes: The light intensity, weather conditions, time type, and environment type are respectively subjected to feature transformation to obtain the corresponding scene features; The scene features are concatenated and mapped to a unified feature space to obtain the scene encoding features.

5. The scene-aware target detection method according to claim 3, characterized in that, The step of interactively encoding the modal features in each of the target detection information through a multi-head cross-attention mechanism and the scene encoding features to obtain the cross-modal encoding features corresponding to each of the perception modalities includes: Each modal feature is linearly mapped to obtain the query matrix and value matrix corresponding to each perceptual modality. The scene encoding features are linearly mapped to obtain the key matrix; Based on the multi-head cross-attention mechanism, multiple cross-attentions for each of the perceptual modalities are calculated based on the query matrix, the value matrix, and the key matrix. By concatenating and linearly mapping multiple cross-attentions for each of the aforementioned perceptual modalities, cross-modal coding features corresponding to each of the aforementioned perceptual modalities are obtained.

6. The scene-aware target detection method according to claim 1, characterized in that, The determination of detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each of the perception modes includes: For each of the aforementioned perception modalities, multiple forward inferences are performed on the detection model used to acquire the target detection information to obtain multiple detection outputs; The detection variances of multiple detection outputs corresponding to each of the aforementioned sensing modes are calculated separately. The detection uncertainty corresponding to each of the aforementioned perception modes is determined based on the detection variance corresponding to each of the aforementioned perception modes and the fundamental uncertainty of the aforementioned detection model.

7. The scene-aware target detection method according to claim 1, characterized in that, The multiple sensing modes include radar sensing mode, optical sensing mode, and acoustic sensing mode; acquiring target detection information corresponding to each of the multiple sensing modes includes: Acquire radar echo data, visible light images, infrared images, and multi-channel audio data; The radar echo data is transformed in the range and velocity dimensions to obtain range-Doppler information. The target's range information, velocity information, and signal strength information are extracted from the range-Doppler information to obtain the target detection information corresponding to the radar sensing mode. The visible light image and the infrared image are preprocessed, and visual features and thermal features are extracted respectively to obtain the target detection information corresponding to the optical sensing mode; The multi-channel audio data is converted from time to frequency to obtain the frequency domain features of the target. The direction of arrival of the target is estimated based on the frequency domain features to obtain the target detection information corresponding to the acoustic sensing mode.

8. The scene-aware target detection method according to claim 1, characterized in that, Before determining the scene adaptation parameters corresponding to each perception modality based on the scene information and the target detection information, the method further includes: Spatial and temporal alignment are performed on each of the target detection information; wherein, the temporal alignment uses the detection time corresponding to one of the target detection information as the base time, and matches the target detection information whose difference between the detection time and the base time meets the preset time deviation condition.

9. The scene-aware target detection method according to any one of claims 1 to 8, characterized in that, The target detection information includes detection output and detection probability; the step of fusing the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each perception mode to obtain the target detection result includes: Based on the fusion weights, the detection outputs are weighted and fused to obtain the target detection output; Based on the detection uncertainty and the fusion weight, the detection probabilities are weighted and fused to obtain the fusion confidence. The target detection output and the fusion confidence score are used as the target detection result.

10. A target detection device based on scene awareness, characterized in that, include: The acquisition module is configured to acquire target detection information corresponding to multiple perception modalities, as well as scene information representing the current detection scene. The first determining module is configured to determine the scene adaptation parameters corresponding to each of the perception modes based on the scene information and the target detection information. The second determining module is configured to determine the detection uncertainty corresponding to each of the perception modes based on the target detection information corresponding to each of the perception modes; The calculation module is configured to calculate the fusion weights corresponding to each of the perception modes based on the detection uncertainty and the scene adaptation parameters corresponding to each of the perception modes. The fusion module is configured to fuse the target detection information corresponding to each of the perception modes based on the fusion weights corresponding to each of the perception modes, so as to obtain the target detection result.