A target monitoring method, device, terminal and medium

Abstract

The present invention discloses a target monitoring method, device, terminal and medium, the method comprising: based on a preset event camera, obtaining multimodal perception data, wherein the multimodal perception data includes optical modal data and radar modal data; inputting the optical modal data and the radar modal data into a multimodal fusion module for multimodal fusion, respectively, to obtain enhanced point cloud features after fusion; based on the enhanced point cloud features after fusion, determining the object bounding box and its corner points of the moving target, and determining the degree of freedom information of the moving target based on the corner points of the object bounding box, so as to realize the monitoring of the moving target. The present invention can realize the monitoring of small targets moving at high speed, and use multimodal perception fusion technology to improve the recognition accuracy and robustness of moving targets.

Description

Technical Field

[0001] The present invention relates to the field of target monitoring technology, and in particular to a target monitoring method, device, terminal and medium. Background Art

[0002] At present, the monitoring perception technology for high-speed moving small objects relies on active perception sensors and passive perception sensors to perform single-modal perception to achieve monitoring.

[0003] The active sensing technologies currently used mainly include: millimeter wave radar, Flash solid-state laser radar. Passive sensing technologies mainly include: RGB camera visual perception, infrared thermal sensing imaging and event camera. However, based on the existing active sensing technology and passive sensing technology for single-mode perception, there are still problems such as insufficient resolution, insufficient response speed, occlusion and environmental noise, which affect the accuracy of recognition and monitoring of small objects in high-speed motion. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a target monitoring method, device, terminal and medium are provided, aiming to solve the problems of the prior art of active sensing technology and passive sensing technology for single modal perception, which still have the problems of insufficient resolution, insufficient response speed, inability to handle occlusion problems and sensitivity to environmental noise.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a target monitoring method, wherein the method comprises:

[0007] Based on a preset event camera, multimodal perception data is acquired, wherein the multimodal perception data includes optical modal data collected by the event camera and radar modal data collected by a radar;

[0008] Inputting the optical modal data and the radar modal data into a multimodal fusion module for multimodal fusion, respectively, to obtain fused enhanced point cloud features;

[0009] Based on the fused enhanced point cloud features, the object bounding box of the moving target is determined, and based on the corner points of the object bounding box, the degree of freedom information of the moving target is determined to achieve monitoring of the moving target.

[0010] In one implementation, the acquiring of multimodal perception data based on a preset event camera includes:

[0011] Obtaining a trigger threshold corresponding to the event camera, wherein the trigger threshold is configured based on environmental conditions;

[0012] When the brightness change value of the environment exceeds the trigger threshold, the event camera and the preset acquisition device are triggered to collect data to obtain the multimodal perception data, wherein the acquisition device includes at least one or more of an RGB camera, an infrared camera, a millimeter-wave radar, and a laser radar;

[0013] The optical modality data also includes: optical data collected by the RGB camera and / or the infrared camera;

[0014] The radar modal data includes: radar data collected by the millimeter wave radar and / or the laser radar.

[0015] In one implementation, when the brightness change value of the environment exceeds the trigger threshold, the event camera and the preset acquisition device are triggered to collect data to obtain the multimodal perception data, including:

[0016] Monitoring the moving target based on a preset attention network;

[0017] The brightness change value of the environment after the moving target generates a motion event is obtained. When the brightness change value exceeds the trigger threshold, the event camera is triggered to capture images, event stream information is obtained, and the acquisition device is triggered to perform data acquisition.

[0018] In one implementation, the optical modality data and the radar modality data are respectively input into a multimodality fusion module for multimodality fusion to obtain fused enhanced point cloud features, including:

[0019] Performing spatial projection on the radar modality data to obtain a radar depth map, fusing the radar depth map with a motion feature edge map determined based on the optical modality data to obtain a motion feature fused depth semantic map, and performing 3D projection based on the motion feature fused depth semantic map to obtain a depth semantic enhanced motion feature;

[0020] Based on the motion feature edge map, an event point cloud feature is obtained, the radar modal data is encoded to obtain a radar point cloud feature, and the event point cloud feature is fused with the radar point cloud feature by adding and superposing to obtain a motion enhanced point cloud feature;

[0021] The depth semantic enhanced motion feature and the motion enhanced point cloud feature are subjected to weighted correction to obtain a fused enhanced point cloud feature.

[0022] In one implementation, spatially projecting the radar modality data to obtain a radar depth map, and fusing the radar depth map with a motion feature edge map determined based on the optical modality data to obtain a motion feature fused depth semantic map, includes:

[0023] The radar modal data is projected in 2D by using relative transformation external parameters between multiple sensors and internal parameters of the camera to obtain the radar depth map;

[0024] determining a motion feature edge map based on the optical modality data;

[0025] The radar depth map and the motion feature edge map are input into a cross-attention transformer model, and the radar depth map and the motion feature edge map are fused using the cross-attention mechanism in the cross-attention transformer model to obtain the motion feature fused depth semantic map.

[0026] In one implementation, obtaining event point cloud features based on the motion feature edge map includes:

[0027] Projecting the motion feature edge map into 3D to obtain 3D back-projected event features of the spatiotemporal point cloud;

[0028] The 3D back-projection event features are encoded to obtain the event point cloud features.

[0029] In one implementation, determining a motion feature edge map based on the optical modality data includes:

[0030] The RGB image captured by the RGB camera and / or the image captured by the infrared camera and the event stream information captured by the event camera are projected and feature extracted to perform weighted correction on a 2D picture to obtain the motion feature edge map.

[0031] In one implementation, determining the object bounding box of the moving target based on the fused enhanced point cloud features includes:

[0032] The fused enhanced point cloud features and target query vector are input into Transformer to obtain the 2D target box and label information of the moving target;

[0033] Based on the 2D target frame, the label information and a preset shape model, combined with a monocular depth estimation network, the object bounding box of the moving target is obtained by joint optimization.

[0034] In one implementation, the object bounding box of the moving target is obtained by joint optimization based on the 2D target box, the label information and the preset shape model in combination with a monocular depth estimation network, including:

[0035] Determining category information of the moving target based on the tag information, and acquiring a shape model corresponding to the category information;

[0036] A deep learning model is used to perform depth estimation on a monocular image, and the shape model and the 2D target box are used as prior knowledge. Through geometric transformation and optimization algorithms, the 2D target box is promoted to a 3D space, and the object bounding box of the moving target is jointly optimized.

[0037] In one implementation, determining the degree of freedom information of the moving target based on the corner points of the object bounding box includes:

[0038] Based on the corner points of the object bounding box, an iterative closest point algorithm is used to determine the degree of freedom information of the moving target, where the degree of freedom information includes: three-dimensional coordinates and a rotation matrix in a three-dimensional scene.

[0039] In a second aspect, an embodiment of the present invention further provides a target monitoring device, wherein the device comprises:

[0040] A multimodal data acquisition module, used to acquire multimodal perception data based on a preset event camera, wherein the multimodal perception data includes optical modal data acquired by the event camera and radar modal data acquired by a radar;

[0041] A multimodal data fusion module, used for inputting the optical modal data and the radar modal data into the multimodal fusion module for multimodal fusion, to obtain enhanced point cloud features after fusion;

[0042] The moving target recognition and monitoring module is used to determine the object bounding box of the moving target based on the fused enhanced point cloud features, and to determine the degree of freedom information of the moving target based on the corner points of the object bounding box, so as to realize the monitoring of the moving target.

[0043] In a third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a target monitoring program stored in the memory and executable on the processor, and when the processor executes the target monitoring program, the steps of the target monitoring method of any one of the above-mentioned schemes are implemented.

[0044] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a target monitoring program is stored on the computer-readable storage medium, and when the target monitoring program is executed by a processor, the steps of the target monitoring method described in any one of the above schemes are implemented.

[0045] Beneficial effects: Compared with the prior art, the present invention provides a target monitoring method. The present invention first obtains multimodal perception data based on a preset event camera. The multimodal perception data of the present invention includes optical modal data collected by the event camera and radar modal data collected by the radar. Since the present invention adopts a new sensor and collects multimodal perception data, it provides a possibility for accurately identifying and monitoring high-speed moving targets. Then, the present invention inputs the optical modal data and the radar modal data into the multimodal fusion module for multimodal fusion, and obtains the enhanced point cloud features after fusion. Finally, based on the enhanced point cloud features after fusion, the object bounding box of the moving target is determined, and based on the corner points of the object bounding box, the degree of freedom information of the moving target is determined to realize the monitoring of the moving target. The present invention uses an event camera to collect optical modal data of small objects moving at high speed, and by fusing the optical modal data and the radar modal data, the resolution of identifying small objects moving at high speed is improved, the subtle movements and details of small objects moving at high speed are effectively captured and tracked, and the problem of missing or misidentifying targets is effectively avoided. Furthermore, since small objects moving at high speed may quickly pass through the field of view or be blocked by other objects, single visual perception technology may not be able to effectively handle the occlusion of small objects moving at high speed during movement. Some single modality perception technologies may be sensitive to noise in the environment and interfere with the detection and tracking of small objects moving at high speed. The present invention identifies and monitors small objects moving at high speed by fusing optical modality data and radar modality data, thereby effectively solving the problems of being unable to detect small objects moving at high speed when they are blocked during movement and the presence of environmental noise during detection, thereby improving recognition accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a preferred embodiment of the target monitoring method provided in the embodiment of the present invention.

[0047] Figure 2 A flowchart of the actual application of the target monitoring method provided by an embodiment of the present invention.

[0048] Figure 3 A schematic diagram of the architecture of a target monitoring device provided in an embodiment of the present invention.

[0049] Figure 4 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations or steps, nor must they be executed in the order described. For example, some operations or steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0052] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first control information and the second control information are only used to distinguish different control information, and their order is not limited.

[0054] Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the differences.

[0055] It should be further understood that the term “and / or” used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0056] The present embodiment provides a target monitoring method, which can effectively solve the problem that a small object moving at high speed cannot be detected when it is blocked and there is environmental noise during detection, and improves the recognition accuracy and robustness. In specific application, the present embodiment first obtains multimodal perception data based on a preset event camera. The multimodal perception data of the present embodiment includes optical modal data collected by the event camera and radar modal data collected by the radar. Since the present embodiment adopts a new sensor and collects multimodal perception data, it is possible to accurately identify and monitor high-speed moving targets. Then, the present embodiment inputs the optical modal data and the radar modal data into a multimodal fusion module for multimodal fusion, and obtains the enhanced point cloud features after fusion. Finally, based on the enhanced point cloud features after fusion, the object bounding box of the moving target is determined, and based on the corner points of the object bounding box, the degree of freedom information of the moving target is determined to realize the monitoring of the moving target. The present invention identifies small objects moving at high speed from multi-dimensional perception data by fusing optical modal data and radar modal data.

[0057] The target monitoring method of this embodiment can be applied to a terminal, which can be a computer, a smart TV, a mobile phone or other intelligent product terminal. Figure 1 As shown in , the target monitoring method includes the following steps:

[0058] Step S100: acquiring multimodal perception data based on a preset event camera, wherein the multimodal perception data includes optical modal data collected by the event camera and radar modal data collected by a radar.

[0059] This embodiment uses an event camera as a starter for the rough recognition of small, fast-moving targets. The working principle of the event camera is to associate the readout time of the sensor pixels with the change in light intensity. When the light intensity change exceeds the set trigger threshold, the pixel will be read out immediately, output as a vector form of time, pixel coordinates, and event polarity and stored. This enables the event camera to record the state of each pixel at a very fast speed and play it in chronological order to show the observed events.

[0060] Since the movement of moving targets will inevitably lead to changes in pixel brightness within the field of view, the event camera will be triggered to generate event stream information on the corresponding pixels. Driven by event stream information, millimeter-wave radars and other sensing devices (such as RGB cameras or lidars) can be guided to collect and generate multimodal sensing data. Compared with traditional methods, event cameras have higher output frame rates, lower power consumption, and have the advantages of fast response and wide dynamic range. They can perform 3D real-time reconstruction, 6-degree-of-freedom tracking, and accurate angular velocity estimation.

[0061] Based on this, the present embodiment can pre-configure the trigger threshold of the event camera according to the environmental conditions. The trigger threshold in the event camera is an important parameter. When the trigger threshold is low, the signal in the image collected by the event camera is more obvious, but there are more noise events. When weak target observation is performed, a trigger threshold that is too low can easily cause noise events and signal events to be indistinguishable. As the trigger threshold increases, the system detection sensitivity decreases, the signal is not obvious, and signal recognition is affected. Therefore, it is necessary to use different trigger thresholds in different environments to compare the output results and find the corresponding trigger thresholds suitable for different low-altitude environments. For example, the threshold in rainy and snowy weather is inconsistent with the trigger threshold in clear weather.

[0062] Combination Figure 2 As shown, this embodiment obtains the trigger threshold corresponding to the event camera, and then compares the brightness change value of the environment with the trigger threshold to determine whether there is an event trigger. When the brightness change value of the environment exceeds the trigger threshold, the event camera and the preset acquisition device are triggered to collect data to obtain the multimodal perception data. The specific formula is as follows:

[0063] like , the event camera will start to collect data and collect event stream information (event). At the same time, the event camera will also output event signals to trigger other preset collection devices to collect data, thereby obtaining a single frame of multimodal perception data. In the above formula, u is a pixel at a certain point of the event camera, t is time, and d is the time. t To represent a small time interval, Int(u, t) is the brightness gain at pixel u at time t, Int(u, t+d t ) is t+d t The brightness gain at pixel u at the moment, C th is the trigger threshold. When the brightness change value exceeds the set trigger threshold C th When the event signal is output, it is in the form of a vector of time, pixel coordinates, and event polarity.

[0064] The event camera is sensitive to the brightness change caused by the motion of a moving target (such as an object), and will generate corresponding event stream information, which is reflected in the image as the change in brightness at the relevant pixel on the image. When the brightness change value reaches the trigger threshold, the event camera starts the image acquisition, and there will be event information output, while triggering data acquisition of other acquisition devices. Based on this, this embodiment can design an attention network of an event camera to focus on supervising the image near the pixel that generates event information. Specifically, the attention network of this embodiment can monitor the moving target in the acquisition area in real time. When the moving target generates a motion event, the brightness of the environment changes, and when the brightness change value of the environment exceeds the trigger threshold set by the event camera, the event camera will start to collect the image corresponding to the motion event, thereby obtaining event stream information. If the brightness change value of the environment does not exceed the trigger threshold, the attention network can continue to monitor the motion event of the moving target, and compare the brightness change value of the environment collected again with the trigger threshold. This embodiment can effectively enhance the detection capability of moving targets based on real-time monitoring of the attention network. The present invention collects event stream information through an event camera and fuses it with radar data in subsequent steps, thereby improving the resolution of identifying small objects moving at high speed, effectively capturing and tracking the subtle movements and details of small objects moving at high speed, and effectively avoiding the problem of missing or misidentifying targets.

[0065] In this embodiment, the output information of the event camera cannot be used directly and needs to be processed. Therefore, this embodiment performs event stream motion compensation on the output results of the event camera to find the point trajectory on the pixel plane that best matches the event data, and generates a motion compensated event image with clear edges to obtain more accurate event stream information.

[0066] Furthermore, the acquisition device of this embodiment includes at least one or more of an RGB camera, an infrared camera, a millimeter wave radar, and a laser radar. Figure 2 As shown, the optical modal data may include not only the event stream information collected by the event camera, but also the optical data collected by the RGB camera and / or the infrared camera, which is usually a 2D picture. The RGB camera is a camera device that uses three basic color channels of red, green and blue to capture images. The visual perception technology can use the RGB camera to obtain image or video information of the environment around the aircraft, and realize the perception and recognition of obstacles, terrain and waterways by processing and analyzing the images. The infrared camera can use the infrared radiation radiated by the target object for detection and perception. By detecting the intensity and frequency of the infrared radiation, the temperature, shape and movement of the target object can be obtained. Compared with the visible light camera technology, the thermal perception imaging of the infrared camera can be observed in the absence of light or low light conditions, and has the characteristics of penetrating haze, smoke and dust. It can obtain the temperature distribution and thermal characteristics of the target by detecting the thermal radiation of the object, thereby realizing many applications. Similarly, the radar modal data includes: radar data collected by the millimeter wave radar and / or the laser radar, which is usually 3D point cloud data. The millimeter wave radar can be a common 4D millimeter wave radar, which can obtain information about the target object in four dimensions, namely, measure and analyze the distance, direction, height and speed, and can effectively analyze the outline, category and behavior of the target object. The laser radar can be a Flash solid-state laser radar, which can quickly emit a large coverage area of ​​laser, and then use a highly sensitive receiver to complete the drawing of the surrounding environment.

[0067] Step S200: input the optical modal data and the radar modal data into a multimodal fusion module respectively for multimodal fusion to obtain fused enhanced point cloud features.

[0068] Furthermore, the event stream information output by the event camera and the radar modal data obtained from the millimeter-wave radar are respectively input into multimodal fusion for multimodal fusion. Since the millimeter-wave radar has accurate speed and long-distance perception capabilities, and the event camera can provide rich motion clues, the fusion of event information and point cloud data can effectively distinguish between moving objects and static backgrounds, thereby adding another layer of dense supervision around the object of interest, which helps to realize the recognition and monitoring of high-speed moving targets.

[0069] Furthermore, after collecting radar data, the millimeter wave radar of this embodiment can output a sampling matrix by modulating the continuous waveform, and then by performing fast Fourier transformation on multiple groups of sampling matrix data, a 4D tensor of the corresponding point can be obtained, including distance, heading angle, pitch angle and Doppler velocity. Finally, some abnormal outputs are eliminated by the Constant False AlarmRate (CFAR) algorithm to obtain the radar modal data of the millimeter wave radar. Since the output of the millimeter wave radar (such as a Flash solid-state laser radar) can be presented in the form of a point cloud map, their point cloud map is directly used as the input (x, y, z, t) during multi-modal fusion, where x, y, z are the coordinates of the moving target collected by the millimeter wave radar, and t is the collection time. In another embodiment, if this embodiment also includes a laser radar, the radar data collected by the laser radar and the radar data collected by the millimeter wave radar can be fused first to obtain radar modal data.

[0070] After the optical modality data and the radar modality data are respectively input into the multimodal fusion module, they can be fused through the multimodal fusion technology. In this embodiment, the radar modality data (i.e., 3D point cloud data) can be spatially projected to obtain a radar depth map (i.e., 2D map), that is, a 3D to 2D projection is realized. Then, the radar depth map is fused with the motion feature edge map determined based on the optical modality data to obtain a motion feature fusion depth semantic map. Specifically, in this embodiment, the radar modality data is first 2D projected through the relative transformation external parameters between multiple sensors and the internal parameters of the camera to obtain a 2D radar depth map, and then the motion feature edge map is determined based on the optical modality data. Next, the radar depth map and the motion feature edge map are input into the Transformer, and the cross-attention mechanism in the Transformer is used to perform feature fusion of the radar depth map and the motion feature edge map to obtain the motion feature fusion depth semantic map.

[0071] In another implementation, when the acquisition device includes an RGB camera and / or an infrared camera, this embodiment can also fuse the RGB camera and / or the infrared camera with the event camera. Therefore, the RGB image collected by the RGB camera and / or the image collected by the infrared camera can be weighted corrected with the 2D picture after projection and feature extraction of the event stream information collected by the event camera to obtain a fused motion feature edge map. Finally, this embodiment performs 3D projection based on the motion feature fused depth semantic map to obtain deep semantic enhanced motion features. In another channel, this embodiment can obtain event point cloud features based on the motion feature edge map. Specifically, this embodiment projects the motion feature edge map to 3D to obtain 3D reverse projection event features of the spatiotemporal point cloud, that is, to achieve reverse projection from 2D to 3D. Then, an encoder (Transformer or ordinary CNN) is used to encode the 3D reverse projection event features to obtain the event point cloud features. Then, this embodiment encodes the radar modal data (i.e., 3D point cloud data) through an encoder (Transformer or ordinary CNN) to obtain radar point cloud features, and uses the add superposition method to fuse the event point cloud features and the radar point cloud features to obtain motion enhanced point cloud features.

[0072] Furthermore, after obtaining the deep semantic enhanced motion features and the motion enhanced point cloud features, this embodiment performs weighted correction on the deep semantic enhanced motion features and the motion enhanced point cloud features through a weight add (weight superposition algorithm) to obtain a fused enhanced point cloud feature.

[0073] Step S300: determine the object bounding box of the moving target based on the fused enhanced point cloud features, and determine the degree of freedom information of the moving target based on the corner points of the object bounding box to achieve monitoring of the moving target.

[0074] Specifically, this embodiment uses the fused enhanced point cloud features and object queries (target query vector) as the input of the Transformer, and inputs them into the Transformer to obtain the 2D target box and label information (i.e., label information) of the moving target. Then, based on the 2D target box, the label information and the preset shape model, combined with the monocular depth estimation network, the object bounding box (i.e., 3D bounding box) of the moving target is jointly optimized. This embodiment can pre-construct the corresponding shape model (shape model) for the category information corresponding to the label information. Then, based on the label information output by the Transformer at this time, the category information of the moving target is determined, and the shape model corresponding to the category information is obtained. Then, the monocular image is depth estimated using a deep learning model, and the shape model shape model and the 2D target box output by the Transformer are used as prior knowledge. Through geometric transformation and optimization algorithms, the 2D target box is promoted to 3D space, and the object bounding box of the moving target is jointly optimized.

[0075] After obtaining the object bounding box of the moving target, this embodiment can use an iterative closest point algorithm (Iterative Closest Point) based on the corner points of the object bounding box to determine the degree of freedom information of the moving target. The degree of freedom information includes: three-dimensional coordinates in a three-dimensional scene (i.e., the x-, y-, and z-axis coordinates of the moving target) and a rotation matrix (i.e., the rotation matrix roll, pitch, and yaw). By determining the degree of freedom information of the moving target, the moving target can be identified and monitored.

[0076] It can be seen that the present embodiment uses an event camera to collect optical modal data of high-speed moving small objects, and identifies and monitors high-speed moving small objects by fusing optical modal data and radar modal data, thereby improving the resolution of identifying high-speed moving small objects, effectively capturing and tracking the subtle movements and details of high-speed moving small objects, and effectively avoiding the problem of missing or misidentifying targets. In addition, since high-speed small objects may quickly pass through the field of view or be blocked by other objects, a single visual perception technology may not be able to effectively handle the occlusion of high-speed small objects during movement. Some single modal perception technologies may be sensitive to noise in the environment, which may interfere with the detection and tracking of high-speed small objects. The present embodiment identifies high-speed moving small objects by fusing multi-dimensional data such as optical modal data and radar modal data, effectively solving the problem of being unable to detect high-speed moving small objects when they are blocked during movement and the presence of environmental noise during detection, thereby improving recognition accuracy and robustness.

[0077] Based on this, this embodiment can be applied to any field or scenario where it is necessary to identify and monitor moving targets. For example, it can be applied to ground-based air detection and monitoring systems to identify the flight status of low-altitude aircraft, especially to effectively monitor non-cooperative aircraft. It can also be applied to airborne air detection systems to identify dynamic and static obstacles around aircraft, and assist aircraft in positioning and obstacle avoidance.

[0078] Based on the above embodiments, the present invention also provides a target monitoring device, such as Figure 3 As shown in , the device includes: a multimodal data acquisition module 10, a multimodal data fusion module 20 and a moving target recognition and monitoring module 30. Specifically, the multimodal data acquisition module 10 is used to obtain multimodal perception data based on a preset event camera, wherein the multimodal perception data includes optical modal data collected by the event camera and radar modal data collected by the radar. The multimodal data fusion module 20 is used to input the optical modal data and the radar modal data into the multimodal fusion module respectively for multimodal fusion to obtain the fused enhanced point cloud features. The moving target recognition and monitoring module 30 is used to determine the object bounding box of the moving target based on the fused enhanced point cloud features, and determine the degree of freedom information of the moving target based on the corner points of the object bounding box, so as to realize the monitoring of the moving target.

[0079] The working principles of each module in the target monitoring device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0080] Each module in the above target monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of the processor in the terminal in the form of hardware, or can be stored in the memory in the terminal in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0081] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 4 The terminal may include one or more processors 100 ( Figure 4 The processor 100 includes a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a sleep analysis program based on multi-sensor data. When the one or more processors 100 execute the computer program 102, the various steps in the embodiment of the sleep analysis method based on multi-sensor data can be implemented. Alternatively, when the one or more processors 100 execute the computer program 102, the functions of the various modules / units in the embodiment of the sleep analysis device based on multi-sensor data can be implemented, which is not limited here.

[0082] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0083] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0084] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0085] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target monitoring method, characterized in that: The method comprises: Based on a preset event camera, multimodal perception data is acquired, wherein the multimodal perception data includes optical modal data collected by the event camera and radar modal data collected by a radar; Inputting the optical modal data and the radar modal data into a multimodal fusion module for multimodal fusion, respectively, to obtain fused enhanced point cloud features; Based on the fused enhanced point cloud features, an object bounding box of the moving target is determined, and based on the corner points of the object bounding box, degree of freedom information of the moving target is determined to achieve monitoring of the moving target; The optical modal data and the radar modal data are respectively input into a multimodal fusion module for multimodal fusion to obtain fused enhanced point cloud features, including: Performing spatial projection on the radar modality data to obtain a radar depth map, fusing the radar depth map with a motion feature edge map determined based on the optical modality data to obtain a motion feature fused depth semantic map, and performing 3D projection based on the motion feature fused depth semantic map to obtain a depth semantic enhanced motion feature; Based on the motion feature edge map, an event point cloud feature is obtained, the radar modal data is encoded to obtain a radar point cloud feature, and the event point cloud feature is fused with the radar point cloud feature by adding and superposing to obtain a motion enhanced point cloud feature; The depth semantic enhanced motion feature and the motion enhanced point cloud feature are subjected to weighted correction to obtain a fused enhanced point cloud feature.

2. The target monitoring method according to claim 1, characterized in that: The method of acquiring multimodal perception data based on a preset event camera includes: Obtaining a trigger threshold corresponding to the event camera, wherein the trigger threshold is configured based on environmental conditions; When the brightness change value of the environment exceeds the trigger threshold, the event camera and the preset acquisition device are triggered to collect data to obtain the multimodal perception data, wherein the acquisition device includes at least one or more of an RGB camera, an infrared camera, a millimeter-wave radar, and a laser radar; The optical modality data also includes: optical data collected by the RGB camera and / or the infrared camera; The radar modal data includes: radar data collected by the millimeter wave radar and / or the laser radar.

3. The target monitoring method according to claim 2, characterized in that: When the brightness change value of the environment exceeds the trigger threshold, the event camera and the preset acquisition device are triggered to collect data to obtain the multimodal perception data, including: Monitoring the moving target based on a preset attention network; The brightness change value of the environment after the moving target generates a motion event is obtained. When the brightness change value exceeds the trigger threshold, the event camera is triggered to capture images, event stream information is obtained, and the acquisition device is triggered to perform data acquisition.

4. The target monitoring method according to claim 1, characterized in that: The step of spatially projecting the radar modality data to obtain a radar depth map, and fusing the radar depth map with a motion feature edge map determined based on the optical modality data to obtain a motion feature fused depth semantic map comprises: The radar modal data is projected in 2D by using relative transformation external parameters between multiple sensors and internal parameters of the camera to obtain the radar depth map; determining a motion feature edge map based on the optical modality data; The radar depth map and the motion feature edge map are input into a cross-attention transformer model, and the radar depth map and the motion feature edge map are fused using the cross-attention mechanism in the cross-attention transformer model to obtain the motion feature fused depth semantic map.

5. The target monitoring method according to claim 1, characterized in that: The obtaining of event point cloud features based on the motion feature edge map includes: Projecting the motion feature edge map into 3D to obtain 3D back-projected event features of the spatiotemporal point cloud; The 3D back-projection event features are encoded to obtain the event point cloud features.

6. The target monitoring method according to claim 4, characterized in that: The determining of a motion feature edge map based on the optical modality data comprises: The RGB image captured by the RGB camera and / or the image captured by the infrared camera and the event stream information captured by the event camera are projected and feature extracted to perform weighted correction on the 2D picture to obtain the motion feature edge map.

7. The target monitoring method according to claim 1, characterized in that: The step of determining the object bounding box of the moving target based on the fused enhanced point cloud features includes: The fused enhanced point cloud features and target query vector are input into Transformer to obtain the 2D target box and label information of the moving target; Based on the 2D target frame, the label information and a preset shape model, combined with a monocular depth estimation network, the object bounding box of the moving target is obtained by joint optimization.

8. The target monitoring method according to claim 7, characterized in that: The object bounding box of the moving target is obtained by joint optimization based on the 2D target box, the label information and the preset shape model in combination with a monocular depth estimation network, including: Determining category information of the moving target based on the tag information, and acquiring a shape model corresponding to the category information; A deep learning model is used to perform depth estimation on a monocular image, and the shape model and the 2D target box are used as prior knowledge. Through geometric transformation and optimization algorithms, the 2D target box is promoted to a 3D space, and the object bounding box of the moving target is jointly optimized.

9. The target monitoring method according to claim 1, characterized in that: The determining the degree of freedom information of the moving target based on the corner points of the object bounding box includes: Based on the corner points of the object bounding box, an iterative closest point algorithm is used to determine the degree of freedom information of the moving target, where the degree of freedom information includes: three-dimensional coordinates and a rotation matrix in a three-dimensional scene.

10. A target monitoring device, characterized in that: The device comprises: A multimodal data acquisition module, used to acquire multimodal perception data based on a preset event camera, wherein the multimodal perception data includes optical modal data acquired by the event camera and radar modal data acquired by a radar; A multimodal data fusion module, used for inputting the optical modal data and the radar modal data into the multimodal fusion module for multimodal fusion, to obtain enhanced point cloud features after fusion; A moving target recognition and monitoring module is used to determine the object bounding box of the moving target based on the fused enhanced point cloud features, and to determine the degree of freedom information of the moving target based on the corner points of the object bounding box, so as to realize the monitoring of the moving target; The multimodal data fusion module is specifically used for: Performing spatial projection on the radar modality data to obtain a radar depth map, fusing the radar depth map with a motion feature edge map determined based on the optical modality data to obtain a motion feature fused depth semantic map, and performing 3D projection based on the motion feature fused depth semantic map to obtain a depth semantic enhanced motion feature; Based on the motion feature edge map, an event point cloud feature is obtained, the radar modal data is encoded to obtain a radar point cloud feature, and the event point cloud feature is fused with the radar point cloud feature by adding and superposing to obtain a motion enhanced point cloud feature; The depth semantic enhanced motion feature and the motion enhanced point cloud feature are subjected to weighted correction to obtain a fused enhanced point cloud feature.

11. A terminal, characterized in that: The terminal includes a memory, a processor, and a target monitoring program stored in the memory and executable on the processor. When the processor executes the target monitoring program, the steps of the target monitoring method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target monitoring program, and when the target monitoring program is executed by the processor, the steps of the target monitoring method according to any one of claims 1 to 9 are implemented.

Images