A scene flow estimation method, system and application based on multi-modal information

Abstract

The application discloses a scene flow estimation method and system based on multi-modal information and application. The method comprises the following steps: acquiring asynchronous event stream data of brightness change of each pixel in a target scene and rasterizing the event stream data into event voxels in a fixed time unit, and processing the event voxels to obtain an edge optical flow; acquiring a plurality of frame images in frame image data of the target scene, and obtaining a global optical flow according to a corresponding frame image sequence; coupling the edge optical flow and the global optical flow to obtain an optimized global optical flow of an edge region; acquiring distance data of the target scene, and projecting the depth information after conversion in combination with the optimized global optical flow to obtain a 3D scene flow; acquiring inertial navigation information, and calculating a 3D rigid body flow according to the inertial navigation information and the depth information; and coupling the 3D scene flow and the 3D rigid body flow to obtain a target scene flow. The method realizes accurate estimation of 3D scene flow motion and accurate simulation of complete scene motion of automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of multimodal systems and visual motion estimation technology, and more specifically, relates to a scene flow estimation method, system and application based on multimodal information. Background Technology

[0002] Autonomous driving is a hot industry right now, and one of the key technologies in autonomous driving is scene motion information perception, which aims to react accordingly based on motion information in the scene. This key technology has two key problems to solve: the acquisition of scene information and the estimation of scene motion. The former is information collection, and the latter is information processing.

[0003] Numerous autonomous driving scene information acquisition systems and datasets have emerged, but few can be applied to scene motion estimation. KITTI is a representative example. Founded jointly by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America, KITTI includes sensors such as LiDAR, stereo frame cameras, and integrated inertial navigation systems, covering major autonomous driving scenarios including urban, suburban, and street scenes. However, the KITTI acquisition system struggles to capture motion information in nighttime scenes and fails to reflect the complex and ever-changing nature of autonomous driving scenarios.

[0004] Motion estimation methods in information processing include optical flow estimation and scene flow estimation. Compared to optical flow estimation, scene flow estimation better reflects the motion of the scene in the objective world. Scene flow motion estimation remains an open problem, and there are many ways to estimate scene flow. Existing scene flow estimation methods mainly estimate optical flow first, and then project it into the 3D scene through depth. However, this method has two problems. On the one hand, in nighttime scenes and scenes with fast motion, existing optical flow estimation methods often fail to accurately estimate motion edges due to texture loss and motion blur. On the other hand, for targets with significant radial motion, the optical flow estimated by existing methods does not contain accurate radial motion, resulting in the estimated scene flow failing to reflect radial motion.

[0005] Therefore, both information acquisition and processing face significant challenges in motion estimation for nighttime and high-speed scenarios. In autonomous driving motion scene estimation, inaccurate scene flow motion edges and a lack of radial motion prevent existing optical flow estimations from accurately reflecting the complete scene motion. Summary of the Invention

[0006] To address the shortcomings of related technologies, the present invention aims to provide a scene flow estimation method, system, and application based on multimodal information, which seeks to solve the problems of motion degradation and edge blurring in all-day autonomous driving scene flow estimation, and the inability of existing optical flow estimation to accurately reflect the motion of the complete scene.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a scene flow estimation method based on multimodal information, comprising:

[0008] Asynchronous event stream data of brightness change of each pixel in the target scene is acquired and rasterized into event voxels in a fixed time unit. The event voxels are then processed to obtain edge optical flow.

[0009] Acquire several frames from the frame image data of the target scene, and obtain the global optical flow based on the corresponding frame image sequence;

[0010] The edge optical flow and the global optical flow are coupled to obtain the global optical flow optimized for the edge region;

[0011] The distance data to the scene target is obtained, and the 3D scene flow is obtained by combining the transformed depth information with the global optical flow projection optimized for edge regions.

[0012] Acquire inertial navigation information, and calculate 3D rigid body flow based on the inertial navigation information and the depth information;

[0013] The target scene flow is obtained by coupling the 3D scene flow and the 3D rigid body flow.

[0014] Optionally, the step of coupling the edge optical flow and the global optical flow to obtain the edge region-optimized global optical flow includes:

[0015] The edge optical flow and the global optical flow are coupled through a motion edge constraint module;

[0016] The motion edge of the global optical flow is optimized by applying an optical flow consistency loss to align the motion edge regions of the edge optical flow and the global optical flow; the motion consistency formula is:

[0017]

[0018] Among them, F frame Indicates the global optical flow corresponding to the frame camera, F event V represents the edge optical flow corresponding to the event camera, and V represents the valid motion edge region, which is obtained through forward and backward consistency detection of edge optical flow.

[0019] Optionally, calculating the 3D rigid body flow based on the inertial navigation information and the depth information includes:

[0020] Obtain the projected coordinates of pixel p1 in the world coordinate system under image frame I1:

[0021] x1=D(p1)K -1 p1;

[0022] Where p1 represents the 2D homogeneous coordinates of a pixel in image frame I1, K represents the calibrated camera intrinsic parameters, and D(p1) is the depth information of pixel p1.

[0023] The geometric motion calculation module based on spatial kinematics calculates the coordinates of the corresponding world coordinate system in the I2 image frame:

[0024] x2=PD(p1)K -1 p1

[0025] The 3D rigid body flow is calculated based on the coordinates x2 and x1 of pixel p1 in the world coordinate system under image frame I2.

[0026] F rigid (x1) = x2 - x1.

[0027] Optionally, the coupling of the 3D scene flow and the 3D rigid body flow to obtain the scene flow includes:

[0028] The 3D scene flow and the 3D rigid body flow are coupled through a 3D motion field constraint module;

[0029] The target scene flow is obtained by aligning the radial motion of the 3D scene flow and the 3D rigid body flow in the rigid body region M by applying a 3D motion consistency loss; the formula for calculating the radial motion is:

[0030]

[0031] Among them, F rigid For 3D rigid body flow, F scene For 3D scene flow.

[0032] Optionally, before acquiring asynchronous event stream data of brightness changes per pixel in the target scene and rasterizing it into event voxels in a fixed time unit, and processing the event voxels to obtain edge optical flow, the method further includes:

[0033] A microcontroller generates a preset frequency synchronous pulse square wave to control the alignment of timestamps of different modal sensors.

[0034] Align and calibrate the coordinate systems between any two different sensors.

[0035] Secondly, the present invention also provides a scene flow estimation system based on multimodal information, comprising: a microcontroller and an industrial control computer, and modal sensors connected to both, wherein the modal sensors include an event camera, a frame camera, a lidar and an IMU sensor;

[0036] When the acquisition system is in motion, the microcontroller is used to control the alignment of timestamps of different modal sensors;

[0037] The event camera is used to record asynchronous event stream data of brightness changes per pixel;

[0038] The frame camera is used to acquire frame image data of dynamic scenes;

[0039] The lidar is used to acquire distance data to targets in the scene;

[0040] The IMU sensor is used to acquire inertial navigation information, which includes angular velocity data from the gyroscope and acceleration data from the accelerometer.

[0041] The industrial control computer is used to acquire data from the modal sensor, process the event stream data to obtain edge optical flow, process the dynamic scene frame image data to obtain global optical flow, and couple the edge optical flow and the global optical flow to obtain edge-optimized global optical flow; based on the distance data, the edge-optimized global optical flow is used to obtain 3D scene flow through a projection module, and based on the distance data and the inertial navigation information, 3D rigid body flow is obtained through geometric motion calculation, and the 3D scene flow and the 3D rigid body flow are coupled to obtain target scene flow.

[0042] Optionally, the microcontroller can be connected to the external trigger interfaces of different modal sensors via PWM IO ports to control the synchronization of the acquisition timing of the modal sensors.

[0043] Optionally, the industrial computer is also used to align and calibrate the coordinate systems between any two different sensors.

[0044] Thirdly, the present invention also provides an autonomous vehicle equipped with a scene flow estimation system based on multimodal information as described in any of the second aspects.

[0045] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0046] 1. The present invention provides a scene flow estimation system based on multimodal information. The system can simultaneously acquire motion information of night scenes and high-speed motion scenes through an event camera and acquire motion information of regular scenes through a frame camera. The system uses a microcontroller to generate a fixed frequency synchronous pulse square wave to control the timestamp alignment of different modal sensors, thereby realizing the synchronous acquisition of multimodal information by the acquisition system.

[0047] 2. This invention provides a scene flow estimation method based on multimodal information. Based on synchronously acquired time-aligned multimodal information, it couples edge optical flow and global optical flow through a motion edge constraint module, thereby optimizing the motion edge region of global optical flow and solving the problem that motion edges are difficult to estimate accurately due to texture loss and motion blur. It couples 3D rigid body flow and 3D scene flow through a 3D motion field constraint module, optimizing the radial motion of the 3D scene flow obtained based on optical flow in the rigid body region. This solves the problem that the optical flow estimated by existing methods does not contain accurate radial motion, which makes it difficult for the estimated scene flow to reflect radial motion. This method achieves accurate estimation of 3D scene flow motion and accurately simulates the complete scene motion of autonomous driving. Attached Figure Description

[0048] Figure 1 A flowchart illustrating a scene flow estimation method based on multimodal information provided in an embodiment of the present invention;

[0049] Figure 2 A flowchart illustrating a scene flow estimation method based on multimodal information provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of a scene flow estimation system based on multimodal information provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0052] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0053] Example 1

[0054] like Figure 1 As shown, a scene flow estimation method based on multimodal information includes:

[0055] S1. Obtain asynchronous event stream data of brightness change of each pixel in the target scene and rasterize it into event voxels in a fixed time unit. Process the event voxels to obtain edge optical flow.

[0056] S2. Obtain several frames from the dynamic scene frame image data of the target scene, and obtain the global optical flow based on the corresponding frame image sequence;

[0057] S3. Couple the edge optical flow and the global optical flow to obtain the global optical flow optimized for the edge region;

[0058] S4. Obtain distance data to the scene target, and obtain the 3D scene flow by combining the converted depth information with the global optical flow projection optimized for the edge region;

[0059] S5. Obtain inertial navigation information and calculate 3D rigid body flow based on inertial navigation information and depth information;

[0060] S6. Couple the 3D scene flow and the 3D rigid body flow to obtain the target scene flow.

[0061] Optionally, before S1, the following are also included:

[0062] A microcontroller generates a preset frequency synchronous pulse square wave to control the alignment of timestamps of different modal sensors.

[0063] Align and calibrate the coordinate systems between any two different sensors.

[0064] To ensure the accuracy of the estimated scene flow, it is necessary to ensure that the acquired data is synchronized in time. This can be achieved by using a microcontroller to generate a fixed-frequency synchronization pulse square wave to control the timestamp alignment of different modal sensors. Specifically, the LiDAR information acquisition frequency is 20Hz, the event camera acquisition frequency is 1MHz, the frame camera acquisition frequency is 50Hz, and the IMU sensor acquisition frequency is 500Hz. Therefore, the microcontroller uses 1MHz to ensure that the inherent acquisition clocks of all sensors can be aligned with the timestamps under the entire 1MHz external trigger frequency.

[0065] The acquisition system also needs to be calibrated and registered, and the coordinate systems between any two different sensors need to be aligned and calibrated. Specifically, the registration between the event camera and the lidar is refined using the ICP algorithm, the registration between cameras is done using the checkerboard calibration method, the registration between the event camera and the frame camera is done using the checkerboard calibration method based on the reconstructed image, and the IMU calibration mainly involves the temperature drift calibration of the gyroscope and accelerometer.

[0066] See Figure 2The industrial control computer performs real-time shooting and storage of outdoor driving scenes, reads event stream data from event cameras, and rasterizes it into event voxels in fixed time units of 5 milliseconds. It also reads several frames from the dynamic scene frame image data of the frame camera; distance data is obtained when a LiDAR hits a target in the scene, and this distance data is read and converted into relative depth through normalization; inertial navigation information is read from the IMU sensor, including angular velocity data from the gyroscope and acceleration data from the accelerometer, and angular and directional velocities are calculated using quaternions and integration to form inertial navigation information. Optical flow is the 2D motion field projected from the objective world coordinate system onto the camera coordinate system. Optical flow directly based on frame images often exhibits abnormal motion edges in special scenarios such as high-speed motion and nighttime scenes. Event cameras, with their wide dynamic range and high temporal resolution, can compensate for these problems with optical flow in these scenarios. The edge optical flow of the event camera and the global optical flow of the frame camera are estimated using an optical flow estimator. The edge optical flow and the global optical flow are coupled in the motion edge constraint module to optimize the motion edge region of the global optical flow. Then, the edge-optimized global optical flow is converted into a 3D scene flow in the projection module by depth. Finally, the depth information and inertial navigation information are used to obtain the 3D rigid body flow through the geometric motion calculation module, and it is coupled with the 3D scene flow in the 3D motion field constraint module to further optimize the radial motion in the 3D scene flow.

[0067] Optionally, S3 specifically includes:

[0068] S31. Couple the edge optical flow and the global optical flow through the motion edge constraint module;

[0069] S32. Align the motion edge regions of the edge optical flow and the global optical flow by applying an optical flow consistency loss, and optimize the motion edge of the global optical flow; the motion consistency formula is:

[0070]

[0071] Among them, F frame Indicates the global optical flow corresponding to the frame camera, F event V represents the edge optical flow corresponding to the event camera, and V represents the valid motion edge region, which is obtained through forward and backward consistency detection of edge optical flow.

[0072] Optionally, S5 specifically includes:

[0073] S51. Obtain the projected coordinates of pixel p1 in the world coordinate system under image frame I1:

[0074] x1=D(p1)K -1 p1;

[0075] Where p1 represents the 2D homogeneous coordinates of a pixel in image frame I1, K represents the calibrated camera intrinsic parameters, and D(p1) is the depth information of pixel p1.

[0076] S52. The geometric motion calculation module based on spatial kinematics calculates the coordinates of the corresponding world coordinate system under the I2 image frame:

[0077] x2=PD(p1)K -1 p1

[0078] S53. Calculate the 3D rigid body flow based on the coordinates x2 and x1 of pixel p1 in the world coordinate system under image frame I2:

[0079] F rigid (x1) = x2 - x1.

[0080] Optical flow and depth have a strict geometric projection relationship. In embodiments of this invention, the depth D obtained by LiDAR is projected onto the corresponding 3D scene flow F in the world coordinate system using a projection module. scene The inertial navigation information obtained from the IMU sensor is used to enhance the motion details of rigid body regions containing radial motion in the 3D scene flow. To establish dense 3D pixel correspondences between adjacent frames, the above steps are used to obtain the 3D rigid body flow.

[0081] Optionally, S6 specifically includes:

[0082] S61. Couple the 3D scene flow and the 3D rigid body flow through the 3D motion field constraint module;

[0083] S62. Align the radial motion of the 3D scene flow and the 3D rigid body flow in the rigid body region M by applying a 3D motion consistency loss to obtain the target scene flow; the formula for calculating the radial motion is:

[0084]

[0085] Among them, F rigid For 3D rigid body flow, F scene For 3D scene flow.

[0086] This invention addresses the problem of scene motion field estimation in autonomous driving by estimating accurate 3D scene flow motion through geometric motion relationships based on synchronously acquired time-aligned multimodal information. Specifically, it couples edge optical flow and global optical flow through a motion edge constraint module, thereby optimizing the motion edge region of the global optical flow and solving the problem of inaccurate motion edge estimation caused by texture loss and motion blur. Furthermore, it couples 3D rigid body flow and 3D scene flow through a 3D motion field constraint module, optimizing the radial motion of the 3D scene flow obtained based on optical flow in the rigid body region. This solves the problem that existing methods do not include accurate radial motion in the estimated optical flow, leading to the estimated scene flow failing to reflect radial motion. This achieves accurate estimation of 3D scene flow motion and precisely simulates the complete scene motion in autonomous driving.

[0087] Example 2

[0088] like Figure 3 As shown, a scene flow estimation system based on multimodal information includes: a microcontroller and an industrial control computer, and modal sensors connected to both, including an event camera, a frame camera, a lidar, and an IMU sensor;

[0089] When the acquisition system is in motion, the microcontroller is used to control the alignment of timestamps of different modal sensors;

[0090] The event camera is used to record asynchronous event stream data of brightness changes per pixel;

[0091] The frame camera is used to acquire frame image data of dynamic scenes;

[0092] The lidar is used to acquire distance data to targets in the scene;

[0093] The IMU sensor is used to acquire inertial navigation information, which includes angular velocity data from the gyroscope and acceleration data from the accelerometer.

[0094] The industrial control computer is used to acquire data from the modal sensor, process the event stream data to obtain edge optical flow, process the dynamic scene frame image data to obtain global optical flow, and couple the edge optical flow and the global optical flow to obtain edge-optimized global optical flow; based on the distance data, the edge-optimized global optical flow is used to obtain 3D scene flow through a projection module, and based on the distance data and the inertial navigation information, 3D rigid body flow is obtained through geometric motion calculation, and the 3D scene flow and the 3D rigid body flow are coupled to obtain target scene flow.

[0095] A high-precision multimodal information synchronous acquisition system is built using an industrial control computer, a microcontroller, an event camera, a frame camera, a LiDAR, and an IMU sensor. For example, ... Figure 1As shown, a carbon fiber plate of 80 cm × 40 cm after structural processing is selected as the base plate for the equipment layout. A 32-line lidar is fixed at a processed structural frame with a height of 18 cm at one-quarter of the length of the base plate. A binocular event camera and a binocular frame camera are fixed at the front end with baselines of 50 cm and 40 cm respectively. An IMU sensor is fixed at the middle perpendicular to the long side. Finally, antennas connected to the built-in GPS module of the IMU sensor are placed at both ends of the base plate.

[0096] The industrial control computer is connected to four modal sensors to acquire data generated by each sensor. The microcontroller is used to control the timestamp alignment of different modal sensors. It controls the timestamp alignment of the four different modal sensors by generating a fixed-frequency synchronous pulse square wave. The LiDAR information acquisition frequency is 20Hz, the event camera acquisition frequency is 1MHz, the frame camera acquisition frequency is 50Hz, and the IMU sensor acquisition frequency is 500Hz. Therefore, the microcontroller uses 1MHz to ensure that the inherent acquisition clocks of all sensors can be aligned with the timestamps under the entire 1MHz external trigger frequency.

[0097] like Figure 2 As shown, the industrial control computer converts different modal information into corresponding information processing forms, including event voxels, frame images, depth, and inertial navigation information. Using frame images and event voxels, it obtains edge optical flow and global optical flow through an optical flow estimator. Then, it optimizes the edge motion region of the global optical flow through a motion edge constraint module. Based on the depth information, it obtains a 3D scene flow through a projection module. Finally, it converts the depth and inertial navigation information into a 3D rigid body flow through a geometric motion calculation module, and couples it with the 3D scene flow in a 3D motion field constraint module to obtain a scene flow with accurate radial motion and clear motion edges, which has the characteristics of simplicity and efficiency.

[0098] Optionally, the microcontroller can be connected to the external trigger interfaces of different modal sensors via PWM IO ports to control the synchronization of the acquisition timing of the modal sensors.

[0099] After the multimodal information acquisition system is built, a Type-C USB interface cable is used to connect the industrial control computer to the event camera and frame camera, an RS232 interface is used to connect the industrial control computer to the IMU sensor, and a 100Mbps Ethernet port is used to connect the industrial control computer to the 32-line LiDAR. Furthermore, to ensure that all sensors maintain synchronization in the acquisition time sequence, a microcontroller is connected to the external trigger interfaces of different modal sensors via a PWM I / O port. The microcontroller generates a preset frequency synchronization pulse square wave to control the timestamp alignment of different modal sensors, thereby ensuring that the data acquired by the industrial control computer is synchronized in time.

[0100] Optionally, the industrial computer is also used to align and calibrate the coordinate systems between any two different sensors.

[0101] During structural design, a preliminary estimate of coordinate system transformation between different sensors can be provided. To further refine the preliminary estimate, the registration between the event camera and the LiDAR is refined using the ICP algorithm, the registration between cameras is done using a checkerboard calibration method, and the registration between the event camera and the frame camera uses a checkerboard calibration method based on reconstructed images. The calibration of the IMU mainly involves temperature drift calibration of the gyroscope and accelerometer.

[0102] The technical solution of this invention acquires different modal information through multimodal sensors, collects nighttime scene motion information through an event camera to improve the autonomous driving scenario, and captures data of high-speed motion scenes through a frame camera. A microcontroller ensures synchronization of each modal information, and an industrial control computer processes the different modal information, estimating accurate 3D scene flow motion through geometric motion relationships. This solves the problems of motion degradation and edge blurring in all-day autonomous driving scene flow estimation, achieving accurate estimation of 3D scene flow motion.

[0103] In summary, this invention provides a scene flow estimation system based on multimodal information, enabling synchronous acquisition of multimodal information, and based on this, provides a scene flow estimation method based on multimodal information. This method can not only optimize the motion edges of scene motion in specific scenarios such as nighttime or high-speed motion, but also accurately model radial motion parallel to the camera normal in the scene, possessing the advantage of complete scene motion modeling.

[0104] Based on the above embodiments, the present invention also provides an autonomous vehicle, which includes the scene flow estimation system based on multimodal information as described in any of the above embodiments, for accurately estimating the motion scene during the autonomous driving process of the vehicle.

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scene flow estimation method based on multimodal information, characterized in that, include: Asynchronous event stream data of brightness change of each pixel in the target scene is acquired and rasterized into event voxels in a fixed time unit. The event voxels are then processed to obtain edge optical flow. Acquire several frames from the frame image data of the target scene, and obtain the global optical flow based on the corresponding frame image sequence; The edge optical flow and the global optical flow are coupled to obtain the global optical flow optimized for the edge region; The distance data to the scene target is obtained, and the 3D scene flow is obtained by combining the transformed depth information with the global optical flow projection optimized for edge regions. Acquire inertial navigation information, and calculate 3D rigid body flow based on the inertial navigation information and the depth information; The target scene flow is obtained by coupling the 3D scene flow and the 3D rigid body flow. The step of coupling the edge optical flow and the global optical flow to obtain the edge region optimized global optical flow includes: The edge optical flow and the global optical flow are coupled through a motion edge constraint module; The motion edge of the global optical flow is optimized by applying an optical flow consistency loss to align the motion edge regions of the edge optical flow and the global optical flow; the motion consistency formula is: in, This indicates the global optical flow corresponding to the frame camera. V represents the edge optical flow corresponding to the event camera, and V represents the valid motion edge region, which is obtained through forward and backward consistency detection of edge optical flow.

2. The scene flow estimation method as described in claim 1, characterized in that, The calculation of 3D rigid body flow based on the inertial navigation information and the depth information includes: Acquire image frames Down Projected coordinates of pixel coordinates in the world coordinate system: ； in, Represents an image frame In the context of a pixel, a 2D homogeneous coordinate is given, and K represents the calibrated camera intrinsic parameter. for Pixel depth information; Geometric motion calculation module based on spatial kinematics calculates... Image frame corresponding to world coordinates: according to Pixels in an image frame coordinates in the next world coordinate system and in image frames coordinates in the next world coordinate system The 3D rigid body flow is calculated as follows: 。 3. The scene flow estimation method as described in claim 1, characterized in that, The process of coupling the 3D scene flow and the 3D rigid body flow to obtain the scene flow includes: The 3D scene flow and the 3D rigid body flow are coupled through a 3D motion field constraint module; The target scene flow is obtained by aligning the radial motion of the 3D scene flow and the 3D rigid body flow in the rigid body region M by applying a 3D motion consistency loss; the formula for calculating the radial motion is: in, 3D rigid body flow For 3D scene flow.

4. The scene flow estimation method as described in claim 1, characterized in that, Before acquiring asynchronous event stream data of brightness changes per pixel in the target scene and rasterizing it into event voxels in a fixed time unit, and processing the event voxels to obtain edge optical flow, the method further includes: The timestamp alignment of different modal sensors is controlled by a preset frequency synchronous pulse square wave. Align and calibrate the coordinate systems between any two different sensors.

5. A scene flow estimation system based on multimodal information, characterized in that, include: The system includes a microcontroller and an industrial control computer, as well as modal sensors connected to both, including event cameras, frame cameras, lidar, and IMU sensors. When the acquisition system is in motion, the microcontroller is used to control the alignment of timestamps of different modal sensors; The event camera is used to record asynchronous event stream data of brightness changes per pixel; The frame camera is used to acquire frame image data of dynamic scenes; The lidar is used to acquire distance data to targets in the scene; The IMU sensor is used to acquire inertial navigation information, which includes angular velocity data from the gyroscope and acceleration data from the accelerometer. The industrial control computer is used to acquire data from the modal sensor, process the event stream data to obtain edge optical flow, process the dynamic scene frame image data to obtain global optical flow, and couple the edge optical flow and the global optical flow to obtain edge-optimized global optical flow; based on the distance data, the edge-optimized global optical flow is used to obtain 3D scene flow through a projection module, and based on the distance data and the inertial navigation information, 3D rigid body flow is obtained through geometric motion calculation, and the 3D scene flow and the 3D rigid body flow are coupled to obtain target scene flow; The step of coupling the edge optical flow and the global optical flow to obtain the edge-optimized global optical flow includes: The edge optical flow and the global optical flow are coupled through a motion edge constraint module; The motion edge of the global optical flow is optimized by applying an optical flow consistency loss to align the motion edge regions of the edge optical flow and the global optical flow; the motion consistency formula is: in, This indicates the global optical flow corresponding to the frame camera. V represents the edge optical flow corresponding to the event camera, and V represents the valid motion edge region, which is obtained through forward and backward consistency detection of edge optical flow.

6. The scene flow estimation system as described in claim 5, characterized in that, The microcontroller connects to the external trigger interfaces of different modal sensors via PWM IO ports to control the synchronization of the acquisition timing of the modal sensors.

7. The scene flow estimation system as described in claim 5, characterized in that, The industrial control computer is also used to align and calibrate the coordinate systems between any two different sensors.

8. An autonomous vehicle, characterized in that, Configure a scene flow estimation system based on multimodal information as described in any of claims 5-7.

Images