Multimodal data acquisition system and electronic device

Abstract

The application provides a multi-modal data acquisition system and electronic equipment, comprising a modular structure support provided from bottom to top with a bottom mounting layer, a middle mounting layer and a top mounting layer; a laser radar sensor in the sensor module is located on the top mounting layer, a millimeter wave radar sensor is located on the middle mounting layer, and a camera is located on the bottom mounting layer; a power supply module has a plurality of output ends, and the plurality of output ends are used for power supply to a plurality of millimeter wave radar sensors, a plurality of laser radar sensors and a plurality of cameras; a data synchronization module comprises an input end and a plurality of output ends, the input end of the data synchronization module receives a clock signal, and controls the plurality of millimeter wave radar sensors, the plurality of laser radar sensors and the plurality of cameras to start collecting millimeter wave radar data, laser radar data and image data at the same time. Thus, the quality and comprehensiveness of the collected data are improved, and the identification accuracy and reliability of the target are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle data acquisition technology, and in particular to a multimodal data acquisition system and electronic device. Background Technology

[0002] The development of autonomous driving algorithms relies heavily on large-scale and high-quality datasets.

[0003] However, existing technologies typically use a single millimeter-wave radar sensor to collect data, resulting in relatively limited data coverage, low data comprehensiveness, and low data quality. Consequently, the collected data is insufficient for accurate and reliable target identification. Summary of the Invention

[0004] This invention provides a multimodal data acquisition system and electronic device, which improves the quality and comprehensiveness of the acquired data, thereby improving the accuracy and reliability of target identification.

[0005] According to a first aspect of the present invention, a multimodal data acquisition system is provided, the system comprising: Modular structure bracket, installed on vehicle chassis, includes a bottom mounting layer, a middle mounting layer and a top mounting layer arranged from bottom to top; The sensor module includes several millimeter-wave radar sensors, several lidar sensors, and several cameras. Several lidar sensors are located on the top mounting layer, several millimeter-wave radar sensors are located on the middle mounting layer, and several cameras are located on the bottom mounting layer. The millimeter-wave radar sensors, lidar sensors, and cameras are used to collect millimeter-wave radar data, lidar data, and image data, respectively. The power supply module has several output terminals, and each millimeter-wave radar sensor, each lidar sensor, and each camera is connected to one of the output terminals. The several output terminals are used to supply power to the several millimeter-wave radar sensors, the several lidar sensors, and the several cameras. The data synchronization module includes an input terminal and several output terminals. The input terminal of the data synchronization module receives a clock signal. Each millimeter-wave radar sensor, each lidar sensor, and each camera is coupled to an output terminal. The data synchronization module is used to output a synchronization control signal based on the clock signal, and to control several millimeter-wave radar sensors, several lidar sensors, and several cameras to simultaneously start collecting millimeter-wave radar data, lidar data, and image data based on the synchronization control signal.

[0006] Optionally, the projection of the top mounting layer onto the bottom mounting layer is located within the middle mounting layer, and the projection of the middle mounting layer onto the bottom mounting layer is located within the bottom mounting layer.

[0007] Optionally, the top mounting layer, the middle mounting layer, and the bottom mounting layer are all rectangular in shape; The sensor module includes 3 lidar sensors, 8 millimeter-wave radar sensors, and 8 cameras; The lidar support bracket is set on the top mounting layer. The lidar support bracket has a rectangular structure with a hollow center. The three lidar sensors are fixed to the top mounting layer through the lidar support bracket. Each of the two lidar sensors is set at a corner of the lidar support bracket, and the other lidar sensor is set on the frame between the other two corners where no lidar sensor is set above. Two millimeter-wave radar receiving brackets are provided on the middle mounting layer. The two millimeter-wave radar receiving brackets are two U-shaped structures with openings facing each other. Each millimeter-wave radar receiving bracket has two corners. Each corner of the millimeter-wave radar receiving bracket corresponds one-to-one with each corner of the middle mounting layer. Two millimeter-wave radar sensors facing different directions are provided on each corner of the millimeter-wave radar receiving bracket. The bottom mounting layer has two rectangular support beams on opposite sides, each rectangular support beam has two ends, and each end is provided with two cameras facing different directions.

[0008] Optionally, the modular structure support includes: a plurality of universal joint structures, wherein the universal joint structures are located on the lidar support bracket, the millimeter-wave radar support bracket, or the rectangular support beam of the bottom mounting layer, and each millimeter-wave radar sensor, each lidar sensor, and each camera are located on a corresponding universal joint.

[0009] Optionally, the millimeter-wave radar sensor is a 4D millimeter-wave radar sensor.

[0010] Optionally, it also includes a data storage module, the input of which receives the millimeter-wave radar data, the lidar data, and the image data, and aligns the formats of the millimeter-wave radar data, the lidar data, and the image data respectively to obtain first millimeter-wave radar data, lidar data, and first image data with the same format, and stores the first millimeter-wave radar data, the lidar data, and the first image data.

[0011] Optionally, it also includes a data fusion processing module, which receives the first millimeter-wave radar data, the lidar data, and the first image data, and fuses the first millimeter-wave radar data, the lidar data, and the first image data to obtain a recognition result.

[0012] Optionally, the modular structure support includes: a detachable outer cover covering several millimeter-wave radar sensors, several triple lidar sensors, and several cameras.

[0013] Optionally, the power supply module employs a portable power source that provides DC inverter voltage to each millimeter-wave radar sensor, each lidar sensor, and each camera.

[0014] According to a second aspect of the present invention, an electronic device is provided, including the multimodal data acquisition system as described above.

[0015] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects: This invention provides a multimodal data acquisition system and electronic device. By employing several millimeter-wave radar sensors, several lidar sensors, and several cameras to collect data, it achieves multimodal data acquisition. Furthermore, due to its modular structure bracket, it is mounted on a vehicle chassis, including a bottom mounting layer, a middle mounting layer, and a top mounting layer arranged from bottom to top. Several cameras are located on the bottom mounting layer, with the millimeter-wave radar sensors, lidar sensors, and cameras respectively used to collect millimeter-wave radar data, lidar data, and image data. This results in a wider field of view for the data acquired by the multimodal data acquisition system. Building upon this foundation, the power supply module, with multiple output terminals, supplies power to each millimeter-wave radar sensor, lidar sensor, and camera. The data synchronization module, comprising one input terminal and multiple output terminals, receives a clock signal at its input. Each millimeter-wave radar sensor, lidar sensor, and camera is coupled to an output terminal. Based on the clock signal, the data synchronization module outputs a synchronization control signal, which then controls the simultaneous acquisition of millimeter-wave radar data, lidar data, and image data by the various sensors. Therefore, by synchronizing multimodal data, the quality and comprehensiveness of the acquired data are improved, thereby enhancing the accuracy and reliability of target identification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a multimodal data acquisition system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the modular structure support in an embodiment of the present invention; Figure 3 This is a schematic diagram showing the placement of the lidar sensor in an embodiment of the present invention; Figure 4 This is a schematic diagram showing the placement of the millimeter-wave radar sensor in an embodiment of the present invention; Figure 5 This is a schematic diagram showing the placement of the camera in an embodiment of the present invention; Figure 6 This is a schematic diagram of another multimodal data acquisition system in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of another multimodal data acquisition system in an embodiment of the present invention. Detailed Implementation

[0018] As described in the background section, data collected using a single millimeter-wave radar sensor is relatively limited in scope, lacking comprehensiveness and quality.

[0019] In view of this, embodiments of the present invention provide a multimodal data acquisition system, which, through a modular structural support, includes a bottom mounting layer, a middle mounting layer, and a top mounting layer arranged from bottom to top; a sensor module, including a plurality of millimeter-wave radar sensors, a plurality of lidar sensors, and a plurality of cameras, wherein the lidar sensors are located on the top mounting layer, the millimeter-wave radar sensors are located on the middle mounting layer, and the cameras are located on the bottom mounting layer, and the millimeter-wave radar sensors, lidar sensors, and cameras are respectively used to acquire millimeter-wave radar data, lidar data, and image data; and a power supply module, having a plurality of output terminals, for each millimeter-wave radar sensor and each lidar sensor... Each LiDAR sensor and each camera is connected to an output terminal. These output terminals supply power to multiple millimeter-wave radar sensors, LiDAR sensors, and cameras. A data synchronization module includes an input terminal and multiple output terminals. The input terminal receives a clock signal. Each millimeter-wave radar sensor, LiDAR sensor, and camera is coupled to an output terminal. Based on the clock signal, the data synchronization module outputs a synchronization control signal, which then controls the simultaneous acquisition of millimeter-wave radar data, LiDAR data, and image data by the multiple sensors. This improves the quality and comprehensiveness of the acquired data, thereby enhancing the accuracy and reliability of target identification.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0023] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Please refer to Figure 1 This invention provides a multimodal data acquisition system, installed on a vehicle chassis, the system comprising: The modular structure bracket 100 includes a bottom mounting layer, a middle mounting layer and a top mounting layer arranged from bottom to top.

[0025] The sensor module 200 includes a plurality of cameras, a plurality of millimeter-wave radar sensors, and a plurality of lidar sensors. The lidar sensors are located on the top mounting layer, the millimeter-wave radar sensors are located on the middle mounting layer, and the cameras are located on the bottom mounting layer. The cameras, the millimeter-wave radar sensors, and the lidar sensors are used to acquire image data, millimeter-wave radar data, and lidar data, respectively.

[0026] The power supply module 300 has several output terminals, one of which is connected to each millimeter-wave radar sensor, each lidar sensor, and each camera. The several output terminals are used to supply power to the several millimeter-wave radar sensors, the several lidar sensors, and the several cameras.

[0027] The data synchronization module 400 includes an input terminal and several output terminals. The input terminal of the data synchronization module 400 receives a clock signal (Clock). Each millimeter-wave radar sensor, each lidar sensor, and each camera is coupled to an output terminal. The data synchronization module is used to output a synchronization control signal based on the clock signal, and to control several millimeter-wave radar sensors, several lidar sensors, and several cameras to simultaneously start acquiring millimeter-wave radar data, lidar data, and image data based on the synchronization control signal.

[0028] As an example, the millimeter-wave radar sensor is a 4D millimeter-wave radar sensor.

[0029] Using a 4D millimeter-wave radar sensor can acquire 4D data, thereby improving the quality of the acquired data.

[0030] In this embodiment, the power supply module uses a portable power source, which provides DC inverter voltage to each millimeter-wave radar sensor, each lidar sensor, and each camera.

[0031] By using a separate portable power source to power the power supply module, while the vehicle's main unit and other power-required structures are powered by other power supply modules, ripple interference and harmonic pollution in electrical circuits at all levels are suppressed. This ensures both operational stability and long battery life. The replaceability of the portable power source allows for uninterrupted data acquisition, making it the most ideal power supply for this acquisition platform.

[0032] As can be seen, this embodiment employs several millimeter-wave radar sensors, several lidar sensors, and several cameras to collect data, and arranges different types of sensors on different layers of a modular structural support, thereby enabling the multimodal data acquisition system to acquire data with a wider field of view. Furthermore, by synchronizing the multimodal data, the quality and comprehensiveness of the acquired data are improved, thus enhancing the accuracy and reliability of target identification.

[0033] As an example, please refer to Figure 2 A schematic diagram of the modular structure support in an embodiment of the present invention.

[0034] The modular structure bracket 100 includes a bottom mounting layer 110, a middle mounting layer 120, and a top mounting layer 130 arranged from bottom to top.

[0035] As one implementation method, please continue to refer to Figure 2 The projection of the top mounting layer 130 onto the bottom mounting layer 110 is located within the middle mounting layer 120, and the projection of the middle mounting layer 120 onto the bottom mounting layer 110 is located within the bottom mounting layer 110.

[0036] For one specific implementation method, please refer to [link / reference]. Figure 2 The top mounting layer 130, the middle mounting layer 120, and the bottom mounting layer 110 are all rectangular in shape.

[0037] The sensor module 200 includes at least eight millimeter-wave radar sensors, three lidar sensors, and eight cameras. Two lidar sensors are located at two corners on the top mounting layer, and one lidar sensor is located on the frame of the adjacent corner on the top mounting layer where there is no lidar sensor. Two millimeter-wave radar sensors facing different directions are located at each corner of the middle mounting layer, and two cameras facing different directions are located on both sides of each corner of the bottom mounting layer.

[0038] As an example, the top mounting layer 130, the middle mounting layer 120, and the bottom mounting layer 110 are all rectangular in shape.

[0039] The sensor module 200 includes 3 lidar sensors, 8 millimeter-wave radar sensors, and 8 cameras.

[0040] Please refer to Figure 3 A schematic diagram of the placement of the lidar sensor in an embodiment of the present invention.

[0041] The lidar support bracket 330 is disposed on the top mounting layer 130. The lidar support bracket 330 has a rectangular structure with a hollow center. The three lidar sensors 230 are fixed to the top mounting layer 130 through the lidar support bracket 330. Each of the two lidar sensors 230 is disposed at a corner of the lidar support bracket 330, and the other lidar sensor is disposed on the frame between the other two corners where no lidar sensor is disposed.

[0042] Please refer to Figure 4 A schematic diagram of the placement of the millimeter-wave radar sensor in an embodiment of the present invention.

[0043] Two millimeter-wave radar receiving brackets 320 are provided on the middle mounting layer 120. The two millimeter-wave radar receiving brackets 320 are two U-shaped structures with openings facing each other. Each millimeter-wave radar receiving bracket 320 has two corners. Each corner of the millimeter-wave radar receiving bracket 320 corresponds one-to-one with each corner of the middle mounting layer 120. Two millimeter-wave radar sensors 220 facing different directions are provided on each corner of the millimeter-wave radar receiving bracket 320.

[0044] Please refer to Figure 5 A schematic diagram showing the placement of the camera in an embodiment of the present invention.

[0045] The bottom mounting layer 110 has two rectangular support beams on opposite sides, each rectangular support beam has two ends, and each end is provided with two cameras 210 facing different directions.

[0046] This setting avoids blind spots in the collected data, making the collected data more comprehensive and enabling more accurate and reliable target identification.

[0047] As one specific implementation, the modular structure support may include: a plurality of universal joint structures, wherein the universal joint structures are located on the lidar support bracket, the millimeter-wave radar support bracket, or the rectangular support beam of the bottom mounting layer, and each millimeter-wave radar sensor, each lidar sensor, and each camera are located on a corresponding universal joint.

[0048] In this embodiment, a gimbal is used to adjust the orientation of each sensor. This allows the sensors to rotate flexibly, thereby enabling the collected data to cover a wider field of view.

[0049] For one specific implementation method, please refer to Figure 6 The multimodal data acquisition system may further include: a data storage module 500, wherein the input terminal of the data storage module 500 receives the millimeter-wave radar data, the lidar data, and the image data, and aligns the formats of the millimeter-wave radar data, the lidar data, and the image data respectively to obtain first millimeter-wave radar data, lidar data, and first image data with the same format, and stores the first millimeter-wave radar data, the lidar data, and the first image data.

[0050] For one specific implementation method, please refer to Figure 7 The multimodal data acquisition system may further include: a data fusion processing module 600, which receives the first millimeter-wave radar data, the lidar data, and the first image data, and performs fusion processing on the first millimeter-wave radar data, the lidar data, and the first image data.

[0051] The first millimeter-wave radar data, the lidar data, and the first image data are fused together, and the fused data can be used for subsequent target identification.

[0052] In summary, this embodiment achieves multimodal data acquisition by employing multiple millimeter-wave radar sensors, multiple lidar sensors, and multiple cameras to collect data. Furthermore, the modular structure bracket, mounted on the vehicle chassis, includes a bottom mounting layer, a middle mounting layer, and a top mounting layer arranged from bottom to top. Multiple cameras are located on the bottom mounting layer, with the millimeter-wave radar sensors, lidar sensors, and cameras respectively used to collect millimeter-wave radar data, lidar data, and image data. This results in a wider field of view for the data acquired by the multimodal data acquisition system. Building upon this foundation, the power supply module, with multiple output terminals, supplies power to each millimeter-wave radar sensor, lidar sensor, and camera. The data synchronization module, comprising one input terminal and multiple output terminals, receives a clock signal at its input. Each millimeter-wave radar sensor, lidar sensor, and camera is coupled to an output terminal. Based on the clock signal, the data synchronization module outputs a synchronization control signal, which then controls the simultaneous acquisition of millimeter-wave radar data, lidar data, and image data by the various sensors. Therefore, by synchronizing multimodal data, the quality and comprehensiveness of the acquired data are improved, thereby enhancing the accuracy and reliability of target identification.

[0053] In addition, this embodiment also provides an electronic device, including the multimodal data acquisition system as described above.

[0054] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A multimodal data acquisition system, characterized in that, The system includes: Modular structure bracket, installed on vehicle chassis, includes a bottom mounting layer, a middle mounting layer and a top mounting layer arranged from bottom to top; The sensor module includes several millimeter-wave radar sensors, several lidar sensors, and several cameras. Several lidar sensors are located on the top mounting layer, several millimeter-wave radar sensors are located on the middle mounting layer, and several cameras are located on the bottom mounting layer. The millimeter-wave radar sensors, lidar sensors, and cameras are used to collect millimeter-wave radar data, lidar data, and image data, respectively. The power supply module has several output terminals, and each millimeter-wave radar sensor, each lidar sensor, and each camera is connected to one of the output terminals. The several output terminals are used to supply power to the several millimeter-wave radar sensors, the several lidar sensors, and the several cameras. The data synchronization module includes an input terminal and several output terminals. The input terminal of the data synchronization module receives a clock signal. Each millimeter-wave radar sensor, each lidar sensor, and each camera is coupled to an output terminal. The data synchronization module is used to output a synchronization control signal based on the clock signal, and to control several millimeter-wave radar sensors, several lidar sensors, and several cameras to simultaneously start collecting millimeter-wave radar data, lidar data, and image data based on the synchronization control signal.

2. The multimodal data acquisition system as described in claim 1, characterized in that, The projection of the top mounting layer onto the bottom mounting layer is located within the middle mounting layer, and the projection of the middle mounting layer onto the bottom mounting layer is located within the bottom mounting layer.

3. The multimodal data acquisition system as described in claim 1, characterized in that, The top mounting layer, the middle mounting layer, and the bottom mounting layer are all rectangular in shape; The sensor module includes 3 lidar sensors, 8 millimeter-wave radar sensors, and 8 cameras; The lidar support bracket is set on the top mounting layer. The lidar support bracket has a rectangular structure with a hollow center. The three lidar sensors are fixed to the top mounting layer through the lidar support bracket. Each of the two lidar sensors is set at a corner of the lidar support bracket, and the other lidar sensor is set on the frame between the other two corners where no lidar sensor is set above. Two millimeter-wave radar receiving brackets are provided on the middle mounting layer. The two millimeter-wave radar receiving brackets are two U-shaped structures with openings facing each other. Each millimeter-wave radar receiving bracket has two corners. Each corner of the millimeter-wave radar receiving bracket corresponds one-to-one with each corner of the middle mounting layer. Two millimeter-wave radar sensors facing different directions are provided on each corner of the millimeter-wave radar receiving bracket. The bottom mounting layer has two rectangular support beams on opposite sides, each rectangular support beam has two ends, and each end is provided with two cameras facing different directions.

4. The multimodal data acquisition system as described in claim 3, characterized in that, The modular structure support includes several universal joint structures, which are located on the lidar support bracket, the millimeter-wave radar support bracket, or the rectangular support beam of the bottom mounting layer. Each millimeter-wave radar sensor, each lidar sensor, and each camera are located on a corresponding universal joint.

5. The multimodal data acquisition system as described in claim 3, characterized in that, The millimeter-wave radar sensor is a 4D millimeter-wave radar sensor.

6. The multimodal data acquisition system as described in claim 1, characterized in that, It also includes a data storage module, whose input terminal receives the millimeter-wave radar data, the lidar data, and the image data, and aligns the formats of the millimeter-wave radar data, the lidar data, and the image data respectively to obtain first millimeter-wave radar data, lidar data, and first image data with the same format, and stores the first millimeter-wave radar data, the lidar data, and the first image data.

7. The multimodal data acquisition system as described in claim 6, characterized in that, It also includes a data fusion processing module, which receives the first millimeter-wave radar data, the lidar data, and the first image data, and fuses the first millimeter-wave radar data, the lidar data, and the first image data.

8. The multimodal data acquisition system as described in claim 1, characterized in that, The modular structure support includes: a detachable outer cover covering several millimeter-wave radar sensors, several triple lidar sensors, and several cameras.

9. The multimodal data acquisition system as described in claim 1, characterized in that, The power supply module uses a portable power source, which provides DC inverter voltage to each millimeter-wave radar sensor, each lidar sensor, and each camera.

10. An electronic device, characterized in that, The system includes the multimodal data acquisition system as described in any one of claims 1 to 9.

Images