Data fusion method and device and intelligent driving equipment

Abstract

The embodiment of the invention provides a data fusion method and device and intelligent driving equipment, and the method comprises the steps: obtaining point cloud data, the point cloud data is obtained through the detection of a first laser beam group and a second laser beam group in a first scanning direction and a second scanning direction, and the first scanning direction is different from the second scanning direction; sequentially triggering camera exposure in the M camera groups according to the triggering time delays of the M camera groups to obtain M groups of picture data; and fusing the picture data and the point cloud data corresponding to each camera group in the M camera groups to obtain data used for sensing processing and corresponding to the camera groups. According to the embodiment of the invention, the fusion effect of the point cloud data obtained by scanning of the laser emission device and the picture data captured by the cameras can be improved; therefore, the vehicle surrounding environment identification is more accurate, and the planning and implementation of an automatic driving task are more facilitated.

Description

Data fusion method, device and intelligent driving equipment Technical Field

[0001] The present application relates to intelligent driving technology, which is applied to the field of fusion perception of intelligent driving, and in particular to a data fusion method, device and intelligent driving equipment. Background Art

[0002] With the development of society, smart cars are gradually becoming part of people's daily lives. Advanced driver assistance systems (ADAS) play a crucial role in smart cars. They utilize a variety of sensors installed on the vehicle to sense the surrounding environment while the vehicle is in motion, collect data, and identify, detect, and track both stationary and moving objects. Combined with navigation map data, they perform systematic calculations and analysis, allowing drivers to proactively detect potential dangers and effectively improving driving comfort and safety. In short, sensors play a crucial role in assisted and autonomous driving in smart cars, often likened to the car's "eyes." Sensors include vision sensors like onboard cameras and radar sensors like lidar and ultrasonic radar.

[0003] Compared with single sensor perception, multi-sensor fusion perception is increasingly becoming the mainstream of intelligent driving. The multiple sources of data make the final perception results more stable and reliable, and can better utilize the advantages of each sensor and avoid defects.

[0004] In current multi-sensor fusion solutions, lidar has a fixed scanning cycle, while cameras are triggered at a fixed frequency. In scenarios where lidar and camera are fused, the fusion effect is often poor due to timestamp mismatches between the lidar and camera, making it impossible to accurately identify the vehicle's surroundings.

[0005] Summary of the Invention

[0006] The present application discloses a data fusion method, device and intelligent driving equipment, which can improve the fusion effect of point cloud data obtained by scanning a laser emitting device and image data captured by a camera, thereby making the identification of the vehicle's surrounding environment more accurate and more conducive to the planning and implementation of autonomous driving tasks.

[0007] In a first aspect, the present application provides a data fusion method, which includes: acquiring point cloud data, wherein the point cloud data is obtained by detecting a first laser beam group and a second laser beam group in a first scanning direction and a second scanning direction respectively, wherein the first scanning direction and the second scanning direction are different; triggering the cameras in the M camera groups to expose in sequence according to the trigger delays of the M camera groups to obtain M groups of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1; fusing the image data corresponding to each camera group in the M camera groups with the point cloud data to obtain data for perception processing corresponding to the camera group.

[0008] The M camera groups can be determined based on the installation positions of the multiple cameras on the intelligent driving device. Based on the driving direction of the intelligent driving device, this application can fuse the point cloud data with the image data of the camera group corresponding to the front field of view on the intelligent driving device to improve the fusion effect of the point cloud data and image data of the front field of view. Similarly, the fusion effect of the point cloud data and image data of the middle field of view and the rear field of view can also be improved, thereby making the recognition of the vehicle environment more accurate and more conducive to the planning and implementation of autonomous driving tasks.

[0009] In one possible implementation, the acquiring of point cloud data includes: acquiring first data, where the first data is obtained by the first laser beam group detecting in the first scanning direction; acquiring second data, where the second data is obtained by the second laser beam group detecting in the second scanning direction; and splicing the first data and the second data to acquire the point cloud data.

[0010] In the above method, the first data and the second data are spliced, fused and enhanced. Compared with the point cloud data obtained by scanning in a single direction, the point cloud data in this application has a higher point density and a higher accuracy in recognizing the vehicle surrounding environment.

[0011] In yet another possible implementation, the first laser beam group and the second laser beam group come from the same laser emitting device.

[0012] In yet another possible implementation, the first laser beam group comes from a first laser emitting device, and the second laser beam group comes from a second laser emitting device.

[0013] In the above method, two laser beam groups are used to enable coverage scanning, thereby making the point density of the point cloud data higher.

[0014] In yet another possible implementation, the method further includes: determining a position of each camera group in the M camera groups; and determining a trigger delay of each camera group according to the scanning period and the position of each camera group in the M camera groups.

[0015] In another possible implementation, the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is a first delay, and the trigger delay of the second camera group is a second delay; triggering the cameras in the M camera groups to be exposed in sequence according to the trigger delays of the M camera groups to obtain M groups of image data includes: starting a first timer to set an initial moment; when the indicated moment of the first timer is separated from the initial moment by the first delay, triggering the cameras in the first camera group to be exposed to obtain the image data corresponding to the first camera group; when the indicated moment of the first timer is separated from the initial moment by the second delay, triggering the cameras in the second camera group to be exposed to obtain the image data corresponding to the second camera group.

[0016] In the above method, by sequentially triggering the exposure of cameras in M ​​camera groups according to M trigger delays to obtain M groups of image data, compared with the method of simultaneously exposing all cameras in the intelligent driving device, which will cause concentrated resource occupation and peak load competition leading to performance problems, the sequential triggering exposure of the present application can balance the load to a certain extent, thereby avoiding the problems caused by simultaneous exposure.

[0017] In another possible implementation, the M camera groups include a front group, a side group, and a rear group; the front group includes a first camera and a second camera, and the first camera and the second camera are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.

[0018] In yet another possible implementation, the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

[0019] In yet another possible implementation, image data corresponding to at least two camera groups among the M camera groups are fused to obtain data for perception processing.

[0020] In a second aspect, the present application provides an intelligent driving device, comprising: a computing platform and M camera groups, the computing platform being used to: obtain point cloud data, the point cloud data being obtained by detecting in a first scanning direction and a second scanning direction according to a first laser beam group and a second laser beam group, respectively, wherein the first scanning direction and the second scanning direction are different; triggering the cameras in the M camera groups to be exposed in sequence according to the trigger delays of the M camera groups to obtain M groups of image data; wherein, the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1; fusing the image data corresponding to each camera group in the M camera groups and the point cloud data to obtain the data for perception processing corresponding to the camera group.

[0021] In one possible implementation, when acquiring the point cloud data, the computing platform is used to: acquire first data, where the first data is obtained by detecting the first laser beam group in the first scanning direction; acquire second data, where the second data is obtained by detecting the second laser beam group in the second scanning direction; and splice the first data and the second data to acquire the point cloud data.

[0022] In yet another possible implementation, the first laser beam group and the second laser beam group come from the same laser emitting device.

[0023] In another possible implementation, the intelligent driving device further includes: a first laser emitting device and a second laser emitting device; the first laser beam group comes from the first laser emitting device, and the second laser beam group comes from the second laser emitting device.

[0024] In another possible implementation, the first laser emitting device and the second laser emitting device are laser radar devices.

[0025] In another possible implementation, the computing platform is further used to: determine the position of each camera group in the M camera groups; and determine the trigger delay of each camera group based on the scanning period and the position of each camera group in the M camera groups.

[0026] In another possible implementation, the M camera groups include a first camera group and a second camera group, the triggering delay of the first camera group is a first delay, and the triggering delay of the second camera group is a second delay, and the computing platform is further configured to: start a first timer to set an initial time; and when the first delay elapses between the time indicated by the first timer and the initial time, trigger the cameras in the first camera group to expose the image data corresponding to the first camera group; and when the second delay elapses between the time indicated by the first timer and the initial time, trigger the cameras in the second camera group to expose the image data corresponding to the second camera group.

[0027] In another possible implementation, the M camera groups include a front group, a side group, and a rear group; the front group includes a first camera and a second camera, and the first camera and the second camera are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.

[0028] In yet another possible implementation, the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

[0029] Regarding the technical effects brought about by the second aspect or possible implementation methods, reference may be made to the introduction to the technical effects of the first aspect or corresponding implementation methods.

[0030] In the third aspect, the present application provides a data fusion device, which is a vehicle-mounted computing platform or a cloud server or a chip system. The data fusion device includes a processor and a memory, and the memory and the processor are interconnected through a line. The processor is used to obtain a computer program stored in the memory. When the computer program is called by the processor, it is used to execute the data fusion method in the above-mentioned first aspect and each implementation method.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a processor, the method described in the first aspect or any possible implementation method of the first aspect is implemented.

[0032] In a fifth aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, it implements the method described in the first aspect or any possible implementation method of the first aspect.

[0033] In a sixth aspect, the present application provides a chip, wherein the chip includes a circuit, and the circuit is used to implement the method described in the first aspect or any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a schematic structural diagram of an intelligent driving device provided in an embodiment of the present application;

[0035] FIG2 is a schematic diagram of camera grouping provided by an embodiment of the present application;

[0036] FIG3 is a schematic diagram of another camera grouping provided by an embodiment of the present application;

[0037] FIG4 is a schematic structural diagram of another intelligent driving device provided in an embodiment of the present application;

[0038] FIG5 is a schematic diagram of a driving control device of an intelligent driving device provided in an embodiment of the present application;

[0039] FIG6 is a schematic diagram of the position distribution of cameras in an intelligent driving device provided in an embodiment of the present application;

[0040] FIG7 is a flow chart of a data fusion method provided in an embodiment of the present application;

[0041] FIG8a is a top view of a first laser beam group and a second laser beam group provided in an embodiment of the present application;

[0042] FIG8 b is a side view of a first laser beam group and a second laser beam group provided in an embodiment of the present application;

[0043] FIG9a is a top view of another first laser beam group and a second laser beam group provided in an embodiment of the present application;

[0044] FIG9 b is a side view of another first laser beam group and a second laser beam group provided in an embodiment of the present application;

[0045] FIG10 is a schematic diagram of a surrounding environment provided in an embodiment of the present application;

[0046] FIG11 is a schematic diagram of point cloud data provided by an embodiment of the present application;

[0047] FIG12 is a schematic diagram of image data corresponding to camera groups provided in an embodiment of the present application;

[0048] FIG13 is a schematic diagram of image data corresponding to another camera grouping provided in an embodiment of the present application;

[0049] FIG14 is a schematic diagram of a front, middle, and rear field of view provided by an embodiment of the present application;

[0050] FIG15 is a schematic structural diagram of a data fusion device provided in an embodiment of the present application;

[0051] FIG16 is a schematic diagram of an intelligent driving device provided in an embodiment of the present application;

[0052] FIG17 is a schematic structural diagram of another data fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0054] In this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0055] The “at least one” mentioned in the embodiments of this application refers to one or more, and “plurality” refers to two or more. “At least one of the following items” or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, “a and b”, “a and c”, “b and c”, or “a and b and c”, where a, b, c can be single or multiple. “And / or” describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character “ / ” generally indicates that the previous and next associated objects are in an “or” relationship.

[0056] The terms "first," "second," "third," and "fourth" mentioned in the embodiments of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to such process, method, product, or device.

[0057] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] Please refer to Figure 1, which is a structural schematic diagram of an intelligent driving device 100 provided in an embodiment of the present application. The intelligent driving device 100 includes a computing platform 101 and M camera groups 102, where M is a positive integer greater than or equal to 1. M can also be a positive integer greater than or equal to 2. The computing platform 101 can be installed in the intelligent driving device 100 or not. For example, the computing platform 101 can be installed in a cloud server, which is not limited in the embodiment of the present application. The computing platform 101 can be understood as a device with computing capabilities. The computing platform 101 is used to obtain point cloud data and M groups of image data, and fuse the image data and point cloud data corresponding to each camera group in the M groups of image data to obtain data corresponding to the camera group for perception processing. For specific related descriptions, see the following embodiments. Each of the M camera groups includes at least one camera. In an example, please refer to Figure 2, which is a schematic diagram of a camera grouping provided in an embodiment of the present application, M=3, the three camera groups are group 1, group 2 and group 3, and the intelligent driving device 100 includes 6 cameras, namely camera 1, camera 2, camera 3, camera 4, camera 5 and camera 6; each camera group includes 2 cameras; among them, camera 1 and camera 2 belong to group 1, and camera 1 and camera 2 are relatively arranged at the front of the intelligent driving device 100; camera 3 and camera 4 belong to group 2, and camera 3 and camera 4 are relatively arranged on both sides of the intelligent driving device 100; camera 5 and camera 6 belong to group 3, and camera 5 and camera 6 are relatively arranged at the rear of the intelligent driving device 100. In another example, see Figure 3, which is a schematic diagram of another camera grouping provided in an embodiment of the present application, where M = 4, and the four camera groups are Group 1, Group 2, Group 3, and Group 4. The intelligent driving device 100 includes six cameras, namely Camera 1, Camera 2, Camera 3, Camera 4, Camera 5, Camera 6, and Camera 7. Each camera group includes at least one camera. Cameras 1 and 2 belong to Group 1, and are positioned relative to each other at the front of the intelligent driving device 100; Cameras 3 and 4 belong to Group 2, and are positioned relative to each other on either side of the intelligent driving device 100; Cameras 5 and 6 belong to Group 3, and are positioned relative to each other at the rear of the intelligent driving device 100; and Camera 7 belongs to Group 4, and is oriented in the direction of travel of the intelligent driving device 100. Figures 2 and 3 above illustrate the positions of the cameras in the intelligent driving device when the number of cameras in the intelligent driving device is an even or odd number. The camera positions may also vary depending on the type of intelligent driving device, etc., and are not limited in this embodiment of the present application.

[0059] The intelligent driving device 100 may further include one or more laser emitting devices 103. When the intelligent driving device 100 includes one laser emitting device 103, the laser emitting device 103 is used to generate a first laser beam group and a second laser beam group, which can be understood as the first laser beam group and the second laser beam group coming from the same laser emitting device. When the intelligent driving device 100 includes two laser emitting devices 103, for example, a first laser emitting device and a second laser emitting device, the first laser emitting device is used to generate the first laser beam group, and the second laser emitting device is used to generate the second laser beam group, which can be understood as the first laser beam group and the second laser beam group coming from different laser emitting devices. The first laser beam group detects in a first scanning direction to obtain first data, and the second laser beam group detects in a second scanning direction to obtain second data. The first data and the second data are spliced ​​to obtain point cloud data. Detailed descriptions are provided in the following embodiments.

[0060] It is understandable that the structure of the intelligent driving device in FIG1 is only an exemplary implementation in the embodiment of the present application, and the intelligent driving device in the embodiment of the present application may further include more components as needed.

[0061] Please refer to Figure 4, which further illustrates the structure of the intelligent driving device 100 based on Figure 1. The intelligent driving device 100 may include various subsystems, such as a travel system 202, a sensor system 204, a control system 206, one or more peripheral devices 208, a power supply 210, a computer system 212, and a user interface 216. Alternatively, the intelligent driving device 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the intelligent driving device 100 may be interconnected via wired or wireless connections.

[0062] The propulsion system 202 may include components that provide power and movement for the intelligent driving device 100. In one embodiment, the propulsion system 202 may include an engine 218, an energy source 219, a transmission 220, and wheels / tires 221. The engine 218 may be an internal combustion engine, an electric motor, an air compression engine, or other engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 218 converts the energy source 219 into mechanical energy.

[0063] Examples of energy source 219 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 219 can also provide energy for other systems of the intelligent driving device 100.

[0064] Transmission 220 can transmit mechanical power from engine 218 to wheels 221. Transmission 220 can include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 220 can also include other components, such as a clutch. The drive shaft can include one or more shafts that can be coupled to one or more wheels 221.

[0065] The sensor system 204 may include several sensors that sense information about the environment surrounding the intelligent driving device 100. For example, the sensor system 204 may include a positioning system 222 (the positioning system may be a GPS system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 224, a radar 226, a laser rangefinder 228, and a camera 230. The sensor system 204 may also include sensors of the internal systems of the monitored intelligent driving device 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of the autonomous intelligent driving device 100.

[0066] The positioning system 222 may be used to estimate the geographic location of the intelligent driving device 100. The IMU 224 is used to sense the position and orientation changes of the intelligent driving device 100 based on inertial acceleration. In one embodiment, the IMU 224 may be a combination of an accelerometer and a gyroscope.

[0067] Radar 226 can use radio signals to sense objects within the surrounding environment of the intelligent driving device 100. In some embodiments, in addition to sensing objects, radar 226 can also be used to sense the speed and / or direction of travel of objects. In one example, radar 226 can be the laser emitting device 103 in Figure 1. For example, the intelligent driving device 100 includes a lidar device, which includes two laser beam groups: a first laser beam group and a second laser beam group. The first laser beam group includes at least one laser beam, and the second laser beam group includes at least one laser beam. The first laser beam group detects in a first scanning direction to obtain first data, and the second laser beam group detects in a second scanning direction to obtain second data. The first scanning direction and the second scanning direction are different, which can be understood as different rotation directions. Optionally, the first scanning direction can be clockwise and the second scanning direction can be counterclockwise. The first data and the second data can be understood as point cloud information. For another example, the intelligent driving device 100 includes two laser radar devices, namely laser radar device 1 and laser radar device 2. Laser radar device 1 includes a group of laser beam groups, which is the first laser beam group; laser radar device 2 includes a group of laser beam groups, which is the second laser beam group; the first laser beam group detects in the first scanning direction to obtain first data; the second laser beam group detects in the second scanning direction to obtain second data; the first scanning direction and the second scanning direction are different.

[0068] The laser rangefinder 228 may utilize laser light to sense objects in the environment in which the intelligent driving device 100 is located. In some embodiments, the laser rangefinder 228 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0069] The camera 230 can be used to capture multiple images of the surrounding environment of the intelligent driving device 100. The camera 230 can be a still camera or a video camera. The number of cameras 230 can be one or more. The cameras 230 can be divided into the M camera groups 102 shown in Figure 1. For example, the distribution of the cameras 230 in the intelligent driving device 100 can be shown in Figure 2 or Figure 3.

[0070] The control system 206 controls the operation of the intelligent driving device 100 and its components. The control system 206 may include various components, including a steering system 232 , a throttle 234 , a brake unit 236 , a sensor fusion algorithm 238 , a computer vision system 240 , a path control system 242 , and an obstacle avoidance system 244 .

[0071] The steering system 232 is operable to adjust the forward direction of the intelligent driving device 100. For example, in one embodiment, it can be a steering wheel system.

[0072] The throttle 234 is used to control the operating speed of the engine 218 and thus control the speed of the intelligent driving device 100 .

[0073] Braking unit 236 is used to control the deceleration of intelligent driving device 100. Braking unit 236 can use friction to slow down wheel 221. In other embodiments, braking unit 236 can convert the kinetic energy of wheel 221 into electric current. Braking unit 236 can also take other forms to slow the rotation speed of wheel 221 and thus control the speed of intelligent driving device 100.

[0074] The computer vision system 240 can be operated to process and analyze images captured by the camera 230 in order to identify objects and / or features in the surrounding environment of the intelligent driving device 100, and / or process and analyze data information captured by the radar 226. The objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 240 can use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision technologies. In some embodiments, the computer vision system 240 can be used to map the environment, track objects, estimate the speed of objects, and the like. Optionally, the computer vision system 240 can be the computing platform 101 in Figure 1.

[0075] The route control system 242 is used to determine the driving route of the intelligent driving device 100. In some embodiments, the route control system 242 can combine data from the sensor fusion algorithm 238, the positioning system 222, and one or more predetermined maps to determine the driving route for the intelligent driving device 100.

[0076] The obstacle avoidance system 244 is used to identify, evaluate, and avoid or otherwise negotiate potential obstacles in the environment of the intelligent driving device 100 .

[0077] Of course, in one example, the control system 206 may include additional or alternative components other than those shown and described, or may also include fewer than some of the components shown.

[0078] The intelligent driving device 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripheral devices 208. The peripheral devices 208 may include a wireless communication system 246, an onboard computer 248, a microphone 250, and / or a speaker 252.

[0079] In some embodiments, the peripheral device 208 provides a means for the user of the intelligent driving device 100 to interact with the user interface 216. For example, the onboard computer 248 can provide information to the user of the intelligent driving device 100. The user interface 216 can also operate the onboard computer 248 to receive user input. The onboard computer 248 can be operated via a touch screen. In other cases, the peripheral device 208 can provide a means for the intelligent driving device 100 to communicate with other devices located in the vehicle. For example, the microphone 250 can receive audio (e.g., voice commands or other audio input) from the user of the intelligent driving device 100. Similarly, the speaker 252 can output audio to the user of the intelligent driving device 100.

[0080] The wireless communication system 246 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 246 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 246 can use Wi-Fi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 246 can use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. The wireless communication system 246 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0081] The power supply 210 can provide power to the various components of the intelligent driving device 100. In one embodiment, the power supply 210 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to the various components of the intelligent driving device 100. In some embodiments, the power supply 210 and the energy source 219 can be implemented together.

[0082] Some or all functions of the intelligent driving device 100 are controlled by a computer system 212. The computer system 212 may include at least one processor 213 that executes instructions 215 stored in a non-transitory computer-readable medium, such as a memory 214. The computer system 212 may also be a plurality of computing devices that control individual components or subsystems of the intelligent driving device 100 in a distributed manner.

[0083] The processor 213 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be an ASIC or other hardware-based processor.

[0084] In various aspects described herein, the processor can be remote from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor.

[0085] In some embodiments, the memory 214 may include instructions 215 (e.g., program logic) that may be executed by the processor 213 to perform various functions of the intelligent driving device 100, including those described above. The memory 214 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or sending control instructions to one or more of the travel system 202, the sensor system 204, the control system 206, and the peripheral devices 208.

[0086] In addition to the instructions 215, the memory 214 may also store data, such as point cloud data and M sets of image data corresponding to the M camera groups, as well as other information. This information may be used by the intelligent driving device 100 and the computer system 212 during operation of the intelligent driving device 100 in autonomous, semi-autonomous, and / or manual modes.

[0087] User interface 216 is used to provide information to or receive information from a user of intelligent driving device 100. Optionally, user interface 216 may include one or more input / output devices within the set of peripheral devices 208, such as wireless communication system 246, onboard computer 248, microphone 250, and speaker 252.

[0088] The computer system 212 can control the functions of the intelligent driving device 100 based on input received from various subsystems (e.g., the travel system 202, the sensor system 204, and the control system 206) and from the user interface 216. For example, the computer system 212 can use input from the control system 206 to control the steering system 232 to avoid obstacles detected by the sensor system 204 and the obstacle avoidance system 244. In some embodiments, the computer system 212 is operable to provide control over many aspects of the intelligent driving device 100 and its subsystems.

[0089] Optionally, one or more of the above components may be installed or associated separately from the intelligent driving device 100. For example, the memory 214 may be partially or completely separate from the intelligent driving device 100. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0090] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 4 should not be understood as a limitation to the embodiments of the present application.

[0091] The intelligent driving device 100 may be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, cart, smart home device, etc., and the embodiments of the present application do not impose any particular limitation.

[0092] It can be understood that the structure of the intelligent driving device in Figure 4 is only an exemplary implementation in the embodiment of the present application. The intelligent driving device in the embodiment of the present application includes but is not limited to the above structure.

[0093] Please refer to Figure 5, which is a schematic diagram of a driving control device for an intelligent driving device provided in an embodiment of the present application. This device, as used in Figure 4 above, is equivalent to the computer system 212 shown in Figure 4 and may include a processor 213 coupled to a system bus 505. Processor 213 may be one or more processors, each of which may include one or more processor cores. Memory 214 may store relevant data information and is coupled to system bus 505. Display adapter 507 may drive display 509, which is coupled to system bus 505. System bus 505 is coupled to input / output (I / O) bus 513 via bus bridge 501. An I / O interface 515 is coupled to I / O bus 513. The I / O interface 515 communicates with various I / O devices, such as input device 517 (e.g., keyboard, mouse, touch screen, etc.) and media tray 521 (e.g., CD-ROM, multimedia interface, etc.). The transceiver 523 (can send and / or receive radio communication signals), the camera 555 (can capture scene and dynamic digital video images) and the external USB interface 525. Optionally, the interface connected to the I / O interface 515 can be a USB interface.

[0094] Processor 213 may be any conventional processor, including a reduced instruction set computer (RISC), a complex instruction set computer (CISC), or a combination thereof. Alternatively, the processor may be an application-specific integrated circuit (ASIC). Alternatively, processor 213 may be a neural network processor, or a combination of a neural network processor and the conventional processors described above.

[0095] Alternatively, in various embodiments described herein, the computer system 212 may be located remotely from the intelligent driving device and may communicate wirelessly with the intelligent driving device. In other aspects, some of the processes described herein are executed on a processor within the intelligent driving device, while others are executed by a remote processor.

[0096] Computer system 212 can communicate with software deployment server 549 via network interface 529. Network interface 529 is a hardware network interface, such as a network card. Network 527 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Alternatively, network 527 can be a wireless network, such as a Wi-Fi network or a cellular network.

[0097] The transceiver 523 (which can send and / or receive radio communication signals) can use various wireless communication methods including but not limited to the second generation mobile communication networks (2G), 3G, 4G, 5G, etc., or DSRC technology, or long-term evolution vehicle-to-everything technology (LTE-V2X), etc. Its main function is to receive information data sent by external devices and send the information data generated by the intelligent driving device when driving on the target road section back to the external device for storage and analysis.

[0098] The hard disk drive interface 531 is coupled to the system bus 505. The hard disk drive interface 531 is connected to the hard disk drive 533. The system memory 535 is coupled to the system bus 505. The data running in the system memory 535 may include the operating system 537 and application programs 543 of the computer system 212.

[0099] The system memory 535 is coupled to the system bus 505. For example, in the present application, the system memory 535 can be used to store the driving information of the vehicle passing through the target road section in a certain format.

[0100] Operating system (OS) 537 includes a shell 539 and a kernel 541. Shell 539 is an interface between the user and the operating system's kernel. Shell 539 is the outermost layer of the operating system. It manages the interaction between the user and the operating system: it waits for user input, interprets user input to the operating system, and processes various operating system outputs.

[0101] The kernel 541 consists of the parts of the operating system that manage memory, files, peripherals, and system resources. It interacts directly with the hardware and typically runs processes and provides inter-process communication, CPU time slice management, interrupts, memory management, I / O management, and more.

[0102] Applications 543 include autonomous driving-related programs 547, such as programs that manage the autonomous vehicle's interactions with obstacles on the road, control the autonomous vehicle's route or speed, and control the autonomous vehicle's interactions with other autonomous vehicles on the road. Applications 543 also reside on a system called a deployment server 549. In one embodiment, when autonomous driving-related programs 547 are needed, computer system 212 can download them from deployment server 549.

[0103] Sensor 553 is associated with computer system 212. Sensor 553 is used to detect the environment surrounding computer system 212. For example, sensor 553 can detect animals, cars, obstacles, and crosswalks. Furthermore, sensor 553 can detect the environment surrounding these objects, such as animals, cars, obstacles, and crosswalks, for example, the environment surrounding the animals, such as other animals around the animals, weather conditions, and ambient light levels. Alternatively, if computer system 212 is located in a self-driving car, the sensor can be a camera, an infrared sensor, a chemical detector, a microphone, and the like.

[0104] It can be understood that the driving control device of the intelligent driving device in Figure 5 is only an exemplary implementation in the embodiment of the present application. The driving control device applied to the intelligent driving device in the embodiment of the present application includes but is not limited to the above structure.

[0105] In current multi-sensor fusion solutions, the LiDAR device has a fixed scanning cycle, while the cameras are triggered at a fixed frequency. As shown in Figure 6, which is a schematic diagram of the camera position distribution in an intelligent driving device, the device includes a LiDAR device and six cameras, namely, T1, T2, T3, T4, T5, and T6. The LiDAR device is located in the center of the device and emits only one laser beam group, each of which includes at least one laser beam. The LiDAR device's scanning start position can be the location of camera T1, and the device can rotate clockwise from the location of camera T1 to complete a scan cycle. The time it takes to scan each camera is different. For example, the time it takes to scan the location of camera T1 differs significantly from the time it takes to scan the location of camera T6. Consequently, the time difference in the corresponding point cloud data is also significant. A common approach is to position the starting scanning angle with the largest difference in the backward direction, where accuracy requirements are lower. For example, the LiDAR device's scanning starting position could be the location of camera T3. The scan could be rotated clockwise from camera T3, completing a full scan. However, this is only a workaround. While this approach can compensate for point cloud errors, the accuracy of point cloud compensation is poor for oncoming vehicles due to inaccurate speed estimation. Regarding camera triggering, one approach involves simultaneously triggering all six cameras, allowing for simultaneous acquisition of six image data points from each camera, thereby triggering model inference. However, this results in concentrated resource usage and peak load contention, leading to performance issues. Another approach involves time-sharing the triggering of different cameras. Specifically, the LiDAR device triggers exposure only when it scans the corresponding camera position, generating the corresponding image data. This approach achieves optimal matching of the single-view point cloud and image, and provides a degree of load balancing. However, due to the timing differences in when each camera is triggered, the resulting image data can be time-shifted, making effective stitching and fusion impossible in high-speed motion scenarios. Therefore, in the scenario of laser radar and camera fusion, the problem of poor fusion effect due to the mismatch between the laser radar and camera timestamps often occurs, resulting in inaccurate recognition of the vehicle's surrounding environment. To solve the above problem, the embodiment of the present application proposes the following solution.

[0106] Based on the structures of the intelligent driving device provided in FIG. 1 and FIG. 4 and the vehicle driving control device applied to the intelligent driving device provided in FIG. 5 , the present application provides a data fusion method.

[0107] Please refer to Figure 7, which is a flow chart of a data fusion method provided in an embodiment of the present application. The method can be applied to the intelligent driving device described in Figure 1 above, wherein the computing platform can be used to support and execute the method flow steps S701 to S703 shown in Figure 7. The method can be applied to intelligent driving scenarios, including intelligent assisted driving, automatic driving, and unmanned driving, etc., which are not limited by the embodiments of the present application. The method can include the following steps S701 to S703.

[0108] S701. Obtain point cloud data.

[0109] The point cloud data is obtained by detecting the first laser beam group and the second laser beam group in the first scanning direction and the second scanning direction respectively, and specifically can include: acquiring the first data and the second data, the first data being obtained by detecting the first laser beam group in the first scanning direction, and the second data being obtained by detecting the second laser beam group in the second scanning direction, and then, splicing the first data and the second data to obtain the point cloud data.

[0110] In one embodiment, the first laser beam group and the second laser beam group come from the same laser emitting device, which can be called a bidirectional laser radar device. In another embodiment, the first laser beam group comes from the first laser beam emitting device, and the second laser beam group comes from the second laser beam emitting device. In other words, it can be understood that the first laser beam group and the second laser beam group come from different laser emitting devices, and the first laser beam emitting device and the second laser beam group can be laser radar devices. The first laser beam group includes at least one laser beam, and the second laser beam group includes at least one laser beam. In one example, when the first laser beam group and the second laser beam group come from the same laser emitting device, the top view of the first laser beam group and the second laser beam group are shown in Figure 8a, and the side view is shown in Figure 8b. In another example, when the first laser beam group and the second laser beam group come from different laser emitting devices, the top view of the first laser beam group and the second laser beam group are shown in Figure 9a, and the side view is shown in Figure 9b.

[0111] Among them, the difference between the first scanning direction and the second scanning direction can be understood as different rotation directions. For example, the first scanning direction is clockwise and the second scanning direction is counterclockwise; for another example, the first scanning direction is counterclockwise and the second scanning direction is clockwise.

[0112] Among them, point cloud data may refer to partial environmental point cloud data, not complete environmental point cloud data. The first data is the data obtained by the first laser beam group scanning a certain sector range in the first scanning direction, and the second data is the data obtained by the second laser beam group scanning a certain sector range in the second scanning direction. Optionally, the scanning periods of the first laser beam group and the second laser beam group are the same, that is, it can be understood as the same scanning speed. Optionally, the starting scanning positions of the first laser beam group and the second laser beam group are adjustable. Splicing the first data and the second data can be understood as rotating and translating the first data and the second data to a unified coordinate system so that they can form part of the environmental point cloud data, that is, it can be understood as the process of aligning the overlapping parts of the first data and the second data with each other. Stitching can be understood as point cloud splicing, integration, etc., which is not limited in the embodiments of the present application. In one example, as shown in Figure 10, Figure 10 is a schematic diagram of a surrounding environment provided by an embodiment of the present application; wherein, (a) in Figure 11 is a schematic diagram of the first laser beam group detecting and obtaining first data in the first scanning direction, wherein the first scanning direction is counterclockwise based on the driving direction of the intelligent driving device; (b) in Figure 11 is a schematic diagram of the second laser beam group detecting and obtaining second data in the second scanning direction, wherein the second scanning direction is clockwise based on the driving direction of the intelligent driving device; (c) in Figure 11 is a schematic diagram of splicing the first data and the second data to obtain point cloud data.

[0113] S702 : triggering the cameras in the M camera groups to expose in sequence according to the triggering delays of the M camera groups to obtain M groups of image data.

[0114] The cameras in the same camera group are exposed simultaneously. For example, if a camera group includes camera 1 and camera 2, then cameras 1 and camera 2 are exposed simultaneously. The image data in each of the M groups of image data is the same as the number of cameras in each group. For example, if a camera group includes camera 1 and camera 2, then there are two image data in that camera group, meaning that these two image data constitute a group of image data, i.e., the image data corresponding to one camera group. M is a positive integer greater than or equal to 1. M can also be a positive integer greater than or equal to 2.

[0115] Wherein, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and the scanning period can be understood as the scanning speed. The trigger delay is obtained in the following manner: determining the position of each camera group in the M camera groups; determining the trigger delay of each camera group according to the scanning period and the position of each camera group in the M camera groups. The position of each camera in the M camera groups may refer to the spatial position of the camera distribution. For example, the cameras may be distributed in the front, both sides or rear of the intelligent driving device. Wherein, the trigger delay of each camera group is determined according to the scanning period and the position of each camera group in the M camera groups, as shown in formula (1), specifically as follows:

[0116]

[0117] Among them, Δ i,j is the trigger delay of the i-th camera (i∈(0,M]) when the j-th group (j∈(0,2]) of laser beams is scanned, θ i,j is the offset angle of the i-th camera relative to the starting scanning angle of the j-th group of laser beams, rotated according to its scanning direction, is the scanning angular velocity of the jth group of laser beams.

[0118] Optionally, the M camera groups include a front group, a side group, and a rear group. The front group includes a first camera and a second camera, which are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, which are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, which are relatively arranged at the rear of the intelligent driving device. In one example, M=3, and the three camera groups are group 1, group 2, and group 3, respectively. The intelligent driving device includes 6 cameras, namely camera 1, camera 2, camera 3, camera 4, camera 5, and camera 6, respectively; each camera group includes 2 cameras. For specific related descriptions, please refer to Figure 2 above. Optionally, the M camera groups may also include a forward group, which includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device. In one example, M=4, the four camera groups are group 1, group 2, group 3, and group 4, and the intelligent driving device 100 includes six cameras, namely camera 1, camera 2, camera 3, camera 4, camera 5, camera 6, and camera 7; each camera group includes at least one camera, and details can be referred to Figure 3 above.

[0119] The M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is a first delay, and the trigger delay of the second camera group is a second delay. The cameras in the M camera groups are sequentially triggered to expose according to the trigger delays of the M camera groups to obtain M sets of image data, including: starting a first timer to set an initial time; when the time indicated by the first timer is separated from the initial time by a first delay, triggering the cameras in the first camera group to expose to obtain image data corresponding to the first camera group; and when the time indicated by the first timer is separated from the initial time by a second delay, triggering the cameras in the second camera group to expose to obtain image data corresponding to the second camera group.

[0120] The first time delay can be 0 or other values. When the first time delay is 0, it can be understood that when the first timer is started, the camera exposure in the first camera group is triggered to obtain the image data corresponding to the first camera group. Optionally, the first laser beam group and the second laser beam group can respectively detect in the first scanning direction and the second scanning direction at the initial moment. Of course, the first laser beam group and the second laser beam group can also respectively detect in the first scanning direction and the second scanning direction before or after the initial moment, which is not limited in the embodiment of the present application. Optionally, the second time delay can be greater than the first time delay.

[0121] The image data corresponding to the first camera group may refer to image data captured by each camera in the first camera group. For example, if the first camera group includes camera 1 and camera 2, the image data corresponding to the first camera group may refer to image data exposed by camera 1 and image data exposed by camera 2. The same applies to the image data corresponding to the second camera group.

[0122] Among them, the number of first timers can be one or more. When the number of first timers is multiple, the number of first timers is the same as the number of cameras in the camera group. For example, the number of cameras in the first camera group is 2, namely camera 1 and camera 2, then the number of first timers is 2, namely timer 1 and timer 2, wherein timer 1 and timer 2 can be synchronized. For example, the first delay is 0, and timer 1 and timer 2 are started to trigger camera 1 and camera 2 in the first camera group to expose to obtain image data corresponding to the first camera group, that is, timer 1 triggers camera 1 to expose to obtain image data corresponding to camera 1, and timer 2 triggers camera 2 to expose to obtain image data corresponding to camera 2.

[0123] In one example, the first delay is 0 milliseconds (ms), the second delay is 30ms, the first camera group includes camera 1 and camera 2, the second camera group includes camera 3 and camera 4, the first timer is started and the initial time is set to t0, when the indicated time of the first timer is t0, and the interval between the indicated time t0 and the initial time t0 is 0ms by the first delay, camera 1 and camera 2 are triggered to expose to obtain image data captured by camera 1 and image data captured by camera 2. For the environment in Figure 10, as shown in Figure 12, (a) in Figure 12 is the image data captured by camera 1, that is, the image data in the front field of view in the driving direction of the intelligent driving device, and (b) in Figure 12 is the image data captured by camera 2, that is, the image data in the front field of view in the driving direction of the intelligent driving device; when the indication time of the first timer is t1, and the second delay between the indication time t1 and the initial time t0 is 30ms, camera 3 and camera 4 are triggered to expose to obtain the image data captured by camera 3 and the image data captured by camera 4. For the environment in Figure 10, as shown in Figure 13, (a) in Figure 13 is the image data captured by camera 3, that is, the image data in the middle field of view in the driving direction of the intelligent driving device, and (b) in Figure 13 is the image data captured by camera 4, that is, the image data in the middle field of view in the driving direction of the intelligent driving device.

[0124] S703: Fusing the image data and point cloud data corresponding to each camera group in the M camera groups to obtain data for perception processing corresponding to the camera group.

[0125] Among them, fusing the image data and point cloud data corresponding to each camera group in the M camera groups to obtain the data corresponding to the camera group for perception processing can mean taking the image data and point cloud data corresponding to each camera group in the M camera groups as input data, and inputting the input data into a fusion perception algorithm, such as a multimodal fusion algorithm, to obtain the data corresponding to the camera group for perception processing.

[0126] For example, (c) in Figure 11 is a schematic diagram of point cloud data obtained by splicing the first data and the second data based on the environment of Figure 10; as shown in Figure 12, (a) in Figure 12 is the image data captured by camera 1 in the first camera group, and (b) in Figure 12 is the image data captured by camera 2 in the first camera group; accordingly, the point cloud data in (c) in Figure 11 and the image data captured by camera 1 in the first camera group in (a) in Figure 12 and the image data captured by camera 2 in the first camera group in (b) in Figure 12 can be fused to obtain the data for perception processing corresponding to the camera group, that is, to obtain the fusion result of the point cloud data and the image data in the front field of view in the driving direction of the intelligent driving device. Furthermore, as shown in FIG14 , FIG14 is a schematic diagram of the front, middle and rear fields of view of the intelligent driving device in the environment shown in FIG10 . Accordingly, similar to the process of obtaining the fusion result of the point cloud data and the image data in the front field of view, the fusion result of the point cloud data and the image data in the middle field of view in the driving direction of the intelligent driving device and the fusion result of the point cloud data and the image data in the rear field of view in the driving direction of the intelligent driving device can be obtained.

[0127] In one possible implementation, the image data corresponding to each of the M camera groups can be fused. This means that the image data corresponding to at least two of the M camera groups can be used as input data, which is then fed into a camera perception algorithm, such as an image 2D object detection algorithm, to obtain a corresponding result. For example, M = 3, and the three camera groups are camera group 1, camera group 2, and camera group 3, respectively. Each camera group includes two cameras, e.g., camera group 1 includes camera 1 and camera 2, camera group 2 includes camera 3 and camera 4, and camera group 3 includes camera 5 and camera 6. For example, the image data of camera 1 and camera 2 in camera group 1, as well as the image data of camera 3 and camera 4 in camera group 2, can be fed into a camera perception algorithm to obtain a corresponding result. This result can be used, for example, for detecting target obstacles.

[0128] In the above method, based on the driving direction of the intelligent driving device, through this application, the point cloud data and the image data of the camera group corresponding to the front field of view on the intelligent driving device can be fused to improve the fusion effect of the point cloud data and image data of the front field of view. Similarly, the fusion effect of the point cloud data and image data of the middle field of view and the rear field of view can also be improved, thereby making the recognition of the vehicle-surrounding environment more accurate and more conducive to the implementation of autonomous driving task planning.

[0129] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0130] Please refer to Figure 15, which is a structural diagram of a data fusion device 1500 provided in an embodiment of the present application. The data fusion device 1500 is a vehicle-mounted computing platform or a cloud server. The data fusion device 1500 includes: a communication unit 1501 and a processing unit 1502. The device 1500 can be implemented by hardware, software, or a combination of hardware and software.

[0131] The processing unit 1502 is configured to acquire point cloud data, the point cloud data being acquired by detecting the first laser beam group and the second laser beam group in a first scanning direction and a second scanning direction, respectively, wherein the first scanning direction and the second scanning direction are different;

[0132] The processing unit 1502 is configured to sequentially trigger the cameras in the M camera groups to expose according to the trigger delays of the M camera groups to obtain M sets of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1;

[0133] The processing unit 1502 is configured to fuse the image data corresponding to each camera group in the M camera groups with the point cloud data to obtain data for perception processing corresponding to the camera group.

[0134] In one possible implementation, the processing unit 1502 is used to obtain first data, where the first data is obtained by detection of the first laser beam group in the first scanning direction; the processing unit 1502 is used to obtain second data, where the second data is obtained by detection of the second laser beam group in the second scanning direction; the processing unit 1502 is used to splice the first data and the second data to obtain the point cloud data.

[0135] In yet another possible implementation, the first laser beam group and the second laser beam group come from the same laser emitting device.

[0136] In yet another possible implementation, the first laser beam group comes from a first laser emitting device, and the second laser beam group comes from a second laser emitting device.

[0137] In another possible implementation, the processing unit 1502 is configured to determine a position of each camera group in the M camera groups; and determine a trigger delay of each camera group according to the scanning period and the position of each camera group in the M camera groups.

[0138] In another possible implementation, the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is a first delay, and the trigger delay of the second camera group is a second delay; the processing unit 1502 is used to start a first timer to set an initial moment; the processing unit 1502 is used to trigger the camera exposure in the first camera group to obtain image data corresponding to the first camera group when the indicated moment of the first timer and the initial moment are separated by the first delay; the processing unit 1502 is used to trigger the camera exposure in the second camera group to obtain image data corresponding to the second camera group when the indicated moment of the first timer and the initial moment are separated by the second delay.

[0139] In another possible implementation, the M camera groups include a front group, a side group, and a rear group; the front group includes a first camera and a second camera, and the first camera and the second camera are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.

[0140] In yet another possible implementation, the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

[0141] The implementation and beneficial effects of each module may also correspond to the corresponding description of the method embodiment shown in FIG. 7 .

[0142] Please refer to FIG. 16 , which is a schematic diagram of an intelligent driving device 1600 provided in an embodiment of the present application. The intelligent driving device 1600 includes a computing platform 1601 and M camera groups 1602 . The computing platform 1601 is used to:

[0143] Acquire point cloud data, where the point cloud data is obtained by detecting the first laser beam group and the second laser beam group in a first scanning direction and a second scanning direction, respectively, wherein the first scanning direction and the second scanning direction are different;

[0144] triggering the cameras in the M camera groups to expose in sequence according to the trigger delays of the M camera groups to obtain M sets of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1;

[0145] The image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain data for perception processing corresponding to the camera group.

[0146] In one possible implementation, when acquiring the point cloud data, the computing platform 1601 is used to: acquire first data, where the first data is obtained by the first laser beam group detecting in the first scanning direction; acquire second data, where the second data is obtained by the second laser beam group detecting in the second scanning direction; and splice the first data and the second data to acquire the point cloud data.

[0147] In yet another possible implementation, the first laser beam group and the second laser beam group come from the same laser emitting device.

[0148] In yet another possible implementation, the first laser beam group comes from a first laser emitting device, and the second laser beam group comes from a second laser emitting device.

[0149] In another possible implementation, the first laser emitting device and the second laser emitting device are laser radar devices.

[0150] In another possible implementation, the computing platform 1601 is further used to: determine the position of each camera group in the M camera groups; and determine the trigger delay of each camera group based on the scanning period and the position of each camera group in the M camera groups.

[0151] In another possible implementation, the M camera groups include a first camera group and a second camera group, the triggering delay of the first camera group is a first delay, and the triggering delay of the second camera group is a second delay, and the computing platform 1601 is further configured to: start a first timer to set an initial time; and when the first delay elapses between the time indicated by the first timer and the initial time, trigger the cameras in the first camera group to expose the image data corresponding to the first camera group; and when the second delay elapses between the time indicated by the first timer and the initial time, trigger the cameras in the second camera group to expose the image data corresponding to the second camera group.

[0152] In another possible implementation, the M camera groups include a front group, a side group, and a rear group; the front group includes a first camera and a second camera, and the first camera and the second camera are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.

[0153] In yet another possible implementation, the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

[0154] The implementation and beneficial effects of each module may also correspond to the corresponding description of the method embodiment shown in FIG. 7 .

[0155] Please refer to Figure 17, which is a schematic diagram of a data fusion device 1700 provided in an embodiment of the present application. The data fusion device 1700 can be an in-vehicle computing platform or a cloud server. The data fusion device 1700 includes at least one processor 1701 and a communication interface 1703, and optionally also includes a memory 1702. The processor 1701, memory 1702, and communication interface 1703 are interconnected via a bus 1704. Optionally, the data fusion device 1700 can be an in-vehicle computing platform or a cloud server.

[0156] Memory 1702 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 1702 is used for storing computer programs and data. Communication interface 1703 is used to receive and send data.

[0157] The processor 1701 may be one or more central processing units (CPUs). When the processor 1701 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0158] The processor 1701 in the data fusion device 1700 is configured to read the computer program code stored in the memory 1702 and perform the following operations:

[0159] Acquire point cloud data, where the point cloud data is obtained by detecting the first laser beam group and the second laser beam group in a first scanning direction and a second scanning direction, respectively, wherein the first scanning direction and the second scanning direction are different;

[0160] triggering the cameras in the M camera groups to expose in sequence according to the trigger delays of the M camera groups to obtain M sets of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1;

[0161] The image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain data for perception processing corresponding to the camera group.

[0162] In one possible implementation, the processor 1701 is used to obtain first data, where the first data is obtained by the first laser beam group detecting in the first scanning direction; the processor 1701 is used to obtain second data, where the second data is obtained by the second laser beam group detecting in the second scanning direction; the processor 1701 is used to splice the first data and the second data to obtain the point cloud data.

[0163] In yet another possible implementation, the first laser beam group and the second laser beam group come from the same laser emitting device.

[0164] In yet another possible implementation, the first laser beam group comes from a first laser emitting device, and the second laser beam group comes from a second laser emitting device.

[0165] In another possible implementation, the processor 1701 is configured to determine a position of each camera group in the M camera groups; and determine a trigger delay of each camera group according to the scanning period and the position of each camera group in the M camera groups.

[0166] In another possible implementation, the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is a first delay, and the trigger delay of the second camera group is a second delay; the processor 1701 is used to start a first timer to set an initial moment; the processor 1701 is used to trigger the camera exposure in the first camera group to obtain image data corresponding to the first camera group when the indicated moment of the first timer and the initial moment are separated by the first delay; the processor 1701 is used to trigger the camera exposure in the second camera group to obtain image data corresponding to the second camera group when the indicated moment of the first timer and the initial moment are separated by the second delay.

[0167] In another possible implementation, the M camera groups include a front group, a side group, and a rear group; the front group includes a first camera and a second camera, and the first camera and the second camera are relatively arranged at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on both sides of the intelligent driving device; the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.

[0168] In yet another possible implementation, the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

[0169] The implementation and beneficial effects of each operation may also correspond to the corresponding description of the method embodiment shown in FIG. 7 .

[0170] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0171] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or nonvolatile storage medium, or may include both volatile and nonvolatile types of storage media.

[0172] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

Claims

1. A data fusion method, characterized in that: Applied in intelligent driving scenarios, the method includes: Acquire point cloud data, where the point cloud data is obtained by detecting the first laser beam group and the second laser beam group in a first scanning direction and a second scanning direction, respectively, wherein the first scanning direction and the second scanning direction are different; triggering the cameras in the M camera groups to expose in sequence according to the trigger delays of the M camera groups to obtain M sets of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1; The image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain data for perception processing corresponding to the camera group.

2. The method according to claim 1, characterized in that The acquiring of point cloud data comprises: Acquire first data, where the first data is detected by the first laser beam group in the first scanning direction; Acquiring second data, where the second data is detected by the second laser beam group in the second scanning direction; The first data and the second data are spliced ​​together to obtain the point cloud data.

3. The method according to claim 1 or 2, characterized in that The first laser beam group and the second laser beam group come from the same laser emitting device.

4. The method according to claim 1 or 2, characterized in that The first laser beam group comes from a first laser emitting device, and the second laser beam group comes from a second laser emitting device.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining a position of each camera group in the M camera groups; The trigger delay of each camera group is determined according to the scanning period and the position of each camera group in the M camera groups.

6. The method according to any one of claims 1 to 5, characterized in that The M camera groups include a first camera group and a second camera group, the triggering delay of the first camera group is a first delay, and the triggering delay of the second camera group is a second delay; The method of sequentially triggering the exposure of cameras in the M camera groups according to the trigger delays of the M camera groups to obtain M groups of image data includes: Starting a first timer to set an initial time; When the interval between the indicated time of the first timer and the initial time is the first time delay, triggering the cameras in the first camera group to expose to obtain image data corresponding to the first camera group; When the second delay elapses between the indicated time of the first timer and the initial time, the cameras in the second camera group are triggered to expose to obtain picture data corresponding to the second camera group.

7. The method according to any one of claims 1 to 6, characterized in that The M camera groups include a front group, a side group, and a rear group; The front group includes a first camera and a second camera, and the first camera and the second camera are arranged opposite to each other at the front of the intelligent driving device; The side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on two sides of the intelligent driving device; The rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are arranged opposite to each other at the rear of the intelligent driving device.

8. The method according to claim 7, characterized in that The M camera groups further include a forward group, the forward group including a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

9. An intelligent driving device, characterized in that: The system comprises a computing platform and M camera groups, wherein the computing platform is used to: Acquire point cloud data, where the point cloud data is obtained by detecting the first laser beam group and the second laser beam group in a first scanning direction and a second scanning direction, respectively, wherein the first scanning direction and the second scanning direction are different; triggering the cameras in the M camera groups to expose in sequence according to the trigger delays of the M camera groups to obtain M sets of image data; wherein the cameras in the same camera group are exposed simultaneously, the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1; The image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain data for perception processing corresponding to the camera group.

10. The intelligent driving device according to claim 9, characterized in that: When acquiring the point cloud data, the computing platform is used to: Acquire first data, where the first data is detected by the first laser beam group in the first scanning direction; Acquiring second data, where the second data is detected by the second laser beam group in the second scanning direction; The first data and the second data are spliced ​​together to obtain the point cloud data.

11. The intelligent driving device according to claim 9 or 10, characterized in that: The first laser beam group and the second laser beam group come from the same laser emitting device.

12. The intelligent driving device according to claim 9 or 10, characterized in that: The intelligent driving device further includes: a first laser emitting device and a second laser emitting device; The first laser beam group comes from the first laser emitting device, and the second laser beam group comes from the second laser emitting device.

13. The intelligent driving device according to claim 12, characterized in that: The first laser emitting device and the second laser emitting device are laser radar devices.

14. The intelligent driving device according to any one of claims 9 to 13, characterized in that: The computing platform is also used to: Determining a position of each camera group in the M camera groups; The trigger delay of each camera group is determined according to the scanning period and the position of each camera group in the M camera groups.

15. The intelligent driving device according to any one of claims 9 to 14, characterized in that: The M camera groups include a first camera group and a second camera group, a trigger delay of the first camera group is a first delay, and a trigger delay of the second camera group is a second delay, and the computing platform is further configured to: Starting a first timer to set an initial time; When the indicated time of the first timer is separated from the initial time by the first time delay, triggering the cameras in the first camera group to expose to obtain image data corresponding to the first camera group; When the second delay elapses between the indicated time of the first timer and the initial time, cameras in the second camera group are triggered to expose to obtain picture data corresponding to the second camera group.

16. The intelligent driving device according to any one of claims 9 to 15, characterized in that: The M camera groups include a front group, a side group, and a rear group; The front group includes a first camera and a second camera, and the first camera and the second camera are arranged opposite to each other at the front of the intelligent driving device; The side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively arranged on two sides of the intelligent driving device; The rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are arranged opposite to each other at the rear of the intelligent driving device.

17. The intelligent driving device according to claim 16, characterized in that: The M camera groups further include a forward group, the forward group including a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.

18. A data fusion device, characterized in that: The data fusion device comprises a processor and a memory, wherein the memory and the processor are interconnected via a line, and the processor is used to obtain a computer program stored in the memory. When the computer program is executed by the processor, the data fusion device is used to implement the method according to any one of claims 1 to 8.

19. The data fusion device according to claim 18, characterized in that: The data fusion device is a vehicle-mounted computing platform or a cloud server.

Images