Sensor data synchronous acquisition and processing method, system, device and electronic equipment

Abstract

The embodiments of the present application relate to a method, system, device and electronic device for synchronous acquisition and processing of sensor data, the method comprising: determining the first and second perspective boundary lines of the camera based on the viewing angle range and the main optical axis of the camera on the vehicle; determining the first moment, the second moment and the third moment when the point cloud acquisition device on the vehicle scans the first perspective boundary line, the second perspective boundary line and the main optical axis; setting the trigger shooting moment of the camera based on the third moment; determining the camera posture corresponding to the camera at the third moment, and establishing a reference coordinate system based on the camera posture; mapping the point cloud data set collected between the first moment and the second moment to the reference coordinate system based on the first external parameter of the camera and the second external parameter of the point cloud acquisition device. The embodiments of the present application realize the real-time synchronous acquisition and coordinate system alignment of image and point cloud data, improve the efficiency of synchronous acquisition and processing of sensor data, and have a wider range of scene applications.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle technology, and in particular to a method, system, device and electronic equipment for synchronously collecting and processing sensor data. Background Art

[0002] With the development of intelligent vehicle technology, in order to ensure the safety of vehicle driving, more sensors need to be installed on the vehicle to collect various data and provide data support for intelligent driving algorithms.

[0003] In the data collection scheme for different models, the differences between models need to be considered. Different models may have differences in sensor configuration, body size, suspension system, etc. Therefore, the current vehicle data collection system needs to customize the data collection scheme according to the characteristics of the specific model, including selecting the appropriate sensor type, installation location and calibration method, and adjusting the data processing and analysis algorithm.

[0004] To achieve the generalization of data collection solutions for different vehicle models, the data collection solutions need to be flexible and scalable enough to adapt to the differences in sensor configuration, data interface, etc. of different vehicle models. The existing image data captured by cameras and point cloud data collected by lidar do not provide a unified data synchronization collection and perspective alignment solution, resulting in the inability of image and point cloud processing to adapt to the actual driving scene of the vehicle. The scene adaptability and solution scalability of data collection are insufficient, which has a great impact on the efficiency and accuracy of executing environmental perception algorithms using the collected data. Summary of the invention

[0005] In view of this, in order to solve some or all of the above-mentioned technical problems, the embodiments of the present application provide a sensor data synchronous acquisition and processing method, system, device and electronic device.

[0006] In a first aspect, an embodiment of the present application provides a method for synchronously acquiring and processing sensor data, the method comprising: determining a first viewing angle boundary line and a second viewing angle boundary line of the camera based on the viewing angle range and the main optical axis of the camera on the vehicle; determining a first moment when a point cloud acquisition device on the vehicle scans the first viewing angle boundary line, a second moment when the point cloud acquisition device scans the second viewing angle boundary line, and a third moment when the point cloud acquisition device scans the main optical axis; based on the third moment, setting a trigger shooting moment of the camera so that the camera captures an environmental image around the vehicle at the third moment; determining a camera posture corresponding to the camera at the third moment, and establishing a reference coordinate system based on the camera posture; obtaining a first external parameter of the camera and a second external parameter of the point cloud acquisition device; based on the first external parameter and the second external parameter, mapping a point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system.

[0007] In one possible embodiment, determining a first moment when a point cloud acquisition device on a vehicle scans a first viewing angle boundary line, a second moment when a point cloud acquisition device scans a second viewing angle boundary line, and a third moment when a point cloud acquisition device on a vehicle scans a main optical axis includes: determining a scanning period, a scanning starting direction, a scanning starting moment, and a scanning angle range of the point cloud acquisition device, wherein the scanning starting direction is the starting direction in which the point cloud acquisition device acquires point cloud data in each scanning period, and the scanning starting moment is the moment when the point cloud acquisition device scans to the scanning starting direction; determining a scanning angular velocity of the point cloud acquisition device based on the scanning period and the scanning angle range; determining a first angle between the first viewing angle boundary line and the scanning starting direction, and a second angle between the second viewing angle boundary line and the scanning starting direction; determining a first moment when the point cloud acquisition device scans the first viewing angle boundary line, and a second moment when the point cloud acquisition device scans the second viewing angle boundary line based on the first angle, the second angle, the scanning angular velocity, and the scanning starting moment; determining a third angle between the main optical axis and the first viewing angle boundary line, and determining a third moment when the point cloud acquisition device scans the main optical axis of the camera based on the third angle, the scanning angular velocity, and the first moment.

[0008] In one possible implementation, determining a camera pose corresponding to a third moment includes: acquiring at least two frames of navigation data generated by a navigation device between a first moment and a second moment; determining baseline navigation data from the at least two frames of navigation data; and determining a camera pose corresponding to the third moment based on the baseline navigation data.

[0009] In a possible implementation, determining the reference navigation data from at least two frames of navigation data includes: determining, from the at least two frames of navigation data, navigation data generated at a time adjacent to the third time as the reference navigation data.

[0010] In one possible implementation, based on a first extrinsic parameter and a second extrinsic parameter, a set of point cloud data collected by a point cloud acquisition device between a first moment and a second moment is mapped to a reference coordinate system, including: determining first coordinate conversion data for converting a coordinate system of at least two frames of navigation data to a coordinate system of a camera, and second coordinate conversion data for converting a coordinate system of the camera to a coordinate system of the point cloud acquisition device based on the first extrinsic parameter and the second extrinsic parameter; for each frame of navigation data other than the reference navigation data in the at least two frames of navigation data, determining a subset of point cloud data corresponding to a moment when the frame of navigation data is generated; determining third coordinate conversion data of a conversion relationship between a vehicle posture represented by the frame of navigation data and a vehicle posture represented by the reference navigation data; based on the first coordinate conversion data, the second coordinate conversion data and the third coordinate conversion data, mapping a subset of point cloud data corresponding to a moment when the frame of navigation data is generated to the reference coordinate system.

[0011] In one possible implementation, determining the point cloud data subset corresponding to the generation time of the frame navigation data includes: determining the point cloud data collected between the generation time of the frame navigation data and the generation time of the next frame navigation data as the point cloud data subset corresponding to the generation time of the frame navigation data.

[0012] In one possible implementation, based on the third moment, the trigger shooting moment of the camera is set so that the camera captures the environmental image around the vehicle at the third moment, including: based on the exposure time of the camera, setting the exposure start moment of the camera so that the camera ends the exposure at the third moment to obtain the environmental image.

[0013] In a second aspect, an embodiment of the present application provides a sensor data synchronous acquisition and processing system, which is arranged on a vehicle, and includes: a visual controller, a camera, a point cloud acquisition device, a gateway, a data processing module and a data storage module; the visual controller is connected to the camera and the gateway, and is used to control the camera to capture the environmental image around the vehicle, and send the environmental image to the data processing module through the gateway; the point cloud acquisition device is connected to the gateway, and is used to collect point cloud data sets around the vehicle, and send the point cloud data sets to the data processing module through the gateway; the data processing module is used to execute the sensor data synchronous acquisition and processing method described in any embodiment of the first aspect above, to obtain environmental images and point cloud data sets with synchronized acquisition time and in the same coordinate system, and store the environmental images and point cloud data sets in the data storage module.

[0014] In a third aspect, an embodiment of the present application provides a sensor data synchronous acquisition and processing device, which includes: a first determination module, used to determine a first perspective boundary line and a second perspective boundary line of a camera based on the perspective range and the main optical axis of the camera on the vehicle; a second determination module, used to determine a first moment when a point cloud acquisition device on the vehicle scans the first perspective boundary line, a second moment when it scans the second perspective boundary line, and a third moment when it scans the main optical axis; a setting module, used to set a trigger shooting moment of the camera based on the third moment, so that the camera captures an environmental image around the vehicle at the third moment; a third determination module, used to determine the camera posture corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera posture; an acquisition module, used to acquire a first external parameter of the camera and a second external parameter of the point cloud acquisition device; a mapping module, used to map a point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system based on the first external parameter and the second external parameter.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the method of any embodiment of the sensor data synchronization acquisition and processing method of the above-mentioned first aspect of the present application is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method of any embodiment of the sensor data synchronization acquisition and processing method of the first aspect described above is implemented.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program, which includes a computer-readable code. When the computer-readable code runs on a device, the processor in the device implements a method as described in any embodiment of the sensor data synchronization acquisition and processing method of the first aspect above.

[0018] The sensor data synchronous acquisition and processing method, system, device and electronic device provided in the embodiments of the present application, by determining the first moment and the second moment when the point cloud acquisition device on the vehicle scans the two viewing angle boundary lines of the camera, and the third moment when the point cloud acquisition device scans the main optical axis of the camera, based on the third moment, sets the trigger shooting moment of the camera, then determines the camera posture at the third moment, and establishes a reference coordinate system based on the camera posture, and finally maps the point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system based on the external parameters of the camera and the point cloud acquisition device. The embodiments of the present application only need to obtain the camera's viewing angle range, main optical axis direction, camera posture, external parameters and other characteristic parameters, as well as the scanning speed, external parameters and other characteristic parameters of the point cloud acquisition device, to perform real-time synchronous acquisition and coordinate system alignment of the image and point cloud data, thereby improving the efficiency of synchronous acquisition and processing of sensor data, helping to improve the execution efficiency and accuracy of the intelligent driving algorithm, and realizing that data synchronous acquisition and processing can be effectively performed for various vehicles, cameras, and point cloud acquisition devices, and the scope of application of scenarios is wider. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0022] Figure 1 A schematic diagram of a flow chart of a sensor data synchronous acquisition and processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the viewing angle ranges of multiple types of cameras installed on a vehicle provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the relative relationship between a camera viewing angle range, a main optical axis and a vehicle orientation provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a flow chart of another sensor data synchronous acquisition and processing method provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of another sensor data synchronous acquisition and processing method provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of another sensor data synchronous acquisition and processing method provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the data collection timing of the navigation device, point cloud collection device, and surround view camera provided in the embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a sensor data synchronous acquisition and processing system provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of a sensor data synchronous acquisition and processing device provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of the components and steps, the numerical expressions and the numerical values ​​described in these embodiments do not limit the scope of the present application.

[0033] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present application are only used to distinguish between different steps, devices, modules and other objects, and do not represent any specific technical meanings, nor do they indicate the logical order between them.

[0034] It should also be understood that in this embodiment, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present application can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0036] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be understood that the description of the various embodiments in this application focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0039] Technologies, circuits, and devices known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, circuits, and devices should be considered as part of the specification.

[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. To facilitate the understanding of the embodiments of the present application, the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] In order to solve the technical problems of low efficiency and weak scene adaptability of vehicle sensor data synchronous acquisition and processing in the prior art, the present application provides a sensor data synchronous acquisition and processing method, which can perform sensor data synchronous acquisition and processing in a universal manner according to the characteristics of cameras and point cloud acquisition devices, with stronger scene adaptability and higher data acquisition and processing efficiency.

[0043] Figure 1 A flow chart of a sensor data synchronous acquisition and processing method provided for an embodiment of the present application. This method can be applied to a vehicle, and the execution subject of this method may include but is not limited to various electronic devices such as a vehicle controller, a server, a smart phone, a portable computer, etc. In addition, the execution subject of this method may be hardware or software. When the above-mentioned execution subject is hardware, the execution subject may be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the above-mentioned execution subject is software, this method can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0044] like Figure 1 As shown, the method specifically includes:

[0045] Step 101 : determining a first viewing angle boundary line and a second viewing angle boundary line of a camera on a vehicle based on a viewing angle range and a main optical axis of the camera.

[0046] In this embodiment, the camera on the vehicle may be a camera installed outside the vehicle to capture the surrounding environment of the vehicle, and the type and installation position of the camera may be set arbitrarily. For example, it may be any of the following cameras: a far-field camera installed above the front windshield of the vehicle, an electronic rearview mirror camera, a surround wide-angle camera installed around the vehicle body, etc.

[0047] like Figure 2 As shown in FIG. 1 , it shows a schematic diagram of the viewing angle range of multiple types of cameras installed on a vehicle. Each sector area in the figure represents the viewing angle range of a camera. Figure 2 It can be seen that the cameras on the vehicle include a forward far-field camera, a forward wide-angle camera, four surround fisheye cameras, two electronic rearview mirror cameras, etc., which can cover all angles centered on the vehicle.

[0048] The viewing angle of a camera is determined by the characteristic parameters of the camera, and the direction of the main optical axis of the camera is determined by the installation position of the camera. According to the viewing angle and the direction of the main optical axis, two boundary lines of the viewing angle can be determined on a reference plane (such as a horizontal plane).

[0049] For example, Figure 3As shown, the camera may be a surround wide-angle camera installed on the right side of the vehicle, with a viewing angle of θ (e.g., 200°), a main optical axis o on a horizontal plane, and in a clockwise direction, forming an angle of 90° with the vehicle's direction d. L1 is a first viewing angle boundary line, and L2 is a second viewing angle boundary line.

[0050] Step 102, determining a first moment when a point cloud acquisition device on the vehicle scans a first viewing angle boundary line, a second moment when a point cloud acquisition device on the vehicle scans a second viewing angle boundary line, and a third moment when a point cloud acquisition device on the vehicle scans a main optical axis.

[0051] In this embodiment, the point cloud acquisition device is usually a laser radar, which is used to periodically scan the environment around the vehicle within the scanning range to obtain point cloud data representing the position of each point in the space around the vehicle. At least one point cloud acquisition device can be installed on the vehicle, and the point cloud acquisition device targeted by this embodiment can be any one.

[0052] Specifically, the starting scanning direction of the point cloud acquisition device can be set, and based on the frame rate, scanning period and other characteristic parameters of the point cloud acquisition device, the starting scanning direction can be used as the starting moment to determine in real time the angle between the current scanning direction of the point cloud acquisition device and the starting scanning direction. Based on the angle, it can be determined whether the scanning direction coincides with the first viewing angle boundary line, the second viewing angle boundary line, and the main optical axis, thereby determining that the coincident moments are the first moment, the second moment, and the third moment respectively.

[0053] Step 103, based on the third moment, setting the trigger shooting moment of the camera, so that the camera captures the environment image around the vehicle at the third moment.

[0054] In this embodiment, after the third moment is determined, the trigger shooting moment of the camera can be set. For example, after the current moment reaches the third moment, a trigger shooting command can be sent to the camera so that the camera captures the environment image. Usually, the camera shoots continuously, that is, the camera frame rate is fixed, and the camera frame rate is greater than the frame rate of the point cloud acquisition device. Therefore, the camera can be re-triggered to shoot each time the third moment is reached, and the method is re-executed next time to determine the third moment, and before the third moment is reached again, the image is captured at a fixed frame rate.

[0055] Step 104: determine the camera posture corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera posture.

[0056] In this embodiment, since the vehicle is moving in real time, the position and posture of the camera relative to the ground is also changing in real time, that is, it is changing in real time relative to the world coordinate system. In order to make the point cloud data and the environmental image in the same coordinate system, it is necessary to set a reference coordinate system. One coordinate axis of the reference coordinate system is consistent with the direction of the main optical axis of the camera at the third moment. Therefore, the reference coordinate system can be considered to be consistent with the camera coordinate system when shooting the environmental image. The camera posture can be determined using inertial navigation equipment, satellite navigation equipment, etc. That is, the vehicle posture can be determined using inertial navigation equipment and satellite navigation equipment, and the camera posture in the world coordinate system can be determined based on the vehicle posture, the position of the camera relative to the vehicle, and the lens orientation.

[0057] Step 105, obtaining a first external parameter of the camera and a second external parameter of the point cloud acquisition device.

[0058] In this embodiment, the external parameters of the camera and the point cloud acquisition device can be pre-calibrated. The external parameters are used to convert between the device coordinate system and the world coordinate system, that is, based on the first external parameter, each point in the environment image taken by the camera can be converted to the world coordinate system; based on the second external parameter, the point cloud data collected by the point cloud acquisition device can be converted to the world coordinate system.

[0059] Step 106 , based on the first external parameter and the second external parameter, map the point cloud data set collected by the point cloud collection device between the first moment and the second moment into a reference coordinate system.

[0060] In this embodiment, since the reference coordinate system is obtained through the camera posture on the basis of the world coordinate system, the conversion relationship between each point in the environment image and the world coordinate system, as well as the conversion relationship between each point cloud in the point cloud data set and the world coordinate system can be determined by using the first external parameter and the second external parameter, and then the conversion relationship between the environment image and the point cloud data set and the reference coordinate system can be determined. According to the conversion relationship, the environment image and the point cloud data set can be mapped to the reference coordinate system. Since the reference coordinate system is consistent with the camera coordinate system when the environment image is taken, the mapped point cloud data set and the environment image are in the same coordinate system, and after the acquisition time is synchronized, it can be considered that the environment image and the point cloud data set are collected at the same time. Therefore, these data can accurately represent the positional relationship of various objects in the vehicle's surrounding environment at the same time, and then these data can be used in algorithms such as obstacle position detection and automatic parking.

[0061] The sensor data synchronous acquisition and processing method provided in the embodiment of the present application determines the first and second moments when the point cloud acquisition device on the vehicle scans the two viewing angle boundary lines of the camera, and the third moment when the point cloud acquisition device scans the main optical axis of the camera, and sets the trigger shooting moment of the camera based on the third moment, then determines the camera posture at the third moment, and establishes a reference coordinate system based on the camera posture, and finally maps the point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system based on the external parameters of the camera and the point cloud acquisition device. The embodiment of the present application only needs to obtain the camera's viewing angle range, main optical axis direction, camera posture, external parameters and other characteristic parameters, as well as the scanning speed, external parameters and other characteristic parameters of the point cloud acquisition device, so as to perform real-time synchronous acquisition and coordinate system alignment of the image and point cloud data, thereby improving the efficiency of synchronous acquisition and processing of sensor data, helping to improve the execution efficiency and accuracy of the intelligent driving algorithm, and realizing that data synchronous acquisition and processing can be effectively performed for various vehicles, cameras, and point cloud acquisition devices, and the scope of application of scenarios is wider.

[0062] In some optional implementations of this embodiment, such as Figure 4 As shown, step 102 includes:

[0063] Step 1021, determining the scanning period, scanning starting direction, scanning starting time and scanning angle range of the point cloud acquisition device.

[0064] The scanning start direction is the starting direction in which the point cloud acquisition device collects point cloud data in each scanning cycle, and the scanning start time is the time when the point cloud acquisition device scans to the scanning start direction. Figure 2 As shown, ray d is the direction of the vehicle, which can be used as the starting direction of the scan. The above scanning angle range is the rotation angle of the scanning line of the point cloud acquisition device in one scanning cycle. For example, the scanning angle range is 360°, that is, in one scanning cycle, the point cloud acquisition device scans the 360° range on the horizontal plane once. The scanning cycle can be determined by the frame rate of the point cloud acquisition device. For example, if the frame rate of the point cloud acquisition device is 10Hz, the scanning cycle is 100ms.

[0065] Step 1022: Determine the scanning angular velocity of the point cloud acquisition device based on the scanning period and the scanning angle range.

[0066] Specifically, the scanning angle range may be divided by the scanning period to obtain the scanning angular velocity. For example, if the scanning angle range is 360° and the scanning period is 100 ms, the scanning angular velocity is 360° / 100 ms.

[0067] Step 1023, determining a first angle between the first viewing angle boundary line and the scanning start direction, and a second angle between the second viewing angle boundary line and the scanning start direction.

[0068] Usually, when determining the angle, you need to set the rotation direction. Figure 2 As shown, assuming that the first angle is α1, the second angle is α2, taking the scanning starting direction as the reference, and the clockwise direction as the positive direction, the first angle α1 between the first viewing angle boundary line and the scanning starting direction is 90°-θ / 2, and the second angle between the second viewing angle boundary line and the scanning starting direction is 90°+θ / 2.

[0069] Step 1024, based on the first angle, the second angle, the scanning angular velocity and the scanning start time, determine the first time when the point cloud acquisition device scans the first viewing angle boundary line and the second time when the point cloud acquisition device scans the second viewing angle boundary line.

[0070] Specifically, assuming that the first angle is α1, the second angle is α2, the scanning start time is t0, and the scanning angular velocity is v, the first time is expressed as follows:

[0071] t1=t0+α1 / v (1)

[0072] The second moment is shown in the following formula (2):

[0073] t2=t0+α2 / v (2)

[0074] If θ is 200°, then α1=-10°, α2=190°, v=360° / 100ms. Substituting α1 and α2 into the above formula, we can calculate that t1=t0-10 / (360 / 100), t2=t0+90 / (360 / 100).

[0075] Step 1025, determine a third angle between the main optical axis and the first viewing angle boundary line, and determine a third moment when the point cloud acquisition device scans the main optical axis of the camera based on the third angle, the scanning angular velocity and the first moment.

[0076] Specifically, Figure 2 As shown, assuming that the third angle is α3, the calculation method of the third moment is as shown in the following formula (3):

[0077] t3=t1+α3 / v (3)

[0078] By determining various characteristic parameters of the point cloud acquisition device, this embodiment can use the characteristic parameters in actual data acquisition scenarios to accurately calculate the moment when the point cloud acquisition device scans the main optical axis and two viewing angle boundary lines of the camera, thereby helping to accurately set the camera's shooting moment and improve the accuracy of synchronous data acquisition between the camera and the point cloud acquisition device.

[0079] In some optional implementations of this embodiment, such as Figure 5 As shown, step 104 includes:

[0080] Step 1041: Acquire at least two frames of navigation data generated by a navigation device between a first moment and a second moment.

[0081] The navigation device may include at least one of the following: an inertial navigation device, a satellite navigation device, etc. The navigation device may be used to determine physical quantities such as the vehicle's direction of travel, speed, and inclination angle relative to the ground, and the generated navigation data may represent the above physical quantities. Optionally, the navigation data may be generated by combining an inertial navigation device, that is, combining an inertial navigation device with a satellite navigation device to generate high-precision navigation data.

[0082] Generally, the frame rate of the navigation device is greater than the frame rate of the camera and the point cloud acquisition device. For example, the frame rate of the point cloud acquisition device is generally 10 Hz, the frame rate of the camera is generally 30 Hz, and the frame rate of the navigation device is generally 100 Hz. Therefore, multiple frames of navigation data can be generated between the first moment and the second moment. According to the timestamp of generating the navigation data, the above-mentioned at least two frames of navigation data can be obtained.

[0083] Step 1042, determining reference navigation data from at least two frames of navigation data.

[0084] Optionally, any navigation data in at least two frames of navigation data may be determined as reference navigation data.

[0085] Step 1043: Determine the camera pose corresponding to the third moment based on the reference navigation data.

[0086] Specifically, since the reference navigation data can represent the position, direction, tilt angle and other characteristics of the vehicle in the world coordinate system, and the camera position and the direction of the main optical axis are fixed relative to the vehicle, the camera's posture can be determined in the world coordinate system. Optionally, the changing trend of the vehicle's posture can be determined based on the above-mentioned at least two frames of navigation data, for example, the changing trend of the vehicle's posture can be represented by the angular velocity of the vehicle when turning, the vehicle's driving speed, the vehicle's displacement distance, etc. Then, based on these data, the posture represented by the reference navigation data is used as the initial posture of the vehicle to predict the vehicle's posture corresponding to the third moment. Finally, the camera's posture corresponding to the third moment is calculated based on the camera's installation position on the vehicle and the lens orientation.

[0087] This embodiment obtains multiple frames of navigation data from the navigation device on the vehicle and determines the baseline navigation data therefrom. The position and posture of the camera at the third moment can be accurately determined based on the baseline navigation data, thereby helping to improve the efficiency and accuracy of establishing the baseline coordinate system.

[0088] In some optional implementations of this embodiment, step 1042 may be performed as follows:

[0089] From the at least two frames of navigation data, navigation data generated at a time point adjacent to the third time point is determined as reference navigation data.

[0090] Each frame of navigation data in the at least two frames of navigation data corresponds to a generation time. Usually, the time closest to the third time can be determined as the time adjacent to the third time used in this step, that is, the frame of navigation data whose generation time is closest to the third time can be determined as the reference navigation data.

[0091] This embodiment determines the navigation data generated at a moment adjacent to the third moment as the reference navigation data. The moment is close to the third moment. Therefore, the reference navigation data can represent the position and posture of the camera at the third moment without any calculation, thereby helping to improve the efficiency and accuracy of establishing a reference coordinate system.

[0092] In some optional implementations of this embodiment, such as Figure 6 As shown, step 106 includes:

[0093] Step 1061: Acquire at least two frames of navigation data generated by a navigation device between a first moment and a second moment.

[0094] Specifically, since the frame rate of the navigation device is relatively high, at least two frames of navigation data may be generated between the first moment and the second moment. The at least two frames of navigation data may be acquired according to the timestamp of generating the navigation data.

[0095] Step 1062, based on the first external parameter and the second external parameter, determine the first coordinate conversion data for converting the coordinate system of at least two frames of navigation data to the coordinate system of the camera, and the second coordinate conversion data for converting the coordinate system of the camera to the coordinate system of the point cloud acquisition device.

[0096] Specifically, the coordinate system of the navigation data can be a world coordinate system. Therefore, the first external parameter can be used to determine the coordinate conversion relationship between the camera coordinate system and the coordinate system of the navigation data. Since the first external parameter and the second external parameter both represent the conversion relationship between the device coordinate system and the world coordinate system, the conversion relationship between the camera coordinate system and the point cloud acquisition device coordinate system can be determined based on the first external parameter and the second external parameter. The above-mentioned first coordinate conversion data and the second coordinate conversion data can be in the form of a rotation and translation matrix, that is, the rotation and translation matrix includes a rotation amount and a translation amount.

[0097] Step 1063, for each frame of navigation data other than the baseline navigation data in at least two frames of navigation data, determine the point cloud data subset corresponding to the generation time of the frame navigation data; determine the third coordinate transformation data of the transformation relationship between the vehicle posture represented by the frame navigation data and the vehicle posture represented by the baseline navigation data; based on the first coordinate transformation data, the second coordinate transformation data and the third coordinate transformation data, map the point cloud data subset corresponding to the generation time of the frame navigation data to the baseline coordinate system.

[0098] Specifically, as an example, Figure 7 As shown, it shows a schematic diagram of the data collection timing of the navigation device, point cloud collection device, and surround view camera. Figure 7 In the example shown, the point cloud acquisition device continuously acquires point cloud data within one acquisition cycle, and the navigation device acquires 5 frames of navigation data between the first moment t1 and the second moment t2, namely F0, F1, F2, F3, and F4, wherein F2 is the reference navigation data, and its corresponding moment is close to the third moment t3, and moment t3 is the end time of exposure of the surround camera, then step 1063 is executed for the four frames of navigation data F0, F1, F3, and F4, respectively.

[0099] For the navigation data F0, the subset of point cloud data collected corresponding to the generation time of the frame navigation data can be first obtained from the above point cloud data set. Then, the relative posture between the vehicle posture corresponding to F0 and the vehicle posture corresponding to F2 is determined, that is, the relative rotation and translation of the vehicle posture at the two moments can be compared, so as to obtain the third coordinate transformation data (usually in the form of a rotation and translation matrix) containing the relative rotation and relative translation. Finally, based on the conversion relationship between the coordinate systems represented by the above first coordinate transformation data, the second coordinate transformation data and the third coordinate transformation data, the subset of point cloud data corresponding to F0 can be mapped to the reference coordinate system.

[0100] As an example, suppose the first coordinate conversion data is Represents the conversion relationship between the coordinate system of the navigation data and the camera coordinate system. The second coordinate conversion data is Represents the conversion relationship between the camera coordinate system and the coordinate system of the point cloud acquisition device. Assume that the third coordinate conversion data between F0 and the reference navigation data F2 is The point cloud data subset corresponding to F0 is P 0 According to the following formula, P 0 Convert to the base coordinate system:

[0101]

[0102] Where R is the rotation amount, t is the translation amount, and the rotation and translation matrix is ​​shown in the following formula (4):

[0103]

[0104] Similarly, the third coordinate conversion data corresponding to other navigation data is You can get the point cloud data subset after converting other point cloud data subsets to the reference coordinate system

[0105] This embodiment determines the coordinate conversion relationship between the multi-frame navigation data generated at the first moment and the second moment and the benchmark navigation data, respectively, and converts the point cloud data subsets corresponding to each navigation data into the benchmark coordinate system. This achieves the mapping of the point cloud data collected by the point cloud acquisition device within the camera's viewing angle to the same camera shooting angle, thereby improving the accuracy of spatial alignment of the image captured by the camera and the point cloud data collected by the point cloud acquisition device, which in turn helps to improve the accuracy of executing the intelligent driving algorithm using the aligned images and point cloud data.

[0106] In some optional implementations of this embodiment, in step 1063, for each frame of navigation data other than the reference navigation data in the at least two frames of navigation data, a point cloud data subset corresponding to the generation time of the frame of navigation data may be determined according to the following steps:

[0107] The point cloud data collected between the generation time of the frame navigation data and the generation time of the next frame navigation data is determined as the point cloud data subset corresponding to the generation time of the frame navigation data.

[0108] For example, if the at least two frames of navigation data mentioned above include 5 frames of data, namely F0, F1, F2, F3, and F4, then the point cloud data subset collected from the generation time of F0 to the generation time of F1 is the point cloud data subset corresponding to F0, and the point cloud data subset collected from the generation time of F4 to the generation time of the next frame of navigation data is the point cloud data subset corresponding to F4.

[0109] This embodiment realizes the rapid division of the point cloud data set collected within the field of view of the camera by using the point cloud data subset collected between the generation time of two frames of navigation data as the point cloud data subset corresponding to one frame of navigation data, and determines the coordinate conversion data corresponding to each point cloud data subset, thereby improving the efficiency of aligning the point cloud data with the environmental image in spatial perspective.

[0110] In some optional implementations of this embodiment, step 103 includes:

[0111] Based on the exposure time of the camera, the exposure start time of the camera is set so that the camera exposure ends at the third time to obtain the environment image.

[0112] For example, Figure 7As shown, assuming that the exposure time of the camera is Δt, the third moment is t3, and the exposure start time is t3-Δt. After the exposure is completed, the environment image can be captured at the third moment.

[0113] This embodiment achieves precise control of the generation time of the environmental image by setting the exposure start time of the camera, reduces the influence of the camera exposure time on determining the exact shooting time of the environmental image, and helps to further improve the accuracy of the synchronous collection of environmental images and point cloud data in time.

[0114] Figure 8 A schematic diagram of the structure of a sensor data synchronization acquisition and processing system provided in an embodiment of the present application. The sensor data synchronization acquisition and processing system is arranged on a vehicle. The system includes: a visual controller 801, a camera 802, a point cloud acquisition device 803, a gateway 804, a data processing module 805 and a data storage module 806. The above-mentioned gateway 804 is used to provide a data forwarding function between the data processing module 805 and the visual controller 801. The data processing module 805 and the data storage module 806 can be arranged on a separate controller or integrated in other controllers.

[0115] In this embodiment, the visual controller 801 is connected to the camera 802 and the gateway 804 , and is used to control the camera 802 to capture the environmental image around the vehicle, and send the environmental image to the data processing module 805 through the gateway 804 .

[0116] The type and quantity of the visual controller 801 and the number and type of the corresponding cameras 802 can be set according to actual needs. For example, the visual controller 801 may include a front-view camera 802 controller connected to two far-field cameras 802; an electronic rearview mirror controller connected to two left and right electronic rearview mirror cameras 802; and a parking controller connected to four surround wide-angle cameras 802.

[0117] In this embodiment, the point cloud acquisition device 803 is connected to the gateway 804 to acquire a point cloud data set around the vehicle and send the point cloud data set to the data processing module 805 through the gateway 804 .

[0118] The number of the point cloud acquisition device 803 may include one or more. For example, the point cloud acquisition device 803 includes three laser radars installed outside the vehicle.

[0119] The data processing module 805 is used to execute the above-mentioned sensor data synchronous acquisition and processing method, obtain the environment image and point cloud data set with synchronized acquisition time and in the same coordinate system, and store the environment image and point cloud data set in the data storage module 806.

[0120] It should be noted that when the data processing module 805 executes the above-mentioned sensor data synchronous acquisition and processing method, the corresponding camera and point cloud acquisition device can be pre-set, that is, the camera and point cloud acquisition device that need to be synchronized in acquisition time and aligned in spatial perspective are pre-set.

[0121] Optionally, the system may also include a navigation device, which may include an inertial navigation device, a satellite navigation device, etc. The two navigation devices may also be combined to generate navigation data in a combined inertial navigation manner. The navigation data is used to determine the vehicle's posture in real time, and then determine the camera's posture based on the vehicle's posture, and establish a reference coordinate system consistent with the camera's shooting direction.

[0122] The sensor data synchronous acquisition and processing system provided in the embodiment of the present application, by setting a data processing module to execute the above-mentioned sensor data synchronous acquisition and processing method, only needs to obtain the camera's viewing angle range, main optical axis direction, camera posture, external parameters and other characteristic parameters, as well as the scanning speed, external parameters and other characteristic parameters of the point cloud acquisition device, so as to perform real-time synchronous acquisition and coordinate system alignment of the image and point cloud data, thereby improving the efficiency of synchronous acquisition and processing of sensor data, helping to improve the execution efficiency and accuracy of the intelligent driving algorithm, and realizing effective data synchronous acquisition and processing for various vehicles, cameras, and point cloud acquisition devices, and the system has a wider range of applicable scenarios.

[0123] Fig. 9 A schematic diagram of the structure of a sensor data synchronous acquisition and processing device provided in an embodiment of the present application. Specifically comprising: a first determination module 901, used to determine the first view boundary line and the second view boundary line of the camera based on the view range and the main optical axis of the camera on the vehicle; a second determination module 902, used to determine the first moment when the point cloud acquisition device on the vehicle scans the first view boundary line, the second moment when the point cloud acquisition device scans the second view boundary line, and the third moment when the point cloud acquisition device scans the main optical axis; a setting module 903, used to set the trigger shooting moment of the camera based on the third moment, so that the camera captures the environment image around the vehicle at the third moment; a third determination module 904, used to determine the camera posture corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera posture; an acquisition module 905, used to acquire the first external parameter of the camera and the second external parameter of the point cloud acquisition device; a mapping module 906, used to map the point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system based on the first external parameter and the second external parameter.

[0124] In some optional implementations of the present embodiment, the second determination module includes: a first determination unit, which is used to determine the scanning period, scanning starting direction, scanning starting time and scanning angle range of the point cloud acquisition device, wherein the scanning starting direction is the starting direction in which the point cloud acquisition device acquires point cloud data in each scanning period, and the scanning starting time is the time when the point cloud acquisition device scans to the scanning starting direction; a second determination unit, which is used to determine the scanning angular velocity of the point cloud acquisition device based on the scanning period and the scanning angle range; a third determination unit, which is used to determine the first angle between the first viewing angle boundary line and the scanning starting direction, and the second angle between the second viewing angle boundary line and the scanning starting direction; a fourth determination unit, which is used to determine the first time when the point cloud acquisition device scans to the first viewing angle boundary line, and the second time when it scans to the second viewing angle boundary line based on the first angle, the second angle, the scanning angular velocity and the scanning starting time; a fifth determination unit, which is used to determine the third angle between the main optical axis and the first viewing angle boundary line, and determine the third time when the point cloud acquisition device scans to the main optical axis of the camera based on the third angle, the scanning angular velocity and the first time.

[0125] In some optional implementations of this embodiment, the third determination module includes: a first acquisition unit, used to acquire at least two frames of navigation data generated by the navigation device between the first moment and the second moment; a sixth determination unit, used to determine baseline navigation data from the at least two frames of navigation data; and a seventh determination unit, used to determine the camera pose corresponding to the third moment based on the baseline navigation data.

[0126] In some optional implementations of this embodiment, the sixth determination unit is further used to: determine, from the at least two frames of navigation data, navigation data generated at a time adjacent to the third time as reference navigation data.

[0127] In some optional implementations of the present embodiment, the mapping module includes: an eighth determination unit, for determining, based on the first external parameter and the second external parameter, first coordinate conversion data for converting the coordinate system of at least two frames of navigation data into the coordinate system of the camera, and second coordinate conversion data for converting the coordinate system of the camera into the coordinate system of the point cloud acquisition device; a mapping unit, for determining, for each frame of navigation data other than the baseline navigation data in the at least two frames of navigation data, a subset of point cloud data corresponding to the generation moment of the frame navigation data; third coordinate conversion data for determining the conversion relationship between the vehicle posture represented by the frame navigation data and the vehicle posture represented by the baseline navigation data; and based on the first coordinate conversion data, the second coordinate conversion data and the third coordinate conversion data, mapping the subset of point cloud data corresponding to the generation moment of the frame navigation data to the baseline coordinate system.

[0128] In some optional implementations of this embodiment, the mapping unit is further used to: determine the point cloud data collected between the generation time of the frame navigation data and the generation time of the next frame navigation data as the point cloud data subset corresponding to the generation time of the frame navigation data.

[0129] In some optional implementations of this embodiment, the setting module is further used to: set the exposure start time of the camera based on the exposure duration of the camera, so that the camera ends exposure at the third time to obtain an environment image.

[0130] The sensor data synchronous acquisition and processing device provided in this embodiment can be as follows Fig. 9 The sensor data synchronous acquisition and processing device shown in can execute all the steps of the above sensor data synchronous acquisition and processing methods, thereby achieving the technical effects of the above sensor data synchronous acquisition and processing methods. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.

[0131] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Fig.10 The electronic device 1000 shown includes: at least one processor 1001, a memory 1002, at least one network interface 1004 and other user interfaces 1003. The various components in the electronic device 1000 are coupled together via a bus system 1005. It is understood that the bus system 1005 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 1005 is not described in detail. Fig.10 Various buses are labeled as bus system 1005 .

[0132] The user interface 1003 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0133] It can be understood that the memory 1002 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1002 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0134] In some implementations, the memory 1002 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 10021 and an application program 10022 .

[0135] The operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 10022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 10022.

[0136] In this embodiment, by calling the program or instruction stored in the memory 1002, specifically, the program or instruction stored in the application 10022, the processor 1001 is used to execute the method steps provided by each method embodiment, for example, including:

[0137] Based on the viewing angle range and the main optical axis of the camera on the vehicle, determine the first viewing angle boundary line and the second viewing angle boundary line of the camera; determine the first moment when the point cloud acquisition device on the vehicle scans the first viewing angle boundary line, the second moment when the point cloud acquisition device scans the second viewing angle boundary line, and the third moment when the point cloud acquisition device scans the main optical axis; based on the third moment, set the trigger shooting moment of the camera so that the camera captures the environmental image around the vehicle at the third moment; determine the camera posture corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera posture; obtain the first external parameter of the camera and the second external parameter of the point cloud acquisition device; based on the first external parameter and the second external parameter, map the point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system.

[0138] The method disclosed in the above embodiment of the present application can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1001. The above processor 1001 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1002, and the processor 1001 reads the information in the memory 1002 and completes the steps of the above method in combination with its hardware.

[0139] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the above functions of the present application, or a combination thereof.

[0140] For software implementation, the technology described above in this article can be implemented by a unit that performs the functions described above in this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0141] The electronic device provided in this embodiment may be Fig.10 The electronic device shown in can execute all the steps of the above-mentioned method for synchronously collecting and processing sensor data, thereby achieving the technical effects of the above-mentioned method for synchronously collecting and processing sensor data. Please refer to the above related description for details. For the sake of brevity, it will not be repeated here.

[0142] The embodiment of the present application also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0143] When one or more programs in the storage medium can be executed by one or more processors, the sensor data synchronous acquisition and processing method executed on the electronic device side can be implemented.

[0144] The processor is used to execute the program stored in the memory to implement the following steps of the sensor data synchronization collection and processing method performed on the electronic device side:

[0145] Based on the viewing angle range and the main optical axis of the camera on the vehicle, determine the first viewing angle boundary line and the second viewing angle boundary line of the camera; determine the first moment when the point cloud acquisition device on the vehicle scans the first viewing angle boundary line, the second moment when the point cloud acquisition device scans the second viewing angle boundary line, and the third moment when the point cloud acquisition device scans the main optical axis; based on the third moment, set the trigger shooting moment of the camera so that the camera captures the environmental image around the vehicle at the third moment; determine the camera posture corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera posture; obtain the first external parameter of the camera and the second external parameter of the point cloud acquisition device; based on the first external parameter and the second external parameter, map the point cloud data set collected by the point cloud acquisition device between the first moment and the second moment to the reference coordinate system.

[0146] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different circuits to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] The steps of the circuits or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, software modules executed by a processor, or a combination of the two. The software modules may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0148] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The circuit steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0149] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for synchronously collecting and processing sensor data, characterized in that: The method comprises: Determining a first viewing angle boundary line and a second viewing angle boundary line of the camera based on a viewing angle range and a main optical axis of the camera on the vehicle; Determine a first moment when the point cloud acquisition device on the vehicle scans the first viewing angle boundary line, a second moment when the point cloud acquisition device scans the second viewing angle boundary line, and a third moment when the point cloud acquisition device on the vehicle scans the main optical axis; Based on the third moment, setting a trigger shooting moment of the camera so that the camera captures an image of the environment around the vehicle at the third moment; Determine a camera pose corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera pose; Acquire a first external parameter of the camera and a second external parameter of the point cloud acquisition device; Based on the first external parameter and the second external parameter, a point cloud data set collected by the point cloud collection device between the first moment and the second moment is mapped to the reference coordinate system.

2. The method according to claim 1, characterized in that: The determining of a first moment when the point cloud acquisition device on the vehicle scans the first viewing angle boundary line, a second moment when the point cloud acquisition device scans the second viewing angle boundary line, and a third moment when the point cloud acquisition device on the vehicle scans the main optical axis includes: Determine the scanning cycle, scanning starting direction, scanning starting time and scanning angle range of the point cloud acquisition device, wherein the scanning starting direction is the starting direction in which the point cloud acquisition device acquires point cloud data in each scanning cycle, and the scanning starting time is the time when the point cloud acquisition device scans to the scanning starting direction; Determining a scanning angular velocity of the point cloud acquisition device based on the scanning period and the scanning angle range; Determine a first angle between the first viewing angle boundary line and the scanning starting direction, and a second angle between the second viewing angle boundary line and the scanning starting direction; Based on the first angle, the second angle, the scanning angular velocity, and the scanning start time, determining a first time when the point cloud acquisition device scans the first viewing angle boundary line, and a second time when the point cloud acquisition device scans the second viewing angle boundary line; Determine a third angle between the main optical axis and the first viewing angle boundary line, and determine a third moment when the point cloud acquisition device scans the main optical axis of the camera based on the third angle, the scanning angular velocity and the first moment.

3. The method according to claim 1, characterized in that: Determining the camera pose corresponding to the camera at the third moment includes: Acquire at least two frames of navigation data generated by a navigation device between the first moment and the second moment; Determining reference navigation data from the at least two frames of navigation data; Based on the reference navigation data, a camera pose corresponding to the third moment is determined.

4. The method according to claim 3, characterized in that The step of determining the reference navigation data from the at least two frames of navigation data comprises: From the at least two frames of navigation data, navigation data generated at a time point adjacent to the third time point is determined as reference navigation data.

5. The method according to claim 3 or 4, characterized in that: The mapping of the point cloud data set collected by the point cloud collection device between the first moment and the second moment to the reference coordinate system based on the first external parameter and the second external parameter includes: Based on the first external parameter and the second external parameter, determine first coordinate conversion data for converting the coordinate system of the at least two frames of navigation data into the coordinate system of the camera, and second coordinate conversion data for converting the coordinate system of the camera into the coordinate system of the point cloud acquisition device; For each frame of navigation data other than the baseline navigation data in the at least two frames of navigation data, determine a subset of point cloud data corresponding to the moment when the frame navigation data is generated; determine third coordinate transformation data of the transformation relationship between the vehicle posture represented by the frame navigation data and the vehicle posture represented by the baseline navigation data; and map the subset of point cloud data corresponding to the moment when the frame navigation data is generated to the baseline coordinate system based on the first coordinate transformation data, the second coordinate transformation data and the third coordinate transformation data.

6. The method according to claim 5, characterized in that The step of determining a point cloud data subset corresponding to the generation time of the frame navigation data comprises: The point cloud data collected between the generation time of the frame navigation data and the generation time of the next frame navigation data is determined as the point cloud data subset corresponding to the generation time of the frame navigation data.

7. The method according to claim 1, characterized in that The triggering and shooting moment of the camera is set based on the third moment, so that the camera captures the environment image around the vehicle at the third moment, including: Based on the exposure duration of the camera, the exposure start time of the camera is set so that the camera ends exposure at the third time to obtain the environment image.

8. A sensor data synchronous acquisition and processing device, characterized in that: The device comprises: A first determining module, configured to determine a first viewing angle boundary line and a second viewing angle boundary line of the camera based on a viewing angle range and a main optical axis of the camera on the vehicle; A second determination module is used to determine a first moment when the point cloud acquisition device on the vehicle scans the first viewing angle boundary line, a second moment when the point cloud acquisition device scans the second viewing angle boundary line, and a third moment when the point cloud acquisition device on the vehicle scans the main optical axis; A setting module, used for setting a trigger shooting moment of the camera based on the third moment, so that the camera captures an image of the environment around the vehicle at the third moment; A third determination module, used to determine a camera pose corresponding to the camera at the third moment, and establish a reference coordinate system based on the camera pose; An acquisition module, used to acquire a first external parameter of the camera and a second external parameter of the point cloud acquisition device; A mapping module is used to map the point cloud data set collected by the point cloud collection device between the first moment and the second moment into the reference coordinate system based on the first external parameter and the second external parameter.

9. An electronic device, characterized in that: include: Memory for storing computer programs; The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the sensor data synchronous acquisition and processing method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sensor data synchronous acquisition and processing method described in any one of claims 1 to 7 is implemented.

Images