A data processing method, device, computer device, medium and product

Abstract

The present disclosure provides a data processing method and device, computer equipment, medium and product, wherein the method comprises: acquiring position information of each camera arranged around a laser radar, and dividing the each camera into a plurality of camera groups according to the position information; determining a trigger time for triggering the each camera to collect images based on the position information of the each camera and scanning time information of the laser radar; triggering the each camera to collect images according to the trigger time, and performing parallel processing on image data of different camera groups; and performing fusion processing on the processed each image data and point cloud data collected by the laser radar. The embodiment of the present disclosure can ensure that the time of collecting image data by the camera and acquiring point cloud data by the laser radar is synchronized at the same position, and can also perform parallel processing on the image data, thereby improving the processing speed of the image data and reducing the computing pressure and time cost of the system.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, computer equipment, medium, and product. Background Technology

[0002] In the field of data processing technology, sensors such as LiDAR and cameras are crucial components for acquiring data about the surrounding environment. For example, in autonomous vehicles, various types of sensors can be used to acquire environmental data, enabling functions such as vehicle localization, route planning, and decision-making.

[0003] However, conventional cameras do not support synchronization with the LiDAR clock, which leads to a time difference between point cloud data and image data at the same location. This results in a certain deviation in the data fusion between the two, especially when there are many cameras, the more out-of-sync data there is, leading to a large computational burden and time overhead on the system. Summary of the Invention

[0004] This disclosure provides at least one data processing method, apparatus, computer equipment, medium, and product.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method, including:

[0006] Obtain the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information;

[0007] Based on the position information of each camera and the scanning time information of the lidar, the trigger time for each camera to acquire images is determined.

[0008] The cameras are triggered to acquire images according to the trigger time, and the image data of different camera groups are processed in parallel.

[0009] The processed image data is then fused with the point cloud data acquired by the lidar.

[0010] In one optional implementation, the camera is divided into multiple camera groups according to the location information, including:

[0011] Based on the location information, multiple cameras located at the same location are divided into different camera groups.

[0012] In one optional implementation, the trigger time for triggering image acquisition by each camera is determined based on the position information of each camera and the scanning time information of the lidar, including:

[0013] The scanning time at the beginning or end of each scanning cycle of the lidar and the scanning frequency of the lidar are obtained.

[0014] Based on the start or end scanning time of each scanning cycle and the scanning frequency of the lidar, the scanning time of the lidar at each camera position within each scanning cycle is determined.

[0015] Based on the position information of each camera and the scanning time of the LiDAR at each camera position within each scanning cycle, the trigger time for each camera to acquire images is determined.

[0016] In one optional implementation, the trigger time for triggering image acquisition by each camera is determined based on the position information of each camera and the scanning time information of the lidar, including:

[0017] Obtain the scanning time of the lidar at each camera position within each scanning cycle;

[0018] Based on the location information of each camera and the scanning time of the lidar at each camera location, the trigger time for each camera to acquire images is determined.

[0019] In one optional implementation, the trigger time for triggering image acquisition by each camera is determined based on the position information of each camera and the scanning time information of the lidar, including:

[0020] Based on the position information of each camera and the scanning time information of the lidar, the scanning time when the lidar scans to the location of each camera is determined;

[0021] Based on the scanning time when the LiDAR scans to the location of each camera and the trigger response time of the camera, the trigger time for triggering the camera to perform image acquisition is determined; the trigger response time is the time from when the camera is triggered to when it is exposed.

[0022] In one optional implementation, if the scanning time information of the lidar is not obtained, a temporary trigger time for triggering each camera to acquire images is set based on the preset scanning frequency of the lidar and the position information of each camera.

[0023] After obtaining the scanning time information of the LiDAR, and based on the position information of each camera and the obtained scanning time information of the LiDAR, determining the trigger time for triggering each camera to perform image acquisition, the temporary trigger time is replaced with the newly determined trigger time.

[0024] In one optional implementation, the parallel processing of image data from different camera groups includes: transmitting image data from different camera groups to different image processing modules for parallel processing;

[0025] The step of triggering each camera to acquire images according to the trigger time includes:

[0026] The image processing module controls each camera to acquire images according to the determined trigger time for image acquisition by each camera.

[0027] In one optional implementation, the step of fusing the processed image data with the point cloud data acquired by the lidar includes:

[0028] Based on the camera position corresponding to each processed image data and the scanning position corresponding to the point cloud data collected by the LiDAR, each image data is matched with the point cloud data, and the fusion processing is performed based on the matching results.

[0029] Secondly, embodiments of this disclosure also provide a data processing apparatus, comprising:

[0030] The acquisition module is used to acquire the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information;

[0031] The determining module is used to determine the trigger time for triggering each camera to acquire images based on the position information of each camera and the scanning time information of the lidar;

[0032] The processing module is used to trigger each camera to acquire images according to the trigger time, and to process the image data of different camera groups in parallel.

[0033] The fusion module is used to fuse the processed image data with the point cloud data collected by the lidar.

[0034] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0035] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0036] Fifthly, embodiments of this disclosure also provide a computer program product carrying program code, the program code including instructions that can be used to execute the steps of the first aspect above, or any possible implementation of the first aspect.

[0037] The data processing method provided in this disclosure can determine the trigger time for each camera to acquire images based on the position information of each camera and the scanning time information of the LiDAR. This ensures that the image acquisition frequency of the camera is the same as the scanning frequency of the LiDAR, thereby ensuring that the time for the camera to acquire image data and the time for the LiDAR to acquire point cloud data at the same location are synchronized. In other words, it ensures that there is no time difference between the point cloud data and the image data at the same location, thus enabling better alignment of the image data and point cloud data, and facilitating accurate fusion processing of the image data and point cloud data. Furthermore, dividing each camera into multiple camera groups and processing the image data of different camera groups in parallel can improve the processing speed of image data, thereby improving the speed of image data and point cloud data fusion processing. It can be seen that the above method achieves the goal of reducing the computational pressure and time overhead of the processing system.

[0038] Furthermore, by separating multiple cameras at the same location in this embodiment, it is possible to ensure that image data acquired by multiple cameras at the same location is processed simultaneously, which can further improve the image data processing speed.

[0039] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0041] Figure 1A flowchart of a data processing method provided by an embodiment of this disclosure is shown;

[0042] Figure 2 A schematic diagram of the camera arrangement provided in an embodiment of this disclosure is shown;

[0043] Figure 3 This diagram illustrates the relationship between a camera and an image processing module provided in an embodiment of this disclosure.

[0044] Figure 4 A flowchart of another data processing method provided by an embodiment of this disclosure is shown;

[0045] Figure 5 A schematic diagram of a data processing apparatus provided in an embodiment of this disclosure is shown;

[0046] Figure 6 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0048] Typically, LiDAR scanners operate at a scanning frequency of 10 Hz, completing one scan in 100 milliseconds. Cameras, on the other hand, typically acquire images at 30 Hz. When the LiDAR scans to the location of a camera, the camera, operating at its inherent frequency, cannot acquire data simultaneously with the LiDAR. This results in a time lag between point cloud data and image data at the same location, leading to discrepancies in data fusion. Furthermore, because the camera's image acquisition frequency is higher than the LiDAR's scanning frequency, especially with a large number of cameras, the amount of asynchronous data increases, resulting in significant computational burden and time overhead for the system.

[0049] Based on this, this disclosure provides a data processing method, apparatus, computer equipment, medium, and product. It can determine the trigger time for each camera to acquire images based on the position information of each camera and the scanning time information of the LiDAR, ensuring that the image acquisition frequency of the camera is the same as the scanning frequency of the LiDAR. This ensures that the time for the camera to acquire image data and the time for the LiDAR to acquire point cloud data are synchronized at the same location, meaning there is no time difference between the point cloud data and image data at the same location. This allows for better alignment of image data and point cloud data, facilitating accurate fusion processing of image data and point cloud data. Furthermore, dividing each camera into multiple camera groups and processing the image data from different camera groups in parallel improves the processing speed of image data, thereby increasing the speed of fusion processing of image data and point cloud data. Therefore, the above method achieves the goal of reducing the computational pressure and time overhead of the processing system.

[0050] The deficiencies of the above solutions and the proposed solutions are the result of the inventor's practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0052] The data processing methods, apparatus, computer equipment, media, and products provided in this disclosure can be applied to any scenario that may require synchronization of LiDAR and cameras, such as the operation of autonomous vehicles. The following description uses an autonomous vehicle equipped with LiDAR and a camera as an example to illustrate the data processing methods, apparatus, computer equipment, media, and products provided in this disclosure.

[0053] To facilitate understanding of this embodiment, a data processing method disclosed in this disclosure will first be described in detail. The data processing method provided in this disclosure is generally executed by a computer device with certain computing capabilities.

[0054] The following describes the data processing method provided in this disclosure embodiment, taking the on-board processor in an autonomous vehicle as an example.

[0055] See Figure 1 The diagram shown is a flowchart of a data processing method provided in an embodiment of this disclosure. The method includes steps S101 to S104, wherein:

[0056] S101: Obtain the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information.

[0057] In one embodiment of this disclosure, the onboard processor may pre-store the location information of each camera. Before the autonomous vehicle begins operation, the onboard processor can retrieve the stored location information of each camera from the memory. In another embodiment, the onboard processor may not store the camera location information; in this case, the onboard processor can obtain the location information of each camera from the backend server and store it locally. In yet another embodiment, the onboard processor can also obtain the changed camera location information from the backend server when it becomes aware that the locally stored camera location information has changed (the onboard processor can become aware of this by receiving a notification message from the backend server indicating a change in camera location, etc., which is not specifically limited here). Here, the method by which the onboard processor obtains the camera location information is not specifically limited.

[0058] The position information of each camera can be the azimuth information of each camera relative to the LiDAR, or the coordinate information of each camera, etc. In this embodiment of the disclosure, the position information of each camera can, exemplarily, refer to the azimuth information of each camera relative to the LiDAR. To acquire image data from different azimuths, exemplarily, each camera can be arranged in various directions around the LiDAR, centered on it. The number and type of cameras arranged in each direction can be different.

[0059] To acquire image data at different distances from the same location, one approach is to simultaneously mount different types of cameras at the same location; that is, multiple cameras can correspond to the same location information. Here, different types of cameras can refer to cameras with different depths of field. Figure 2 The diagram illustrates a camera arrangement where, exemplarily, multiple cameras are arranged around a LiDAR sensor, positioned at eight points in each direction. This arrangement includes three types of cameras: camera A, camera B, and camera C. Specifically, camera A has a greater depth of field than camera B, and camera B has a greater depth of field than camera C. Figure 2 As can be seen, two types of cameras are mounted at certain locations. It should be noted that cameras A4 and B4, as well as cameras A2 and B1, are shown separately to distinguish them. In practice, the positional information of the two types of cameras at any of the four locations mentioned above is identical. Furthermore, Figure 2 The rotation direction of the lidar shown is only an example direction. In a specific embodiment, the lidar can rotate in a clockwise direction or a counterclockwise direction.

[0060] Here, the cameras can be divided into multiple camera groups. During subsequent image data processing, image data from different camera groups can be processed in parallel, thereby improving processing efficiency. In one approach, multiple cameras located at the same position can be divided into different camera groups based on their location information. This allows for parallel processing of image data acquired by cameras at the same location, further improving processing efficiency for image data at that location.

[0061] In this embodiment of the disclosure, the number of cameras in each camera group is not limited. For example, in order to balance the computing pressure of the data processing modules corresponding to each camera group, the number of cameras in each camera group can be the same.

[0062] S102: Based on the position information of each camera and the scanning time information of the lidar, determine the trigger time for each camera to perform image acquisition.

[0063] In this embodiment of the disclosure, the lidar can rotate at a constant speed according to a preset scanning frequency. For example, the lidar scanning frequency is 10 Hz, meaning it rotates once every 100 milliseconds. One complete scan cycle can be considered as the lidar completing one scanning period. The lidar can acquire point cloud data in real time during its rotation.

[0064] Here, the LiDAR scanning time information can include the scanning time at the beginning or end of each scanning cycle, and the scanning time of the LiDAR at each camera position within each scanning cycle. The scanning time at the beginning or end of each scanning cycle, or the scanning time of the LiDAR at each camera position within each scanning cycle, can be broadcast by the LiDAR.

[0065] To ensure that the point cloud data scanned by the LiDAR and the images acquired by the camera are synchronized in time at the same location, the trigger time for each camera to acquire images can be determined based on the scanning time information of the LiDAR during implementation.

[0066] When the scanning time broadcast by the LiDAR varies, the process for determining the trigger time for each camera to acquire images can differ. Specifically, when the LiDAR broadcasts the scanning time at the beginning or end of each scanning cycle, one approach is to acquire the broadcast scanning time and the LiDAR's scanning frequency at the beginning or end of each scanning cycle; then, based on the broadcast scanning time and the LiDAR's scanning frequency, determine the LiDAR's scanning time at each camera position within each scanning cycle; and finally, based on the position information of each camera and the LiDAR's scanning time at each camera position within each scanning cycle, determine the trigger time for each camera to acquire images.

[0067] Specifically, taking the scanning time at the start of each scanning cycle of the LiDAR as an example, the scanning time of the LiDAR in each direction within one scan cycle can be determined based on the start scanning time and the LiDAR's scanning frequency. Therefore, based on the position information of each camera, the scanning time of the LiDAR at each camera position can be determined. Next, based on the scanning time of the LiDAR at each camera position, the trigger time for image acquisition by each camera can be determined. For example, before the scan begins, the rotation angle of the LiDAR is recorded as 0 degrees, and the scanning time at the start of the first scanning cycle is recorded as 0 milliseconds. Assuming the LiDAR's scanning frequency is 10 Hz (i.e., one scanning cycle is 100 milliseconds), then the scanning time of the LiDAR at a rotation angle of 90 degrees is determined to be 25 milliseconds, the scanning time at a rotation angle of 180 degrees is 50 milliseconds, and so on. Therefore, knowing the position information of each camera, the scanning time of the LiDAR at each camera position can be determined.

[0068] In another approach, when the LiDAR broadcasts the scanning time at each camera location, the scanning time of the LiDAR at each camera location within each scanning cycle can be directly obtained; then, based on the location information of each camera and the scanning time of the LiDAR at each camera location, the trigger time for triggering each camera to perform image acquisition can be determined.

[0069] Here, the scanning time of the LiDAR at each camera position within each scanning cycle can be directly broadcast by the LiDAR. Therefore, the trigger time for each camera to acquire images can be determined directly based on the scanning time of the LiDAR at each camera position within each scanning cycle. The process of determining the trigger time can refer to the aforementioned embodiment, and will not be repeated here.

[0070] Considering that camera triggering requires a certain amount of time, in order to further ensure that the point cloud data scanned by the LiDAR and the image acquired by the camera are synchronized in time at the same location, in one implementation, the scanning time when the LiDAR scans to the location of each camera can be determined based on the location information of each camera and the scanning time information of the LiDAR; then, based on the scanning time when the LiDAR scans to the location of each camera and the trigger response time of the camera, the trigger time for triggering the camera to acquire images can be determined; wherein the trigger response time is the time from when the camera is triggered to when it is exposed.

[0071] The process of determining the scanning time when the lidar scans the location of each camera can be referred to the aforementioned implementation method, and will not be repeated here.

[0072] Here, the camera's trigger response time refers to the time interval between when the camera is triggered and when it exposes itself. The camera's trigger response time can be related to the camera's own properties, so the trigger response time for each camera can be obtained from the camera's property information.

[0073] In practical implementation, the scanning time when the LiDAR scans to the location of each camera can be defined as T1, and the trigger response time of the camera can be defined as T2. Then the corresponding trigger time of the camera is T1-T2.

[0074] Considering that in practical implementation, the camera and LiDAR may not start working simultaneously (e.g., the camera starts working first, and the LiDAR starts working later), or the LiDAR may stop working midway, one implementation method allows for setting a temporary trigger time for each camera to acquire images, based on a preset LiDAR scanning frequency and the position information of each camera, when the LiDAR scanning time information is unavailable. Then, each camera is triggered to acquire images according to the temporary trigger time. This process ensures that the frequency of image data acquisition by each camera is consistent with the LiDAR scanning frequency.

[0075] Once the LiDAR is functioning correctly, meaning its scanning time information can be obtained, and based on the position information of each camera and the acquired LiDAR scanning time information, the trigger time for each camera to acquire images can be determined. Then, the temporary trigger time can be replaced with this newly determined trigger time. Each camera is then triggered to acquire images according to this newly determined trigger time. By updating the trigger times of each camera, it can be ensured that the trigger time for image acquisition by each camera is consistent with the time when the LiDAR scans to that camera location. This guarantees that the point cloud data scanned by the LiDAR and the images acquired by the cameras are synchronized in time at the same location.

[0076] S103: Trigger each camera to acquire images according to the trigger time, and process the image data of different camera groups in parallel.

[0077] After determining the trigger time for each camera to acquire images, each camera can be triggered separately according to its trigger time. For multiple cameras located at the same position, the trigger time can be the same. In this embodiment, the image processing module can control each camera to acquire images according to its trigger time. Specifically, the image processing module can generate a trigger pulse signal based on the trigger time and then send the trigger pulse signal to each camera through a signal transmission interface. To achieve parallel processing of image data from different camera groups, multiple image processing modules can be set up. Image data from different camera groups are transmitted to different image processing modules for parallel processing. It should be noted that in the parallel processing method described in this disclosure, multiple cameras are assigned to different image processing modules in the form of camera groups. The cameras in different camera groups do not have a temporal order. Once a camera acquires image data, its corresponding image processing module can process that camera's image data. The processing order for that camera does not depend on the processing results of other cameras, unlike the serial processing method where all camera image data are processed sequentially in time.

[0078] As mentioned earlier, based on the location information of each camera, multiple cameras located at the same position can be divided into different camera groups, and the positions of any two cameras in the camera group corresponding to each image processing module will be different. Figure 3 The relationship between a camera and an image processing module is shown in the diagram. Figure 3 The positions of the various cameras can be referenced. Figure 2 The diagram shows the arrangement of the cameras, where cameras A1 and C1 are in the same orientation; cameras A2 and B1 are in the same orientation; cameras A3 and C3 are in the same orientation; and cameras A4 and B4 are in the same orientation. Figure 3 As can be seen, the two cameras located in the same direction are divided into two camera groups. That is, the cameras located in the same direction are connected to different image processing modules. Specifically, camera A1 in the same direction is connected to image processing module 1, and camera A2 in the same direction is connected to image processing module 1, and camera B1 in the same direction is connected to image processing module 2; camera A4 in the same direction is connected to image processing module 1, and camera C3 in the same direction is connected to image processing module 2; camera A3 in the same direction is connected to image processing module 1, and camera B4 in the same direction is connected to image processing module 2.

[0079] In other embodiments, multiple image processing units may be used to control the camera triggering unit to send trigger signals to each camera based on the scanning time information of the LiDAR. For example, each image processing unit may control the camera triggering unit to send trigger signals to each camera in the corresponding camera group. However, in order to ensure that multiple cameras at the same location can receive the trigger pulse signal in a timely and accurate manner, any image processing module can control each camera to perform image acquisition according to the determined trigger time for image acquisition by each camera, thus avoiding the situation where multiple image processing modules trigger multiple cameras at the same location, causing disorder of the trigger pulse signal.

[0080] In specific implementation, for example, the image processing module can perform processing operations such as adjustment, format conversion, and encoding on the image data of each camera, without being specifically limited here.

[0081] S104: The processed image data is fused with the point cloud data collected by the lidar.

[0082] In practical implementation, image data and point cloud data at the same location can be fused to obtain fused data from different sensors at the same location. When image data at the same location comes from multiple cameras, image data from multiple cameras at the same location can be fused with point cloud data at that location. In one implementation, the image data and point cloud data can be matched based on the camera positions corresponding to each processed image data and the scanning positions corresponding to the point cloud data acquired by the LiDAR, and fusion processing can be performed based on the matching results.

[0083] The processed image data can contain the position information of each camera, and the point cloud data can contain the scanning positions corresponding to the point cloud data collected by LiDAR. Based on the position information of each camera and the corresponding scanning positions of the LiDAR, the point cloud data at each camera position can be determined. Then, the matched point cloud data at each camera position and the image data are fused together. Based on the fused data, localization, route planning, and decision-making for autonomous vehicles can be achieved, ensuring the normal operation of autonomous vehicles.

[0084] Figure 4 The flowchart illustrates another data synchronization method provided in this embodiment, wherein multiple cameras are arranged around a LiDAR, with different types and numbers of cameras positioned at each location. The LiDAR rotates clockwise and sends the acquired point cloud data to an onboard processor. The LiDAR can also send the scanning time at each camera location within each scanning cycle to an image processing module 1.

[0085] Image processing module 1 determines the trigger time for each camera based on the position information of each camera, the scanning time information of the LiDAR at each camera position within each scanning cycle, and the scanning frequency of the LiDAR, and generates trigger pulse signals to trigger each camera according to the trigger time. Image processing module 1 controls each camera to perform image acquisition through trigger pulse signals.

[0086] The cameras are divided according to their location information. Multiple cameras located at the same location are divided into two camera groups. Image processing module 1 can acquire the raw image data collected by one camera group, and image processing module 2 can acquire the raw image data collected by the other camera group.

[0087] Image processing module 1 and image processing module 2 respectively perform operations such as adjustment, format conversion, and encoding on the raw image data in their respective camera groups to obtain processed image data, and then send the processed image data to the vehicle-mounted processor.

[0088] Finally, the onboard processor fuses the image data and point cloud data according to the camera position corresponding to each processed image data and the scanning position corresponding to the point cloud data collected by the LiDAR.

[0089] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0090] Based on the same inventive concept, this disclosure also provides a data processing device corresponding to the data processing method. Since the principle of the device in this disclosure for solving the problem is similar to that of the data processing method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0091] Reference Figure 5 The diagram shown is an architectural schematic of a data processing device provided in an embodiment of this disclosure. The device includes: an acquisition module 501, a determination module 502, a processing module 503, and a fusion module 504; wherein,

[0092] The acquisition module 501 is used to acquire the position information of each camera arranged around the lidar, and divide the camera into multiple camera groups according to the position information.

[0093] The determining module 502 is used to determine the trigger time for triggering each camera to perform image acquisition based on the position information of each camera and the scanning time information of the lidar.

[0094] Processing module 503 is used to trigger each camera to acquire images according to the trigger time, and to process the image data of different camera groups in parallel;

[0095] The fusion module 504 is used to fuse the processed image data with the point cloud data collected by the lidar.

[0096] In one optional implementation, the acquisition module 501 is specifically used for:

[0097] Based on the location information, multiple cameras located at the same location are divided into different camera groups.

[0098] In one optional implementation, the determining module 502 is specifically used for:

[0099] The scanning time at the beginning or end of each scanning cycle of the lidar and the scanning frequency of the lidar are obtained.

[0100] Based on the start or end scanning time of each scanning cycle and the scanning frequency of the lidar, the scanning time of the lidar at each camera position within each scanning cycle is determined.

[0101] Based on the position information of each camera and the scanning time of the LiDAR at each camera position within each scanning cycle, the trigger time for each camera to acquire images is determined.

[0102] In one optional implementation, the determining module 502 is specifically used for:

[0103] Obtain the scanning time of the lidar at each camera position within each scanning cycle;

[0104] Based on the location information of each camera and the scanning time of the lidar at each camera location, the trigger time for each camera to acquire images is determined.

[0105] In one optional implementation, the determining module 502 is specifically used for:

[0106] Based on the position information of each camera and the scanning time information of the lidar, the scanning time when the lidar scans to the location of each camera is determined;

[0107] Based on the scanning time when the LiDAR scans to the location of each camera and the trigger response time of the camera, the trigger time for triggering the camera to perform image acquisition is determined; the trigger response time is the time from when the camera is triggered to when it is exposed.

[0108] In one optional embodiment, the apparatus further includes:

[0109] The setting module is used to set a temporary trigger time for each camera to perform image acquisition based on the preset scanning frequency of the lidar and the position information of each camera when the scanning time information of the lidar is not obtained.

[0110] The replacement module is used to replace the temporary trigger time with the newly determined trigger time after obtaining the scanning time information of the lidar and determining the trigger time for triggering image acquisition by each camera based on the position information of each camera and the obtained scanning time information of the lidar.

[0111] In one optional implementation, the processing module 503 is specifically used to: transmit image data from different camera groups to different image processing modules for parallel processing;

[0112] The image processing module controls each camera to acquire images according to the determined trigger time for image acquisition by each camera.

[0113] In one optional implementation, the fusion module 504 is specifically used for:

[0114] Based on the camera position corresponding to each processed image data and the scanning position corresponding to the point cloud data collected by the LiDAR, each image data is matched with the point cloud data, and the fusion processing is performed based on the matching results.

[0115] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0116] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 6 The diagram shows the structure of a computer device 600 provided in this embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 stores execution instructions and includes main memory 6021 and external memory 6022. The main memory 6021, also called internal memory, is used to temporarily store computational data in the processor 601 and data exchanged with external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the main memory 6021. When the computer device 600 is running, the processor 601 and the memory 602 communicate through the bus 603, causing the processor 601 to execute the following instructions:

[0117] Obtain the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information;

[0118] Based on the position information of each camera and the scanning time information of the lidar, the trigger time for each camera to acquire images is determined.

[0119] The cameras are triggered to acquire images according to the trigger time, and the image data of different camera groups are processed in parallel.

[0120] The processed image data is then fused with the point cloud data acquired by the lidar.

[0121] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0122] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the X data processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0123] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, include: Obtain the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information; Based on the position information of each camera and the scanning time information of the lidar, the trigger time for each camera to acquire images is determined. The cameras are triggered to acquire images according to the trigger time, and the image data from different camera groups are processed in parallel. The processed image data is then fused with the point cloud data acquired by the lidar. Based on the position information of each camera and the scanning time information of the LiDAR, the trigger time for each camera to acquire images is determined, including: The scanning time at the beginning or end of each scanning cycle of the lidar is obtained, as well as the scanning frequency of the lidar, and the lidar rotates at a constant speed according to the scanning frequency; Based on the start or end scanning time of each scanning cycle and the scanning frequency of the lidar, the scanning time of the lidar at each camera position within each scanning cycle is determined. Based on the position information of each camera and the scanning time of the LiDAR at each camera position within each scanning cycle, the trigger time for each camera to acquire images is determined.

2. The method according to claim 1, characterized in that, Based on the location information, the camera is divided into multiple camera groups, including: Based on the location information, multiple cameras located at the same location are divided into different camera groups.

3. The method according to claim 1, characterized in that, Based on the position information of each camera and the scanning time information of the LiDAR, the trigger time for each camera to acquire images is determined, including: Obtain the scanning time of the lidar at each camera position within each scanning cycle; Based on the location information of each camera and the scanning time of the lidar at each camera location, the trigger time for each camera to acquire images is determined.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the position information of each camera and the scanning time information of the LiDAR, the trigger time for each camera to acquire images is determined, including: Based on the position information of each camera and the scanning time information of the lidar, the scanning time when the lidar scans to the location of each camera is determined; Based on the scanning time when the LiDAR scans to the location of each camera and the trigger response time of the camera, the trigger time for triggering the camera to perform image acquisition is determined; the trigger response time is the time from when the camera is triggered to when it is exposed.

5. The method according to claim 1, characterized in that, The method further includes: In the absence of the scanning time information of the lidar, a temporary trigger time is set to trigger the image acquisition of each camera based on the preset scanning frequency of the lidar and the position information of each camera. After obtaining the scanning time information of the LiDAR, and based on the position information of each camera and the obtained scanning time information of the LiDAR, determining the trigger time for triggering each camera to perform image acquisition, the temporary trigger time is replaced with the newly determined trigger time.

6. The method according to claim 1, characterized in that, The parallel processing of image data from different camera groups includes: transmitting image data from different camera groups to different image processing modules for parallel processing; The step of triggering each camera to acquire images according to the trigger time includes: The image processing module controls each camera to acquire images according to the determined trigger time for image acquisition by each camera.

7. The method according to claim 1, characterized in that, The process of fusing the processed image data with the point cloud data acquired by the lidar includes: Based on the camera position corresponding to each processed image data and the scanning position corresponding to the point cloud data collected by the LiDAR, each image data is matched with the point cloud data, and the fusion processing is performed based on the matching results.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the position information of each camera arranged around the lidar, and divide the cameras into multiple camera groups according to the position information; The determining module is used to determine the trigger time for triggering each camera to acquire images based on the position information of each camera and the scanning time information of the lidar; The processing module is used to trigger each camera to acquire images according to the trigger time, and to process the image data of different camera groups in parallel. The fusion module is used to fuse the processed image data with the point cloud data collected by the lidar. The determining module is specifically used for: The scanning time at the beginning or end of each scanning cycle of the lidar is obtained, as well as the scanning frequency of the lidar, and the lidar rotates at a constant speed according to the scanning frequency; Based on the start or end scanning time of each scanning cycle and the scanning frequency of the lidar, the scanning time of the lidar at each camera position within each scanning cycle is determined. Based on the position information of each camera and the scanning time of the LiDAR at each camera position within each scanning cycle, the trigger time for each camera to acquire images is determined.

9. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and the processor communicates with the memory via the bus when the computer device is running, and the machine-readable instructions, when executed by the processor, perform the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer product carries program code, the program code including instructions that can be used to perform the steps of the data processing method as described in any one of claims 1 to 7.

Images