Automatic driving-oriented 4D annotation data processing method and system

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to an automatic driving-oriented 4D annotation data processing method and system. A 4D annotation data processing method oriented to automatic driving comprises the steps that a WebAssessment module is loaded at a browser end, multiple frames of point cloud data are fused through the browser end, and the method comprises the steps that the multiple frames of point cloud data to be fused are obtained; taking a first frame of point cloud data in multiple frames of point cloud data to be fused as a reference frame, and converting pose information of subsequent frames of point cloud data into a spatial transformation matrix according to a time sequence to obtain a target spatial transformation matrix; according to the target space transformation matrix, calculating frame point cloud coordinates except the first frame point cloud data to obtain a multi-frame point cloud data coordinate set; and according to the multi-frame point cloud data coordinate set, creating a rendering space data structure, and obtaining automatic driving oriented 4D annotation data. The real-time performance of point cloud data fusion can be remarkably improved through real-time processing of a browser end.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and in particular, relates to a method and system for processing 4D annotation data for autonomous driving. Background Art

[0002] The method of centralized processing of multi-frame point cloud data in the cloud has significant defects in practical applications, especially in application scenarios such as autonomous driving and security monitoring that require real-time updates.

[0003] First, due to its high computing requirements, cloud computing resources often reach bottlenecks when faced with sudden large-scale data processing, resulting in limited processing capabilities. Second, this method is difficult to meet the needs of real-time processing and display, which is crucial for application scenarios that require rapid response, such as autonomous driving. Furthermore, existing technologies lack flexibility when point cloud data is abnormal, and are unable to process data in a specific range in a targeted manner, reducing the accuracy and efficiency of data processing. Finally, traditional point cloud data processing and visualization tools are complex to operate and have poor user experience, which is not conducive to users' efficient data operations and analysis.

[0004] These problems highlight the shortcomings of existing technologies in processing large-scale, real-time, and dynamic point cloud data. Technological innovation is urgently needed to improve the efficiency, real-time performance, and flexibility of data processing and improve user experience. Summary of the invention

[0005] Based on this, it is necessary to provide a 4D annotation data processing method and system for autonomous driving to solve the above technical problems.

[0006] In a first aspect, the present application provides a method for processing 4D annotation data for autonomous driving, characterized in that a WebAssembly module is loaded on a browser side, and multi-frame point cloud data is fused using the browser side, the method comprising:

[0007] Obtain multi-frame point cloud data to be fused;

[0008] The first frame of point cloud data in the multiple frames of point cloud data to be fused is used as a reference frame, and the position and posture information of the subsequent frames of point cloud data are converted into a spatial transformation matrix in time sequence to obtain a target spatial transformation matrix;

[0009] According to the target space transformation matrix, the frame point cloud coordinates other than the first frame point cloud data are calculated to obtain a multi-frame point cloud data coordinate set;

[0010] A rendering space data structure is created based on the multi-frame point cloud data coordinate set to obtain 4D annotation data for autonomous driving.

[0011] In some implementable ways, the step of obtaining multiple frames of point cloud data to be fused includes:

[0012] Obtain multiple frames of initial point cloud data to be fused from a server or local storage through a browser;

[0013] Use JavaScript parsing to perform format conversion on the initial point cloud data to obtain point cloud data in a binary data structure suitable for WebAssembly processing.

[0014] In some implementable ways, the step of using the first frame of point cloud data among the multiple frames of point cloud data to be fused as a reference frame, and converting the pose information of subsequent frames of point cloud data into a spatial transformation matrix in chronological order to obtain a target spatial transformation matrix includes:

[0015] Obtain the first frame of point cloud data to obtain reference frame point cloud data;

[0016] In chronological order, use the reference frame point cloud data as a reference for the subsequent frames of point cloud data to be fused, and obtain their pose information relative to the reference frame point cloud data, where the pose information of the subsequent frames of point cloud data includes rotation and translation parameters relative to the reference frame point cloud data;

[0017] According to the pose information of each frame of point cloud data in the subsequent frames of point cloud data, obtain multiple frames of transformation matrices to be fused, where the multiple frames of transformation matrices to be fused indicate that the point cloud data of each frame in the subsequent frames of point cloud data forms a transformation matrix, which are combined into multiple frames of transformation matrices;

[0018] Perform calculations on the multiple frames of transformation matrices to be fused to obtain a target spatial transformation matrix.

[0019] In some implementable ways, the step of performing calculations on the multiple frames of transformation matrices to be fused to obtain a target spatial transformation matrix includes:

[0020] Perform chain multiplication calculations on the multiple frames of transformation matrices to be fused in chronological order to obtain a target spatial transformation matrix.

[0021] In some implementable ways, the step of calculating the frame point cloud coordinates of frames other than the first frame of point cloud data based on the target spatial transformation matrix to obtain a multi-frame point cloud data coordinate set includes:

[0022] Convert the frame point cloud coordinates of each frame of point cloud data in the target spatial transformation matrix into the coordinate system where the reference frame is located to obtain multi-frame point cloud data coordinates;

[0023] Integrate the coordinates of the multi-frame point cloud data to obtain a multi-frame point cloud data coordinate set.

[0024] In some implementable ways, the step of uploading the multi-frame point cloud data coordinate set to the WebGL to obtain 4D annotation data for autonomous driving includes:

[0025] Create a 3D rendering context using WebGL and initialize the shader program;

[0026] Load the multi-frame point cloud data coordinate set into the GPU buffer and set the corresponding vertex attribute pointers;

[0027] Use the WebGL drawing command to render the point cloud data to obtain a real-time 3D display of 4D annotation data for autonomous driving.

[0028] In some implementable ways, it is characterized in that the point cloud data format is one of PCD and PLY.

[0029] In a second aspect, the present application provides a 4D annotation data processing system for autonomous driving, which is applied to the foregoing method, loads a WebAssembly module on the browser side, and uses the browser side to fuse multi-frame point cloud data. The system includes:

[0030] An acquisition unit for acquiring multi-frame point cloud data to be fused;

[0031] A processing unit for using the first-frame point cloud data in the multi-frame point cloud data to be fused as a reference frame, and converting the pose information of the subsequent frame point cloud data into a spatial transformation matrix in chronological order to obtain a target spatial transformation matrix;

[0032] The processing unit is further configured to calculate the coordinates of the frame point cloud except the first-frame point cloud data according to the target spatial transformation matrix to obtain a multi-frame point cloud data coordinate set;

[0033] A result unit for creating a rendering space data structure according to the multi-frame point cloud data coordinate set to obtain 4D annotation data for autonomous driving.

[0034] In a third aspect, the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and it is characterized in that when the processor executes the computer program, the steps of the foregoing method are implemented.

[0035] In a fourth aspect, the present application provides a computer program, which is characterized in that when the computer program is executed by a processor, the steps of the foregoing method are implemented.

[0036] Beneficial effects: The present application provides a method for processing 4D annotation data for autonomous driving, characterized in that a WebAssembly module is loaded on the browser side, and the browser side is used to fuse multi-frame point cloud data. The method includes: obtaining multi-frame point cloud data to be fused; using the first frame of point cloud data in the multi-frame point cloud data to be fused as a reference frame, and in chronological order, converting the pose information of subsequent frames of point cloud data into a spatial transformation matrix to obtain a target spatial transformation matrix; according to the target spatial transformation matrix, calculating the coordinates of the point cloud for frames other than the first frame of point cloud data to obtain a multi-frame point cloud data coordinate set; according to the multi-frame point cloud data coordinate set, creating a rendering space data structure to obtain 4D annotation data for autonomous driving. The above method can significantly improve the real-time performance of point cloud data fusion through real-time processing on the browser side, is applicable to application scenarios that require real-time updates such as autonomous driving and security monitoring, and uses the computing power of the terminal to reduce the computing power requirements of users for cloud devices, and can be run on ordinary computing devices. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0038] Figure 1 It is a flowchart of a method for processing 4D annotation data for autonomous driving in an embodiment;

[0039] Figure 2 It is a logic diagram of a method for processing 4D annotation data for autonomous driving in an embodiment. Detailed Embodiments

[0040] To facilitate the understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings. Embodiments of the present application are given in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used in the description of the present application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0042] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0043] The following explains some terms involved in this application for better understanding:

[0044] WebAssembly (Wasm) is a low-level virtual machine technology that can run in a browser and allows code to be executed in the browser with near-native performance.

[0045] WebGL (Web Graphics Library) is a technology for rendering 3D graphics in a browser without relying on plugins.

[0046] This application provides a method for processing 4D annotation data for autonomous driving. A WebAssembly module is loaded on the browser side, and the browser side is used to fuse multiple frames of point cloud data. Exemplarily, the WebAssembly module can be written in a computer language such as C / C++. Next, the WebAssembly module is loaded onto the browser side for execution on the browser side. In this way, the browser side is used to perform fusion processing on the point cloud data, which may include pose transformation, coordinate calculation, etc. Finally, the point cloud data is rendered onto a web page to provide an interactive 3D view.

[0047] As Figure 1 and Figure 2 shown, a method for processing 4D annotation data for autonomous driving includes:

[0048] S100, obtaining multiple frames of point cloud data to be fused.

[0049] Specifically, multiple frames of point cloud data can be collected through sensors (such as lidar, cameras, etc.) of an autonomous vehicle. These data contain three-dimensional spatial information of the vehicle's surrounding environment for subsequent steps of configuring the multiple frames of point cloud data.

[0050] S200, using the first frame of point cloud data in the multiple frames of point cloud data to be fused as a reference frame, and sequentially converting the pose information of the subsequent frames of point cloud data into a spatial transformation matrix to obtain a target spatial transformation matrix.

[0051] Among them, the first frame in the multiple frames of point cloud data is used as a reference frame, and then the pose information of each frame of point cloud data relative to the reference frame is converted into a spatial transformation matrix, and these matrices are used for subsequent coordinate transformation.

[0052] Specifically, obtaining the target spatial transformation matrix may include the following steps:

[0053] S101. Obtain the first frame of point cloud data to get the reference frame point cloud data.

[0054] Specifically, from the multiple frames of point cloud data to be fused, select the first frame of point cloud data as the reference frame point cloud data in chronological order, that is, use the first frame of point cloud data as the reference frame for alignment and fusion of subsequent frames. Exemplarily, there are four frames of point cloud data from the sensors of an autonomous vehicle at different time points. The first frame of point cloud data (Frame 1) contains the point cloud information of the vehicle's surrounding environment and is set as the reference frame.

[0055] It should be noted that the point cloud data format can be PCD, PLY or other supported formats.

[0056] S102. In chronological order, take the subsequent frames of point cloud data to be fused, and obtain their pose information relative to the reference frame point cloud data with the reference frame point cloud data as the reference, to get the pose information of the subsequent frames of point cloud data.

[0057] Among them, the pose information of the subsequent frames of point cloud data includes the rotation and translation parameters relative to the reference frame point cloud data.

[0058] Specifically, process each subsequent frame of point cloud data in chronological order and calculate their pose information relative to the reference frame point cloud data, including the parameters of rotation and translation.

[0059] S103. According to the pose information of each frame of point cloud data in the subsequent frames of point cloud data, obtain the multiple transformation matrices to be fused, where the multiple transformation matrices to be fused mean that each frame of point cloud data in the subsequent frames of point cloud data forms a transformation matrix, which are combined into multiple transformation matrices.

[0060] Specifically, for each frame of point cloud data in the subsequent frames of point cloud data, extract its pose information relative to the reference frame point cloud data, which may include operations such as rotation, translation and / or scaling parameters, etc. For example, use rotation parameters (such as Euler angles, quaternions or rotation matrices) to construct the rotation matrix of each frame of point cloud data. Use translation parameters to construct the translation vector of each frame of point cloud data. Next, combine the rotation matrix and the translation vector to construct the spatial transformation matrix of each frame of point cloud data. The spatial transformation matrix can transform the point cloud data from its original coordinate system to the coordinate system of the reference frame.

[0061] It should be noted that each frame of point cloud data forms a transformation matrix, so that multiple frames of point cloud data will be combined into multiple transformation matrices for changing the multiple transformation matrices into spatial transformation matrices in subsequent steps.

[0062] S104. Calculate the multi-frame transformation matrices to be fused to obtain the target space transformation matrix.

[0063] Specifically, according to the time sequence, perform chain multiplication on the multi-frame transformation matrices to be fused to obtain the target space transformation matrix.

[0064] Exemplarily, starting from the last transformation matrix (i.e., the transformation matrix of the point cloud data of the latest acquired frame relative to the reference frame point cloud data), multiply it by the previous transformation matrix. The result of the multiplication will be used as the input for the next multiplication operation. If there are three subsequent data frames with three transformation matrices T2, T3, and T4, where T2 is the transformation matrix relative to the reference frame, T3 is the transformation matrix relative to T2, and T4 is the transformation matrix relative to T3. Calculate T3×T2 to obtain the transformation matrix from the reference frame to T3. Then, multiply T4 by the result of the previous step, i.e., T4×(T3×T2), to obtain the complete transformation matrix from the reference frame to T4. The finally obtained transformation matrix is the target space transformation matrix, which contains the complete transformation information from the reference frame to all subsequent frames. In this way, step S104 ensures that the transformation information of all subsequent frames is integrated into a single matrix, facilitating the subsequent point cloud data fusion. This method can ensure the accuracy and consistency of the transformation.

[0065] S300. According to the target space transformation matrix, calculate the coordinates of the point cloud of frames other than the first frame of point cloud data to obtain a multi-frame point cloud data coordinate set.

[0066] Specifically, obtaining the multi-frame point cloud data coordinate set may include the following steps:

[0067] S301. Convert the coordinates of the point cloud of each frame of point cloud data in the target space transformation matrix to the coordinate system where the reference frame is located to obtain the multi-frame point cloud data coordinates.

[0068] Specifically, for the coordinates of the point cloud of each frame of point cloud data, apply the target space transformation matrix for coordinate transformation. Multiply the coordinates of each point in the point cloud by the transformation matrix to obtain the new coordinates in the reference frame coordinate system.

[0069] S302. Integrate the multi-frame point cloud data coordinates to obtain a multi-frame point cloud data coordinate set.

[0070] Collect the point cloud coordinates in each frame of point cloud data that have undergone transformation, generate a new point cloud data set for each frame, and these point cloud data sets are now located in the coordinate system of the reference frame. Next, integrate the multi-frame point cloud data to form a complete multi-frame point cloud data coordinate set. During the merging process, check and remove or merge duplicate points, optimize the data set to reduce redundancy, and improve the accuracy of the data.

[0071] Exemplarily, there are three frames of point cloud data, and each frame contains the following three points (represented by two-dimensional coordinates for simplicity):

[0072] Frame 1: (1, 1), (1.1, 1.1), (2, 2);

[0073] Frame 2: (1, 1), (2, 2), (3, 3);

[0074] Frame 3: (1.1, 1.1), (2, 2), (3, 3);

[0075] Select the voxel size to be 1x1;

[0076] Create a grid covering all the point cloud data. For example, if the range of the point cloud data is 0 - 3, then a 3x3 grid can be created.

[0077] Assign each point to the corresponding voxel. For example, the points (1, 1) and (1, 1) will be assigned to the same voxel (1, 1). For the voxel (1, 1), there are two points, and their average position (1, 1) can be taken as the representative point. The final point cloud data set contains the following points: (1, 1), (2, 2), (3, 3). These points are obtained by merging the duplicate or very close points in the original point cloud data set.

[0078] By this method, it can help integrate multi-frame point cloud data, remove duplicate points, reduce the data volume, and at the same time retain the main geometric features, providing an optimized point cloud data set for subsequent rendering and analysis.

[0079] S400. According to the multi-frame point cloud data coordinate set, create a rendering space data structure to obtain 4D annotation data for autonomous driving.

[0080] Specifically, obtaining the 4D annotation data for autonomous driving may include the following steps:

[0081] S401. In the browser, initialize WebGL.

[0082] Specifically, obtain the canvas element in the browser and initialize the WebGL context. Write and compile vertex shader and fragment shader programs for point rendering. Set the view matrix and projection matrix to define the position and perspective of the camera. Create buffer objects to store the vertex data of the point cloud data.

[0083] S402. Upload the multi-frame point cloud data coordinate set to the WebGL to obtain 4D annotation data for autonomous driving.

[0084] Specifically, the point cloud data (the point cloud data, after being processed, can be recognized by the WebGL buffer) is transmitted to WebGL for processing and rendering. Through WebGL, a large number of data points can be efficiently processed and displayed in real time in the browser. Adding a time dimension to manage and synchronize data for different frames so that the time series of the data can be correctly represented during rendering. Based on WebGL in the browser, an interactive 3D point cloud data visualization tool is provided, and users can selectively fuse the data to rotate, scale, and translate the point cloud data.

[0085] In step S402, the following steps may be included:

[0086] S4021, upload the multi-frame point cloud data coordinate sets to the buffer object of the WebGL.

[0087] S4022, use the data in the multi-frame point cloud data coordinate sets of the WebGL buffer to perform rendering in terms of the time dimension to obtain visual 4D annotation data for autonomous driving.

[0088] Specifically, when uploading the multi-frame point cloud data coordinate sets to the buffer object of the WebGL, a separate buffer object can be created for each frame of point cloud data. In this way, when a frame needs to be processed, each buffer object can be independently updated or deleted without affecting the data of other frames. For example, when the data of some frames needs to be modified or replaced. During rendering, the corresponding buffer object can be directly bound for rendering without re-uploading the data every time, thus improving the rendering performance. In addition, when uploading the multi-frame point cloud data coordinate sets to the buffer object of the WebGL, the data of all frames can be merged into a large buffer object, and different frames can be distinguished by indexes or offsets. In this way, the number of buffer objects can be reduced, and the memory consumption and management complexity can be lowered.

[0089] It should be noted that if a separate buffer object is created for each frame of point cloud data, although there are advantages in frame processing, it will occupy GPU memory. In this case, the following steps can be included:

[0090] Construct a listener and use the listener to monitor the GPU memory usage;

[0091] If the listener monitors that the GPU memory usage is greater than the set threshold, merge the earliest created separate buffers to obtain a larger buffer object; in this way, when the memory usage reaches the set threshold, the earlier frame data is merged into a larger buffer object to reduce the number of buffer objects, thereby releasing some GPU memory.

[0092] Through the above method, the memory usage can be dynamically adjusted, enabling the application to adapt to the performance and memory limitations of different hardware. Reducing the number of buffer objects may improve the rendering performance. By merging buffer objects, the GPU memory usage is reduced, avoiding memory overflow.

[0093] However, in the case of merging individual buffers into a larger buffer object, it will lead to an adjustment of the rendering logic. Therefore, an index buffer is created for the merged buffer (the larger buffer object), and the index buffer records the starting index and data length of each frame of data in the merged buffer. Additionally, the merged buffer object is divided into multiple blocks, and each block contains a certain number of frames of data. Next, during rendering, according to the block to which the frame belongs, the corresponding buffer object is bound, and the offset is set for rendering. In this way, while reducing the number of buffer objects, precise control over each frame of data can be maintained, and data chunking provides better memory management and potential performance improvement.

[0094] Exemplarily, there are 100 frames of point cloud data, each frame contains 1000 points, and each point has 3 floating-point coordinates (x, y, z).

[0095] Buffer object management: Create a buffer object for each frame of data separately.

[0096] Memory threshold setting: Set the GPU memory usage threshold to 50MB.

[0097] Data upload and buffer creation: Create a separate buffer object for each frame of data and upload the data:

[0098] Each buffer object stores the point cloud data of one frame, has independent management, and is easy to update and delete specific frames.

[0099] Memory listener: Regularly check the GPU memory usage. If it exceeds the threshold, merge the earlier buffer objects to free up memory.

[0100] Listening mechanism: Check the memory usage in the timer or requestAnimationFrame loop. For example, there is a function getGPU memoryUsage() to obtain the GPU memory usage.

[0101] When the memory usage exceeds the threshold, merge the earliest several frames of data into a larger buffer object. For example, select the earliest 10 frames for merging. Connect the data of these 10 frames in sequence to form a large array. Create a new buffer object and upload the merged data. Update the frame information to record the starting index and number of points of each frame in the merged buffer.

[0102] Rendering logic adjustment:

[0103] Index management: Create an index table for the merged buffer object to record the starting index and number of points of each frame of data in the buffer.

[0104] Render a specific frame: When rendering a specific frame, bind the corresponding buffer object according to the index table and set the offset for rendering.

[0105] Data chunking:

[0106] Chunking strategy: Further divide the merged buffer object into multiple chunks, each chunk containing a certain number of frames of data.

[0107] Select a chunk during rendering: According to the chunk to which the frame belongs, bind the corresponding buffer object and set the offset for rendering.

[0108] By the above method, while reducing the number of buffer objects, precise control over each frame of data can be maintained, thereby optimizing memory usage and rendering performance.

[0109] Embodiment

[0110] A method for processing 4D annotation data for autonomous driving may include:

[0111] Obtaining multi-frame point cloud data:

[0112] Obtain multi-frame point cloud data from a server or local storage through a browser.

[0113] The point cloud data format can be PCD, PLY, or other supported formats.

[0114] Data preprocessing:

[0115] Use JavaScript to parse the point cloud data file format and convert it into a binary data structure suitable for WebAssembly processing.

[0116] Pass the parsed data to the WebAssembly module for further processing.

[0117] Matrix transformation:

[0118] Perform matrix transformation of the point cloud data in the WebAssembly module, including but not limited to operations such as rotation, translation, and scaling.

[0119] Utilize the high computing power of WebAssembly to accelerate large-scale matrix operations to reduce processing time.

[0120] Data transmission:

[0121] Transfer the converted point cloud data from the WebAssembly module back to the JavaScript environment and prepare to pass it to WebGL for rendering.

[0122] 3D rendering:

[0123] Create a 3D rendering context using WebGL and initialize the shader program.

[0124] Load the point cloud data into the GPU buffer and set the corresponding vertex attribute pointers.

[0125] Use the WebGL drawing commands to render the point cloud data and achieve real-time 3D display.

[0126] In a second aspect, the present application provides an autonomous driving 4D annotation data processing system, which is applied to the aforementioned method. The WebAssembly module is loaded on the browser side, and the browser side is used to fuse multiple frames of point cloud data. The system includes:

[0127] An acquisition unit for acquiring multiple frames of point cloud data to be fused;

[0128] A processing unit for using the first frame of point cloud data in the multiple frames of point cloud data to be fused as a reference frame, and converting the pose information of the subsequent frames of point cloud data into a spatial transformation matrix in chronological order to obtain a target spatial transformation matrix;

[0129] The processing unit is further configured to calculate the point cloud coordinates of the frames other than the first frame of point cloud data according to the target spatial transformation matrix to obtain a multi-frame point cloud data coordinate set;

[0130] A result unit for creating a rendering space data structure according to the multi-frame point cloud data coordinate set to obtain autonomous driving 4D annotation data.

[0131] In a third aspect, the present application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the aforementioned method are implemented.

[0132] In a fourth aspect, the present application provides a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0133] In summary, an autonomous driving 4D annotation data processing method and system of the present application have the following beneficial effects:

[0134] 1. Real-time processing: Through real-time processing on the browser side, the real-time performance of point cloud data fusion is significantly improved, which is suitable for application scenarios that require real-time updates such as autonomous driving and security monitoring.

[0135] 2. Reduce computing requirements: By leveraging the computing power of the terminal, the computing power requirements for users to centrally access cloud devices are reduced, and ordinary computing devices can be used to run.

[0136] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0137] Each embodiment in this disclosure is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

[0138] The protection scope of this disclosure is not limited to the above embodiments. Obviously, those skilled in the art can make various changes and modifications to this disclosure without departing from the scope and spirit of this disclosure. If these changes and modifications fall within the scope of the claims of this disclosure and their equivalent technologies, the intention of this disclosure also includes these changes and modifications.

Claims

1. A method for processing 4D annotation data for autonomous driving, characterized in that: The WebAssembly module is loaded on the browser side, and the multi-frame point cloud data is fused by using the browser side, the method comprising: Obtain multi-frame point cloud data to be fused; The first frame of point cloud data in the multiple frames of point cloud data to be fused is used as a reference frame, and the position and posture information of the subsequent frames of point cloud data are converted into a spatial transformation matrix in time sequence to obtain a target spatial transformation matrix; According to the target space transformation matrix, the frame point cloud coordinates other than the first frame point cloud data are calculated to obtain a multi-frame point cloud data coordinate set; A rendering space data structure is created based on the multi-frame point cloud data coordinate set to obtain 4D annotation data for autonomous driving.

2. The method for processing 4D annotation data for autonomous driving according to claim 1, characterized in that: The step of obtaining the multi-frame point cloud data to be fused includes: Obtain the multi-frame initial point cloud data to be fused from the server or local storage through the browser; The initial point cloud data is parsed using JavaScript to convert the format and obtain point cloud data with a binary data structure suitable for WebAssembly processing.

3. The method for processing 4D annotation data for autonomous driving according to claim 1, characterized in that: The step of taking the first frame of point cloud data in the multiple frames of point cloud data to be fused as a reference frame, and converting the position and posture information of subsequent frames of point cloud data into a spatial transformation matrix in chronological order to obtain a target spatial transformation matrix includes: Acquire the first frame of point cloud data to obtain the reference frame of point cloud data; In chronological order, the subsequent frame point cloud data to be fused is taken as a reference by the reference frame point cloud data to obtain its pose information relative to the reference frame point cloud data, thereby obtaining the pose information of the subsequent frame point cloud data, wherein the pose information of the subsequent frame point cloud data includes rotation and translation parameters relative to the reference frame point cloud data; According to the pose information of each frame of point cloud data in the subsequent frame point cloud data, a multi-frame transformation matrix to be fused is obtained, wherein the multi-frame transformation matrix to be fused indicates that the point cloud data of each frame in the subsequent frame point cloud data forms a transformation matrix and is combined into a multi-frame transformation matrix; The transformation matrices of the multiple frames to be fused are calculated to obtain a target space transformation matrix.

4. The method for processing 4D annotation data for autonomous driving according to claim 3, characterized in that: The step of calculating the transformation matrix of the multiple frames to be fused to obtain the target space transformation matrix includes: According to the time sequence, the multi-frame transformation matrices to be fused are subjected to chain multiplication calculation to obtain the target space transformation matrix.

5. The method for processing 4D annotation data for autonomous driving according to claim 1, characterized in that: The step of calculating the point cloud coordinates of frames other than the first frame of point cloud data according to the target space transformation matrix to obtain a coordinate set of multiple frames of point cloud data includes: Convert the frame point cloud coordinates of each frame of point cloud data in the target space transformation matrix to the coordinate system of the reference frame to obtain the coordinates of multiple frames of point cloud data; The multi-frame point cloud data coordinates are integrated to obtain a multi-frame point cloud data coordinate set.

6. The method for processing 4D annotation data for autonomous driving according to claim 1, characterized in that: The step of uploading the coordinate set of the multi-frame point cloud data to the WebGL to obtain 4D annotation data for autonomous driving includes: Use WebGL to create a 3D rendering context and initialize the shader program; Load the multi-frame point cloud data coordinate set into the GPU buffer and set the corresponding vertex attribute pointer; Use WebGL drawing commands to render point cloud data to obtain real-time 3D display of 4D annotation data for autonomous driving.

7. The method for processing 4D annotation data for autonomous driving according to any one of claims 1 to 6, characterized in that: The point cloud data format is one of PCD and PLY.

8. A 4D annotation data processing system for autonomous driving, characterized in that: The method applied to any one of claims 1 to 7, loading a WebAssembly module on a browser, and using the browser to fuse multiple frames of point cloud data, the system comprises: An acquisition unit, used for acquiring multi-frame point cloud data to be fused; A processing unit, configured to use the first frame of point cloud data among the multiple frames of point cloud data to be fused as a reference frame, and convert the position and posture information of subsequent frames of point cloud data into a spatial transformation matrix in a time sequence to obtain a target spatial transformation matrix; The processing unit is further used to calculate the point cloud coordinates of frames other than the first frame of point cloud data according to the target space transformation matrix to obtain a coordinate set of multiple frames of point cloud data; The result unit manually creates a rendering space data structure according to the multi-frame point cloud data coordinate set to obtain 4D annotation data for autonomous driving.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images