Automatic driving multi-sensor data processing method and system

Abstract

The invention provides an automatic driving multi-sensor data processing method and system, and belongs to the technical field of automatic driving. By performing unified formatting processing on each type of sensor data, adding the sensor data meeting the preset condition into the thread security queue, and based on removing processing on the data in the thread security queue, determining actual frame data, different data formats of various sensors can be intelligently analyzed; the bird's-eye view is drawn based on the actual frame data, and the information list for displaying the data in the bird's-eye view is constructed, so that the data under each timestamp can be ensured to accurately correspond and synchronously display, and the analysis efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving multi-sensor data processing method and system. Background Technology

[0002] During road testing of autonomous driving test vehicles, various types of sensors typically equipped on the vehicle collect a variety of data. In certain scenarios where the autonomous driving system malfunctions, it is particularly important to analyze this collected data and determine which sensor's data loss or misidentification caused the driving strategy to fail to meet expectations.

[0003] Due to the different sensor types and varying file formats for storing data, traditional multi-sensor data processing methods face numerous challenges. Finding and obtaining corresponding parameters requires opening multiple data files, leading to cumbersome operations and unintuitive data presentation, significantly reducing analysis efficiency and accuracy. Therefore, how to parse the different data formats from various sensors and ensure that data at each timestamp can be accurately matched and displayed synchronously is an urgent problem to be solved. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art and proposes a method for processing multi-sensor data for autonomous driving.

[0005] In a first aspect, the present invention provides a method for processing multi-sensor data in autonomous driving, the method comprising:

[0006] The data from various sensors are formatted in a unified manner, and based on the formatted sensor data, the minimum and maximum timestamps of all sensor data are determined.

[0007] Based on the minimum timestamp and the maximum timestamp, sensor data that meets the preset conditions is selected from all sensor data;

[0008] Sensor data that meets the preset conditions is added to a thread-safe queue. Based on the data removal process in the thread-safe queue, the actual frame data is determined.

[0009] A bird's-eye view is drawn based on the actual frame data, and an information list is constructed to display the data in the bird's-eye view.

[0010] Optionally, the various sensor data include at least one or a combination of chassis data, lidar data, millimeter-wave radar data, visual data, and fused data, wherein the fused data is formed by fusing visual data and millimeter-wave radar data.

[0011] Optionally, the unified formatting of various sensor data includes:

[0012] For the fused data, the file content is read line by line and the data is parsed, capturing any abnormal states that may occur during the data parsing process;

[0013] If an abnormal state is detected, the merged data of the current row is skipped and the next row is parsed; if no abnormal state is detected, the merged data of the current row is parsed.

[0014] After the data parsing is complete, the resource release function is called to release the occupied resources.

[0015] Optionally, the step of filtering sensor data that meets preset conditions from all sensor data based on the minimum and maximum timestamps includes:

[0016] Using the minimum timestamp as the baseline value, for each sensor data, the difference between the timestamp of the sensor data and the minimum timestamp is calculated, and the difference is divided by a preset value to obtain the conversion value relative to the baseline value.

[0017] Initialize the timer and set its running time interval; obtain the cumulative time after the timer starts; each time the timer starts, based on the cumulative time and the conversion value of the sensor data relative to the reference value, filter out the sensor data that meets the preset conditions from all sensor data.

[0018] Optionally, drawing the bird's-eye view based on the actual frame data includes:

[0019] Set the center point coordinates of the location of the data collection vehicle as the origin of the coordinate system in the bird's-eye view, and draw an initial rectangle representing the location of the data collection vehicle based on its dimensions.

[0020] Iterate through each sensor data in the actual frame data, and determine whether the data source corresponding to the sensor data is selected. If so, retain the sensor data; otherwise, filter the sensor data.

[0021] For each piece of sensor data that is retained, its corresponding target identifier and color are determined; the target identifier is displayed in that color inside the target rectangle, thereby completing the drawing of the bird's-eye view; wherein, the target rectangle is obtained by transforming the initial rectangle using a rotation transformation method.

[0022] Optionally, after drawing an initial rectangle representing the location of the data collection vehicle based on its dimensions, the method further includes:

[0023] Set the grid line spacing, and draw vertical and horizontal grid lines based on the iterative algorithm and the grid line spacing.

[0024] Optionally, the construction of the information list for displaying data in the bird's-eye view includes:

[0025] Obtain the target list corresponding to the bird's-eye view;

[0026] Iterate through each target in the target list, check if the data source corresponding to the target is selected, and if it is selected, create a new row and populate it with the data from the data source.

[0027] Secondly, embodiments of the present invention provide an autonomous driving multi-sensor data processing system, comprising:

[0028] The first determining module is used to perform uniform formatting processing on each type of sensor data, and based on the formatted sensor data, determine the minimum and maximum timestamps of all sensor data.

[0029] The filtering module is used to filter sensor data that meets preset conditions from all sensor data based on the minimum timestamp and the maximum timestamp;

[0030] The second determining module is used to add sensor data that meets preset conditions to a thread-safe queue, and determine the actual frame data based on the data removal process in the thread-safe queue.

[0031] The display module is used to draw a bird's-eye view based on the actual frame data and to construct an information list for displaying the data in the bird's-eye view.

[0032] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0033] One or more processors;

[0034] Memory, used to store one or more programs;

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described autonomous driving multi-sensor data processing method.

[0036] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program, which, when executed by a processor, implements the steps of the above-described autonomous driving multi-sensor data processing method.

[0037] This invention provides a multi-sensor data processing method for autonomous driving. By uniformly formatting the data from each sensor, sensor data that meets preset conditions is added to a thread-safe queue. Based on the removal of data from the thread-safe queue, the actual frame data is determined. This method can intelligently parse different data formats from multiple sensors. Based on the actual frame data, a bird's-eye view is drawn, and an information list for displaying the data in the bird's-eye view is constructed. This ensures that the data at each timestamp can be accurately matched and displayed synchronously, improving analysis efficiency and accuracy. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a multi-sensor data processing method for autonomous driving provided in an embodiment of the present invention;

[0039] Figure 2 This is a flowchart illustrating an optional specific implementation method of step S4 in an embodiment of the present invention;

[0040] Figure 3 This is a flowchart illustrating another optional specific implementation method of step S4 in an embodiment of the present invention;

[0041] Figure 4 A structural block diagram of an autonomous driving multi-sensor data processing system provided in an embodiment of the present invention;

[0042] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0044] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0045] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0047] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0048] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0049] Current multi-sensor data processing methods have many shortcomings, especially given the diverse types of sensors and varying file formats used to store data, which increases the difficulty of multi-sensor data processing. Most existing methods rely on manual data retrieval, which is cumbersome and the data presentation is not intuitive.

[0050] To address at least one of the technical problems existing in the aforementioned related technologies, this invention provides a method and system for processing multi-sensor data in autonomous driving. By uniformly formatting the collected data, it achieves the integration and processing of multi-sensor data, and can intelligently parse different data formats of various sensors, ensuring that the data at each timestamp can be accurately matched and displayed synchronously.

[0051] Figure 1 A flowchart illustrating an autonomous driving multi-sensor data processing method provided in this embodiment of the invention includes the following steps:

[0052] S1, perform unified formatting processing on various sensor data, and determine the minimum and maximum timestamps of all sensor data based on the formatted sensor data;

[0053] The data collected by the sensors can be stored in the form of data files. The data collected by each sensor can contain a variety of parameters. Common parameters may include key information such as the lateral and longitudinal displacement of the object (also known as the target) relative to the data collection vehicle, the offset angle, the relative speed, and the vehicle type.

[0054] In practical applications, this invention does not limit the specific storage method of the data files. For example, some data files use one line to represent all object data within a timestamp, where data collected from multiple objects are merged into one line, and different objects are distinguished by the column names of data[n].X. Other data files can list the data of each object under each timestamp separately.

[0055] In one embodiment of the present invention, the various sensor data include at least one or a combination of chassis data, lidar data, millimeter-wave radar data, visual data, and fused data, wherein the fused data is formed by fusing visual data and millimeter-wave radar data.

[0056] The data collection vehicle can be equipped with various types of sensors, such as vision sensors, LiDAR, millimeter-wave radar, and chassis sensors. The data from each sensor corresponds to the data collected by that specific sensor; for example, the data collected by the aforementioned sensors corresponds to visual data, LiDAR data, millimeter-wave radar data, and chassis data, respectively. Furthermore, fused data can be obtained by fusing visual data and millimeter-wave radar data. Because the fused data contains both visual and millimeter-wave radar data, it can intuitively distinguish different types of objects and has the advantage of accurately measuring distances.

[0057] Accordingly, different data converters can be set up to perform unified formatting processing of different data. If there are five sources of sensor data, five corresponding data converters can be set up: Fusion Data Converter (FusionOutput_convert), LiDAR Data Converter (Ch4_convert), Millimeter-Wave Radar Data Converter (Hmb_radar_convert), Chassis Data Converter (VehicleConvert), and Vision Data Converter (Vision_convert). Each data converter can parse the data from the original file and store it in a custom ObjectFrame class. Specifically, the main fields in the ObjectFrame class are explained in Table 1 below.

[0058] Table 1 Explanation of ObjectFrame Class Fields

[0059]

[0060] In one embodiment of the present invention, the unified formatting processing of various sensor data includes:

[0061] For the fused data, it reads and parses line by line, and captures any anomalies that may occur during the parsing process;

[0062] After catching the exception, skip the merged data of the current row and continue parsing the data of the next row;

[0063] After the data parsing is complete, call the resource release function to release the occupied resources.

[0064] The data file format for fused data can be such that each line represents all object data within a single timestamp, meaning each line stores all target parameter information collected in one frame. Specifically, a fusion data converter (FusionOutput_convert) can be used to perform unified formatting processing on the fused data.

[0065] For ease of understanding, the present invention will use the following processing procedure as an example to illustrate the specific formatting process of the fused data:

[0066] Step a, file opening and initial setup reading; create a StreamReader file stream object sr for reading file content line by line.

[0067] Step b: Read the file content line by line and parse the data; use a while loop to traverse each line of the file until the file is completely read. Within the loop, call sr.ReadLine() to read the current line. First, check if the current line is empty; if it is, skip the current line and continue reading the next line. If the current line is not empty, parse the data for that line.

[0068] Step c, exception handling; use a try-catch structure to catch possible exceptions to prevent program crashes. After catching an exception, the data in the current line will be ignored, ensuring that subsequent data parsing continues.

[0069] Step d: Update the state and close the resources; call the resource release functions in sequence, such as the Close() and Dispose() methods of FileStream and StreamReader, to release the occupied resources.

[0070] In one specific implementation, the data parsing process in step b is as follows:

[0071] Step b1: Separate each row of data into an array `row` using commas, and check `row.Length` to see if the data length meets the minimum requirement to ensure data integrity. If a row of data is incomplete (e.g., `row.Length` is insufficient), it can be skipped or an error log can be recorded.

[0072] Step b2: Create an ObjectFrame object; initialize an ObjectFrame object realFrame to record all target information collected at the current timestamp. Since the fused data format is relatively simple and the correspondence between parameters and column indices is relatively stable, relevant information can be directly obtained using the column indices of the parameters.

[0073] Step b3, parsing timestamps and target counts: By calling the custom GetInt32Value(row[0]) method, the timestamp data of the first column is assigned to realFrame.timespan; the GetInt32Value(row[3]) method is called to obtain the target count objCount in the current row. It is understandable that the GetInt32Value method will check whether the passed string is null or "na". If it is, it returns the default value 0; otherwise, it returns the original value. The GetInt32Value method can ensure the robustness of numerical parsing and avoid program exceptions caused by invalid data.

[0074] Step b4: Construct the target list; initialize targetList and create an empty List. <target>This is used to store information about each target. The target information can be represented by a custom Target class, which can contain multiple fields, including [objId, objClass, position_X, position_Y, width, length, heading, source, speed]. The main fields are explained in Table 2 below.

[0075] Table 2

[0076]

[0077] Step b5: Read the target data. For each target in any column, iterate through them one by one using a for loop. For each target, check if its X and Y coordinates are zero. If they are zero, it means no object information was collected, so skip this iteration. If they are not zero, create an instance of type Target and populate the relevant data for that target according to the corresponding columns. It's understandable that since the number of parameters for each target is fixed, the field is defined as `obj_param_count` in the code. During iteration, only the parameter column index `n` corresponding to the first target needs to be found. The parameter column indices for subsequent targets can be calculated using the following formula: `i * obj_param_count + n`.

[0078] Step b6: Set target information; assign the target ID, category, location, speed, width, length, angle, and data source information from the parsed data to the corresponding fields in the Target class. Populate the targetList with all Target information for targets collected at the same timestamp.

[0079] Step b7: Store the data; assign the populated targetList to realFrame.targetList, and add the realFrame objects to the external frameList list. It can be seen that the frameList is a collection of realFrames. After parsing the entire file (such as a CSV file), frameList stores all realFrame objects at all timestamps, which will be used as parameter information for drawing the bird's-eye view later.

[0080] The data files acquired by the chassis can be CAN bus data. The file format is characterized by each line representing one frame of data (i.e., data within a timestamp). Each frame consistently collects data from only nine targets. Furthermore, the number of columns in the file varies depending on the number of CAN signals acquired, and the correspondence between each parameter is not fixed. Therefore, direct indexing by column count is not possible. A chassis data converter can be used to uniformly format the chassis data.

[0081] The present invention uses the following processing procedure as an example to illustrate the specific formatting process of chassis data:

[0082] Step A: Use a StreamReader object to read the file stream and call the ReadLine() method in the StreamReader class to get the first line of the file. Then, use the Split method to split the string into two parts, each separated by a comma, generating a list of strings named listCols. Each element in this list corresponds to a column name in the CSV file.

[0083] Step B involves using a for loop to iterate through each element in listCols. In the chassis data, the column names for the same parameter of different targets do not follow a strict pattern. For example, the x-axis coordinate column name for the first target is HAD_CAMERA_OBJ1LatDst, and the x-axis coordinate column name for the 9th target is HAD_P2P_OBJ9LatDst. Given the limited number of targets, this invention uses the strict equality operator "==" to strictly match the traversed column names. Specifically, when traversing to the HAD_CAMERA_OBJ1LatDst column, the current column index i is assigned to Obj2IndexList[0].x, thus marking the row number of the first target's x-axis coordinate. Similarly, through a series of conditional judgments, the index values ​​of each target and its related parameters are stored in Obj2IndexList, ensuring that subsequent data processing can correctly reference these parameters.

[0084] Among them, Obj2IndexList is a List collection containing 9 Obj2Index objects. The main fields in the custom Obj2Index class are [x, y, id, type, head, speed], and their explanations are shown in Table 3 below.

[0085] Table 3

[0086]

[0087] In this way, the code can dynamically adapt to the data structure stored in the CSV file, ensuring accurate reading and storage of the data.

[0088] The remaining steps are basically the same as the fusion converter process described above: reading data from each line of the CSV file line by line and adding the realFrame object of each frame to the external frameList. The difference lies in that when assigning information such as target ID, category, position, speed, width, length, orientation, and data source to the Target, the index of the read row is located by the elements stored in Obj2IndexList. For example, when reading the x-axis coordinate, the column index of the current i-th target is Obj2IndexList[i-1].x.

[0089] The data file format acquired by LiDAR differs from other data formats. The parameter information of each target acquired at each timestamp is stored in a separate row. Therefore, the number of columns is small, the format is relatively simple, and the correspondence between parameters and column indices is relatively stable. Based on this characteristic, its processing is the same as that of the fusion converter, and relevant information can be obtained directly using the column index of the parameters.

[0090] LiDAR data can be uniformly formatted using a LiDAR converter. The specific processing procedure is similar to that of a fusion converter, including reading each line of data from a CSV file and adding the realFrame object of each frame to an external frameList. The main difference is that the LiDAR converter needs to create a temporary timestamp (timestemp) as a frame identifier. When reading line data, it compares the current line's timestamp (timesnow) with the values ​​of timestemp. If they are different, the value of timesnow is assigned to the temporary timestamp, and an ObjectFrame is created. The read parameters are then assigned to this ObjectFrame. If they are the same, the LINQ FirstOrDefault method is used to find the first realFrame object in the frameList whose timestamp attribute equals the temporary timestamp. The parameters of the current line are assigned to the newly created Target, and the new Target is added to the realFrame.TargetList, while the timestemp value remains unchanged. This solution uses the FirstOrDefault method to ensure that only the first instance is processed when multiple matching frames exist. This design effectively handles the row-by-row storage characteristic of multiple targets in a single frame in LiDAR data while maintaining the integrity of the time series.

[0091] Visual data and millimeter-wave radar data are similar, therefore the basic principles of their converters are also the same. Taking millimeter-wave radar data as an example, the data file format acquired by millimeter-wave radar is the same as the fused data. Parameter information for all targets acquired at each timestamp is stored in a single line. There are many parameters, and the data is uploaded to multiple folders. The sensors acquiring the data include front-angle millimeter-wave radar, rear-angle millimeter-wave radar, and forward-facing millimeter-wave radar. The CSV files generated by different sensors have differences in the number of columns and inconsistent parameter positions. The number of targets detected in a single frame is also variable. Therefore, it is not possible to directly obtain relevant information using the column index of the parameters, nor can the strict equality operator "==" be used to strictly match the traversed column names with the required column names. To integrate the parsing of data from different CSV files into a single method, considering the generally strong regularity of column names for the same parameter across different targets in millimeter-wave radar data storage, a dictionary approach is adopted. The column number is obtained through strings in the dictionary, thus yielding the corresponding value.

[0092] The specific processing procedure can be the same as that of the fusion converter, including: reading data from each row of the CSV file line by line, and finally adding the realFrame object of each frame to an external frameList. The difference lies in using a dictionary mapping mechanism to handle parameter positioning. Specifically, this involves: iterating through the first row of the column name array, storing the column name as the key and the column index as the value in the dictionary DicName6Index, and retrieving the dynamic column index by querying the dictionary. For example, the column name corresponding to the x-axis coordinate of the (i-1)th target is:

[0093] The method to read the x-axis coordinate parameter of a certain frame is: row[DicName6Index["front_radar.objects_dataobjects_data[" + i + "].x"]].

[0094] S2, based on the minimum timestamp and the maximum timestamp, filter out sensor data that meets the preset conditions from all sensor data;

[0095] In one embodiment of the present invention, the step of filtering sensor data that meets preset conditions from all sensor data based on the minimum and maximum timestamps includes:

[0096] Using the minimum timestamp as the baseline value, for each sensor data, the difference between the timestamp of the sensor data and the minimum timestamp is calculated, and the difference is divided by a preset value to obtain the conversion value relative to the baseline value.

[0097] Initialize the timer and set its running time interval; obtain the cumulative time after the timer starts; each time the timer starts, based on the cumulative time and the conversion value of the sensor data relative to the reference value, filter out the sensor data that meets the preset conditions from all sensor data.

[0098] In general, the timestamps in the parsed data files are arranged in ascending order by row. Therefore, the minimum timestamp for each type of sensor data is the first record, and the maximum timestamp is the last record. Thus, we can use `realFrame.first()` and `realFrame.last()` to find the minimum and maximum timestamp values ​​in the data collected by each sensor in the `frameList` list. By comparing the maximum and minimum values ​​found for each sensor, we can find the minimum and maximum timestamps in all the data.

[0099] Furthermore, using the minimum timestamp as a reference (which can be denoted as the 0-point timestamp), the difference between all timestamps and the minimum timestamp is calculated and divided by a preset value (such as 1000), thus converting it into seconds relative to 0. The converted value represents the number of seconds each timestamp is from 0. This embodiment can align the timestamps of all sensors to a unified reference, ensuring time consistency of cross-sensor data.

[0100] Furthermore, a timer can be set, with a timer object named `_stopwatch` and its running time interval set, thus triggering at fixed intervals (e.g., 16ms corresponding to 60Hz). Each time the timer triggers, the `ElapsedMilliseconds` property returns the elapsed time since the timer started, denoted as `interval`. After each timer starts, frames meeting preset conditions can be searched in the `frameList` to obtain sensor data that meets these conditions. The preset conditions can be designed in advance according to requirements; this invention does not limit this. For example, the preset conditions could be: the timestamp of the current frame ≤ `interval`, and the timestamp of the next frame > `interval`, thus finding the data frame corresponding to the current time point `interval`. Applying this embodiment of the invention, the timer can trigger at fixed intervals, simulating a real-time data stream and dynamically selecting the frame to be displayed.

[0101] S3, add sensor data that meets the preset conditions to the thread-safe queue, and determine the actual frame data based on the data removal process in the thread-safe queue;

[0102] S4. Draw a bird's-eye view based on the actual frame data, and construct an information list to display the data in the bird's-eye view.

[0103] After confirming that the sensor data meets the preset conditions, it can be added to a thread-safe queue (such as _realQueue). Subsequently, the head element of the queue _realQueue can be removed by using the TryDequeue method and assigned to realFrames. The parameters in realFrames (such as horizontal and vertical displacement, offset angle, relative speed, and vehicle type) can be used to draw the bird's-eye view of the current frame.

[0104] The `for` loop iterates through the `realFrames` list, where each element `realFrames[i]` represents data collected from one sensor at a given moment. By iterating through the loop, data from all sensors is processed frame by frame. Then, the `draw` method, using `realFrames[i]` as a parameter, draws the current sensor data onto a bird's-eye view, thus visualizing the target detection results (such as vehicles and pedestrians) from different sensors in a unified bird's-eye view coordinate system as rectangles and IDs. Target rectangles from different sensors may overlap (e.g., radar and vision simultaneously detect the same vehicle); the source is distinguished by color, and their display or hiding is controlled via an interface switch.

[0105] Below the bird's-eye view, you can use GridControl and GridView components to build one or more information lists, displaying information such as the target identifier (or simply id) of the data in the bird's-eye view, horizontal distance, vertical distance, speed, and data source. This allows users to quickly match the data in the list using the id and color within the rectangle in the bird's-eye view, and intuitively obtain the information of the collection target.

[0106] This invention provides a multi-sensor data processing method for autonomous driving, mainly including unified formatting of different types of collected data, drawing of a bird's-eye view, and filling of a bird's-eye view information list. By uniformly formatting each type of sensor data, sensor data meeting preset conditions are added to a thread-safe queue. Based on data removal from the thread-safe queue, the actual frame data is determined. A bird's-eye view is drawn based on the actual frame data, and an information list for displaying the data in the bird's-eye view is constructed. This invention can intelligently parse multiple sensor data in different data formats; it can ensure that data at each timestamp can be accurately matched and displayed synchronously, improving analysis efficiency and accuracy.

[0107] like Figure 2 As shown, in one embodiment of the present invention, S4, drawing a bird's-eye view based on the actual frame data, includes:

[0108] S41, set the center point coordinates of the location of the data collection vehicle as the origin of the coordinates in the bird's-eye view, and draw an initial rectangle representing its position according to the size of the data collection vehicle;

[0109] S42, traverse each sensor data in the actual frame data, determine whether the data source corresponding to the sensor data is selected, if yes, retain the sensor data; otherwise, filter the sensor data.

[0110] S43, for each retained sensor data, determine its corresponding target identifier and color; display the target identifier in the color inside the target rectangle to complete the drawing of the bird's-eye view; wherein, the target rectangle is obtained by transforming the initial rectangle using a rotation transformation method.

[0111] After formatting all the data, a bird's-eye view of each target captured by each sensor can be drawn, using the data acquisition vehicle as the coordinate origin. Since the bird's-eye view needs to be updated frequently with the video frame rate, drawing directly on the display screen may cause noticeable screen flicker. To solve this problem, BufferedGraphics technology can be used for drawing the bird's-eye view. This technology creates a "graphics buffer" in memory, performs all drawing operations within this buffer, and then renders the results to the screen all at once. Using BufferedGraphics not only effectively reduces screen flicker and improves drawing efficiency, but also optimizes memory usage, simplifies code structure, and ultimately improves the quality of graphics rendering.

[0112] For example, a specific rendering implementation method can be:

[0113] The `BufferedGraphics` object is initialized to null, indicating that no buffer has been allocated yet. Next, the `BufferedGraphicsContext` instance, which manages the allocation and deallocation of `BufferedGraphics` objects in the current application domain, is obtained via `BufferedGraphicsManager.Current`. Then, the `Allocate` method in the `BufferedGraphicsContext` class is called to allocate memory for the `BufferedGraphics` object, thus implementing double-buffered drawing.

[0114] Since the focus of observation is usually on the target vehicle in front of the data collection vehicle, the data collection vehicle can be placed in the lower center of the drawing area and set as the origin of the coordinates in the bird's-eye view. Specifically, a Panel control in WebForms can be used and named panel_bev. Before starting to draw, first obtain the length and width of the control, and then calculate the coordinates based on these dimensions and the size of the data collection vehicle to determine the center point coordinates of the data collection vehicle. Finally, set the midpoint of the control's width and the position 100 pixels from the bottom as the center of the rectangle, and use the DrawRectangle method in the Graphics class to draw a rectangle representing the position of the data collection vehicle according to its size. The color can be set to red or other colors; this invention does not limit this.

[0115] The target data collected in the same scene comes from various sources, including chassis data, LiDAR data, millimeter-wave radar data, visual data, and fused data integrating visual and millimeter-wave radar data. Understandably, rectangles drawn from different sources for the same target may overlap, leading to visual clutter and difficulty in analysis. To address this, data can be displayed differently based on its source. Before drawing the bird's-eye view, a custom `getPenByType` method is used to determine the data source (source) value of each target object in `realFrame[i].targetList`, assigning different colors to the Pen objects accordingly. For example, if the source is LiDAR (jg_radar), it returns red, thus drawing rectangles of different colors. Furthermore, to address the issue of overlapping rectangles of different data types, corresponding buttons are added to the interface for different data sources. For example, a `DevExpress.XtraBars.BarToggleSwitchItem` control is used. If the control corresponding to the data source is not enabled during iteration, the loop is skipped, and subsequent bird's-eye view drawing is not performed. This allows users to flexibly choose to display radar, visual, or fused data, avoiding overlap and confusion, and ensuring that only the data the user needs is displayed on the interface.

[0116] Before drawing the bird's-eye view, the target rectangle can be rotated and labeled with its ID. The specific rotation process can be as follows: save the current transformation matrix, calculate the rotation center, ensure that the rotation center is the geometric center of the rectangle to avoid offset, apply the rotation center to rotate, and draw the target rectangle.

[0117] For example, the implementation can be as follows: Obtain the transformation matrix of the current Graphics object through the `bufferedGraphics.Graphics.Transform` property and store it in a variable named `matrix`. Then, iterate through `realFrame[i].targetList`. For each `target` object, first use `Matrix.RotateAt` to rotate it around the center point of the rectangle (`target.position_X + target.width / 2, target.position_Y + target.length / 2`). Apply the defined rotation transformation matrix to `bufferedGraphics.Graphics`. Use `Pen`, `target.position_X`, `target.width`, `target.position_Y`, and `target.length` as parameters to draw the target rectangle using `DrawRectangle` in Graphics. After drawing, rotate the transformation matrix of the Graphics object by `-target.heading` to restore the initial position. After drawing the rectangle, use `DrawString` to display `target.id` inside the rectangle, facilitating a one-to-one correspondence with the data in the bird's-eye view information list.

[0118] In summary, the autonomous driving multi-sensor data processing method provided by this invention can, at the same timestamp, identify different types of sensor data with rectangles of different colors on a bird's-eye view, comprehensively and intuitively displaying the sample vehicle data and key parameter information collected at the current moment, including lateral and longitudinal displacement, offset angle, relative speed, and vehicle type. This intuitive data presentation method greatly simplifies the complexity of data analysis, improves the efficiency of problem localization and analysis, and thus provides strong support for the optimization and troubleshooting of autonomous driving systems.

[0119] In one embodiment of the present invention, after drawing an initial rectangle representing the position of the data collection vehicle according to its dimensions, the method further includes:

[0120] Set the grid line spacing, and draw vertical and horizontal grid lines based on the iterative algorithm and the grid line spacing.

[0121] To draw grid lines, a custom `draw_grid` method can be called. The main principle of `draw_grid` is to first obtain the length and width of the control, set the grid line spacing (which can be simply called `jg`), and then use a loop to draw both vertical and horizontal grid lines. Taking drawing a vertical grid line as an example, a timer `count` is initialized outside the loop. Inside the loop, the timer is incremented, and it is checked whether the product of the timer `count` and the grid spacing `jg` exceeds the width of the control. If it does, the loop is exited; otherwise, the `DrawLine` method of the `Graphics` class is used to draw a vertical line from (count * jg, 0) to (count * jg, h) within the control.

[0122] like Figure 3 As shown, in one embodiment of the present invention, step S4 involves constructing an information list for displaying data in the bird's-eye view, including:

[0123] S44, Obtain the target list corresponding to the bird's-eye view;

[0124] S45: Loop through each target in the target list, check if the data source corresponding to the target is selected. If it is selected, create a new row and populate it with the data from the data source.

[0125] For example, the process of populating a specific information list may include: constructing the information list, initializing a data table, and populating the data table. For instance, using GridControl and GridView controls to construct the information list `table_niaohan_list`, initializing the data table using the `InitMainExList` method, and using `FillTableNiaoHanList(List<NiaoHan>)`... <target>The `targetList` method is used to populate this data table. In addition to the two methods `InitMainExList` and `FillTableNiaoHanList`, a member object `DataTable` named `DataSource_table_niaohan_list` can be defined outside of these methods to serve as the data source for the list. The initialization process for the data table is as follows:

[0126] Check if it has been initialized: First, check if DataSource_table_niaohan_list is empty; if it is not empty, it means it has already been initialized, so the DataTable will not be created again to avoid wasting memory; if it is empty, create a new DataTable and add multiple columns to it, each with a name and a data type, and all columns are defined with the data type System.String; set the data source, including setting DataSource_table_niaohan_list as the data source for table_niaohan_list.

[0127] The process of populating the data table is as follows:

[0128] Target list not empty check: First, check if the data source target list realFrame[i].targetList passed in for each frame is empty. If it is empty, no further logic is executed; if it is not empty, clear all existing data rows before filling in new data to avoid old data residue. Then, iterate through each target in the target list and determine if the control corresponding to the source of the data displayed in the list is enabled. If it is not enabled, skip the current iteration; if it is enabled, create a row of data for the target and fill in the corresponding fields. Use the NewRow() method to create a new data row, assign values ​​to each column of the row, and add the row to the data source.

[0129] In summary, for the various data collected by different types of sensors on autonomous driving test vehicles during road testing, this invention can significantly enhance the efficiency and accuracy of problem analysis in defect scenarios. It allows different types of sensor data to be marked with rectangles of different colors on a bird's-eye view at the same time stamp, comprehensively and intuitively displaying the vehicle data and key parameter information collected at the current moment, including lateral and longitudinal displacement, offset angle, relative speed, and vehicle type. Of course, this invention can also be applied to other scenarios in its specific implementation.

[0130] Based on the same inventive concept, embodiments of the present invention also provide an autonomous driving multi-sensor data processing system. Figure 4 This is a structural block diagram of an autonomous driving multi-sensor data processing system provided in an embodiment of the present invention. Figure 4 As shown, an autonomous driving multi-sensor data processing system includes:

[0131] The first determining module 10 is used to perform uniform formatting processing on each type of sensor data, and based on the formatted sensor data, determine the minimum and maximum timestamps of all sensor data.

[0132] The filtering module 20 is used to filter sensor data that meets preset conditions from all sensor data based on the minimum timestamp and the maximum timestamp;

[0133] The second determining module 30 is used to add sensor data that meets preset conditions to a thread-safe queue, and determine the actual frame data based on the data removal process in the thread-safe queue.

[0134] Display module 40 is used to draw a bird's-eye view based on the actual frame data and to construct an information list for displaying the data in the bird's-eye view.

[0135] Optionally, the various sensor data include at least one or a combination of chassis data, lidar data, millimeter-wave radar data, visual data, and fused data, wherein the fused data is formed by fusing visual data and millimeter-wave radar data.

[0136] Optionally, the first determining module 10 performs unified formatting processing on the various sensor data, specifically as follows:

[0137] For the fused data, the file content is read line by line and the data is parsed, capturing any abnormal states that may occur during the data parsing process;

[0138] If an abnormal state is detected, the merged data of the current row is skipped and the next row is parsed; if no abnormal state is detected, the merged data of the current row is parsed.

[0139] After the data parsing is complete, the resource release function is called to release the occupied resources.

[0140] Optionally, the filtering module 20 filters sensor data that meets preset conditions from all sensor data based on the minimum and maximum timestamps, specifically as follows:

[0141] Using the minimum timestamp as the baseline value, for each sensor data, the difference between the timestamp of the sensor data and the minimum timestamp is calculated, and the difference is divided by a preset value to obtain the conversion value relative to the baseline value.

[0142] Initialize the timer and set its running time interval; obtain the cumulative time after the timer starts; each time the timer starts, based on the cumulative time and the conversion value of the sensor data relative to the reference value, filter out the sensor data that meets the preset conditions from all sensor data.

[0143] Optionally, the display module 40 draws a bird's-eye view based on the actual frame data, specifically:

[0144] Set the center point coordinates of the location of the data collection vehicle as the origin of the coordinate system in the bird's-eye view, and draw an initial rectangle representing the location of the data collection vehicle based on its dimensions.

[0145] Iterate through each sensor data in the actual frame data, and determine whether the data source corresponding to the sensor data is selected. If so, retain the sensor data; otherwise, filter the sensor data.

[0146] For each piece of sensor data that is retained, its corresponding target identifier and color are determined; the target identifier is displayed in that color inside the target rectangle, thereby completing the drawing of the bird's-eye view; wherein, the target rectangle is obtained by transforming the initial rectangle using a rotation transformation method.

[0147] Optionally, the display module 40 is further configured to: after drawing an initial rectangle representing the position of the acquisition vehicle according to its size, set the grid line spacing, and draw vertical grid lines and horizontal grid lines respectively based on the cyclic algorithm and the grid line spacing.

[0148] Optionally, the display module 40 constructs an information list for displaying data in the bird's-eye view, including:

[0149] Obtain the target list corresponding to the bird's-eye view;

[0150] Iterate through each target in the target list, check if the data source corresponding to the target is selected, and if it is selected, create a new row and populate it with the data from the data source.

[0151] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement an autonomous driving multi-sensor data processing method as described in any of the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0152] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0153] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0154] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0155] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of an autonomous driving multi-sensor data processing method as described in any of the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0156] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described method for processing multi-sensor data for autonomous driving.

[0157] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0158] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0159] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0160] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0161] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0162] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0163] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0164] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0166] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.< / target> < / target>

Claims

1. An automatic driving multi-sensor data processing method, characterized by, The method comprises: uniformly formatting various sensor data, determining a minimum timestamp and a maximum timestamp of all sensor data based on the formatted sensor data; filtering sensor data meeting preset conditions from all sensor data based on the minimum timestamp and the maximum timestamp; adding the sensor data meeting the preset conditions to a thread-safe queue, and determining actual frame data based on data removal processing in the thread-safe queue; drawing an aerial view based on the actual frame data, and constructing an information list for displaying data in the aerial view.

2. The method of claim 1, wherein, The various sensor data comprises at least one or a combination of chassis data, laser radar data, millimeter wave radar data, visual data, and fusion data, wherein the fusion data is obtained by fusing visual data and millimeter wave radar data.

3. The method of claim 2, wherein, The uniformly formatting various sensor data comprises: reading file content line by line and performing data parsing for the fusion data, and capturing abnormal states that may occur in the data parsing process; if an abnormal state is captured, skipping the fusion data of the current line and continuing to parse the next line of data; if no abnormal state is captured, parsing the fusion data of the current line; after the data parsing is completed, calling a resource release function to release the occupied resources.

4. The method of claim 1, wherein, The filtering sensor data meeting preset conditions from all sensor data based on the minimum timestamp and the maximum timestamp comprises: taking the minimum timestamp as a reference value, calculating, for each sensor data, a difference between the timestamp of the sensor data and the minimum timestamp, and dividing the difference by a preset value to obtain a conversion value relative to the reference value; initializing a timer and setting its running time interval; obtaining the cumulative time after the timer is started; and whenever the timer is started, filtering sensor data meeting preset conditions from all sensor data based on the cumulative time and the conversion value of the sensor data relative to the reference value.

5. The method of claim 1, wherein, The drawing an aerial view based on the actual frame data comprises: setting the center point coordinates of the position of the collection vehicle as the coordinate origin in the aerial view, and drawing an initial rectangle representing the position of the collection vehicle according to the size of the collection vehicle; traversing each sensor data in the actual frame data to determine whether the data source corresponding to the sensor data is in a selected state, and if so, retaining the sensor data; if not, filtering the sensor data; for each retained sensor data, determining the target identifier and color corresponding thereto; and displaying the target identifier in the target rectangle in the color, thereby completing the drawing of the aerial view; wherein the target rectangle is obtained by applying a rotation transformation method to the initial rectangle.

6. The method of claim 5, wherein, After drawing an initial rectangle representing the position of the collection vehicle according to the size of the collection vehicle, the method further comprises: setting a grid line spacing, and drawing vertical and horizontal grid lines based on a loop algorithm and the grid line spacing.

7. The method of claim 1, wherein, The constructing an information list for displaying data in the aerial view comprises: obtaining a target list corresponding to the aerial view; Circulating each target of the target list, judging whether the data source corresponding to the target is checked, if checked, creating a new row and filling the data from the data source of the target.

8. An autonomous driving multi-sensor data processing system, characterized by, The method comprises the following steps: The first determination module is configured to perform uniform formatting processing on each kind of sensor data, and determine a minimum timestamp and a maximum timestamp of all sensor data based on the formatted sensor data; The filtering module is configured to filter sensor data satisfying a preset condition from all sensor data based on the minimum timestamp and the maximum timestamp; The second determination module is configured to add the sensor data satisfying the preset condition to a thread-safe queue, and determine actual frame data based on data removal processing on the thread-safe queue; The display module is configured to draw a bird's eye view based on the actual frame data, and construct an information list for displaying data in the bird's eye view.

9. An electronic device, comprising: The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7. ​ ​ 10. A computer readable medium having stored thereon a computer program, characterized in that ​

Images