Automatic driving data processing method and device and upper computer

Abstract

The invention relates to the technical field of automatic driving, and provides an automatic driving data processing method and device and an upper computer. The method comprises the following steps: acquiring automatic driving data; determining whether operation of the intelligent driving controller is abnormal or not and whether data transmission between the intelligent driving controller and a millimeter wave radar, a laser radar, a camera, a combined inertial navigation system, a vehicle body chassis sensor and an automatic driving system is abnormal or not based on the automatic driving data; target data corresponding to the target event are determined from the automatic driving data, the target data are extracted, a target file is generated, and the target file is used for analyzing the target event through a preset algorithm and performing scene reproduction on a scene corresponding to the target event. According to the embodiment of the invention, multiple functions related to automatic driving development testing are integrated, potential problems of controller software and algorithms can be quickly positioned, and the real vehicle testing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving data processing method, device and host computer. Background Art

[0002] During the development and debugging of autonomous driving functions, developers often need to control a variety of information in real time, such as the operating status of the controller, the raw data input by the sensor, and the data obtained after the raw data is sensed and fused in the controller, so as to develop and debug various functions of autonomous driving based on the above information. At present, most of the process links of autonomous driving development and debugging need to be completed manually by developers, which is labor-intensive and inefficient. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the embodiments of the present invention provide an automatic driving data processing method, device and host computer.

[0004] The present invention is achieved through the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides an autonomous driving data processing method, including:

[0006] Acquire autonomous driving data, where the autonomous driving data includes millimeter wave radar data, lidar data, camera data, combined inertial navigation data, vehicle chassis data, intelligent driving controller status data, and autonomous driving system data;

[0007] Determine whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system;

[0008] Target data corresponding to a target event is determined from the autonomous driving data, the target data is extracted and a target file is generated, wherein the target file is used to analyze the target event through a preset algorithm and to reproduce a scene corresponding to the target event.

[0009] The above-mentioned autonomous driving data processing method, after obtaining the autonomous driving data, determines whether there is any abnormality in the operation of the intelligent driving controller, or whether there is any abnormality in the data transmission between the intelligent driving controller and the millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system based on the autonomous driving data. It can also determine the target data corresponding to the target event based on the autonomous driving data, extract the target data and generate a target file. Through the target file, the target event can be analyzed, and the scene corresponding to the target event can be reproduced. It integrates multiple functions involved in autonomous driving development and testing into one, and can quickly locate potential problems in controller software and algorithms, thereby improving the efficiency of actual vehicle testing.

[0010] Based on the first aspect, in some embodiments, after acquiring the autonomous driving data sent by the intelligent driving controller, the autonomous driving data processing method includes:

[0011] Add a homologous timestamp to each data in the received autonomous driving data.

[0012] Based on the first aspect, in some embodiments, the autonomous driving data processing method includes:

[0013] If the scene corresponding to the target event cannot be reproduced according to the target file, the target data is printed and visually rendered to determine the cause of the loss of control of the autonomous driving;

[0014] Optimizing the algorithm in the intelligent driving controller based on the cause of the loss of control of the autonomous driving;

[0015] The target data is transmitted back to the intelligent driving controller, so that the intelligent driving controller processes the target data through an optimized algorithm and reproduces the scene of the processed target data again.

[0016] Based on the first aspect, in some embodiments, the autonomous driving data processing method includes:

[0017] Visualize the autonomous driving data and print target information in the autonomous driving data, wherein the target information includes location attributes and self attributes of the target, wherein the location attributes include at least one of azimuth, lateral and longitudinal distances, horizontal height, lane, and CIPV of the nearest vehicle on the path, and the self attributes include at least one of length, width, height, lateral and longitudinal speeds, and direction;

[0018] The target information in the laser radar data is taken as the true value, and the millimeter wave radar and the camera are calibrated according to the target information in the millimeter wave radar data and the camera data.

[0019] Based on the first aspect, in some embodiments, determining whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, the laser radar, the camera, the combined inertial navigation, the body chassis sensor and the autonomous driving system includes:

[0020] When at least one of whether the camera can capture images normally, the operating status of the millimeter-wave radar, the operating status of the ultrasonic radar, the operating status of the combined inertial navigation, the CAN bus load rate, the SPI load rate and packet loss rate, and the ETH packet loss rate and load rate is abnormal, it is determined that the operation of the intelligent driving controller is abnormal, or the data transmission between the intelligent driving controller and the millimeter-wave radar, the laser radar, the camera, the combined inertial navigation, the body chassis sensor and the automatic driving system is abnormal;

[0021] Visualize abnormal information;

[0022] Receive new version data burned by the user, and send the new version data to the intelligent driving controller, so that the intelligent driving controller performs version update.

[0023] Based on the first aspect, in some embodiments, determining target data corresponding to a target event from the autonomous driving data, extracting the target data and generating a target file includes:

[0024] If target motion data is detected, recording a time period corresponding to the target motion data, the target motion data being vehicle motion data determined based on the target event;

[0025] using data in the autonomous driving data corresponding to the time period as the target data, and marking the target data with a target event;

[0026] extracting the target data from the autonomous driving data;

[0027] The target data is used to generate a first target file and a second target file, wherein the first target file is used to analyze the target event through a preset algorithm, and the second target file is used to reproduce the scene through an autonomous driving scene simulation tool.

[0028] Based on the first aspect, in some embodiments, the autonomous driving data processing method includes:

[0029] Print the key information of the entire process of autonomous driving data processing and output a log file in TXT format. The key information includes: whether the SOCKET connection is successful, whether the CAN data is received normally, whether the log is printed successfully, and whether the file is read successfully.

[0030] In a second aspect, an embodiment of the present invention provides an autonomous driving data processing device, including:

[0031] An acquisition module is used to acquire the autonomous driving data sent by the intelligent driving controller, wherein the autonomous driving data includes millimeter wave radar data, lidar data, camera data, combined inertial navigation data, vehicle chassis data, intelligent driving controller status data and autonomous driving system data;

[0032] an abnormality determination module, used to determine whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, laser radar, camera, combined inertial navigation, body chassis sensor and autonomous driving system;

[0033] A generation module is used to determine target data corresponding to a target event from the autonomous driving data, extract the target data and generate a target file, wherein the target file is used to analyze the target event through a preset algorithm and reproduce the scene corresponding to the target event.

[0034] In a third aspect, an embodiment of the present invention provides a host computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the autonomous driving data processing method as described in any one of the first aspects is implemented.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing autonomous driving data as described in any one of the first aspects is implemented.

[0036] In a fifth aspect, an embodiment of the present invention provides a computer program product. When the computer program product runs on a host computer, the host computer executes the autonomous driving data processing method described in any one of the first aspects above.

[0037] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A schematic diagram of a flow chart of an autonomous driving data processing method provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of an autonomous driving data processing device provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the structure of a host computer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0043] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0044] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0045] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0046] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0048] Figure 1 A schematic flow chart of an automatic driving data processing method provided by an embodiment of the present invention. Figure 1 , the details of the autonomous driving data processing method are as follows:

[0049] Step 101, obtaining autonomous driving data.

[0050] Among them, autonomous driving data may include millimeter-wave radar data, lidar data, camera data, combined inertial navigation data, vehicle body and chassis data, intelligent driving controller status data and autonomous driving system data.

[0051] In this embodiment, the intelligent driving controller is connected to the millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor and automatic driving system. Based on this, the status data of the intelligent driving controller may include: whether the camera can take pictures normally, the operating status of the millimeter-wave radar, the operating status of the ultrasonic radar, the operating status of the combined inertial navigation, the CAN bus load rate, the SPI (Serial Peripheral Interface) load rate and packet loss rate, and the ETH packet loss rate and load rate and other information.

[0052] For example, millimeter-wave radar data, vehicle chassis data, combined inertial navigation data and Mobileye Q4 data are all CAN bus data and can be transmitted through the CAN bus. Therefore, millimeter-wave radar data, vehicle chassis data, combined inertial navigation data and Mobileye Q4 data can be received through a USB-CAN card and transmitted to a host computer.

[0053] For example, lidar data, camera data and intelligent driving controller status data are all SOCKET data and can be transmitted to the host computer via Ethernet.

[0054] In some embodiments, after step 101, the above-mentioned autonomous driving data processing method may include a step of time synchronization of the autonomous driving data. Specifically, a homologous timestamp may be added to each data in the received autonomous driving data.

[0055] For example, autonomous driving data involves multiple data such as millimeter-wave radar data, lidar data, camera data, combined inertial navigation data, body chassis data, and intelligent driving controller status data. Each type of data is generated by a corresponding hardware system, so the various data usually do not have a common timestamp. Therefore, all autonomous driving data need to be time synchronized.

[0056] For example, a vehicle usually has multiple millimeter-wave radars, at least one lidar, multiple cameras (such as one front-view camera and four surround-view cameras), an autonomous driving system (such as Mobileye Q4), a body chassis, an integrated inertial navigation system (INS), and an intelligent driving controller. The timestamps of the data generated by the above hardware systems are not universal, so the same timestamp can be added to the data sent by each hardware system to ensure the consistency of various data times, so as to facilitate the subsequent processing of autonomous driving data.

[0057] Step 102, based on the autonomous driving data, determines whether there is any abnormality in the operation of the intelligent driving controller, and whether there is any abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system.

[0058] In some embodiments, step 102 may include:

[0059] When at least one of the following is abnormal: whether the camera can normally acquire images, the operating status of the millimeter-wave radar, the operating status of the ultrasonic radar, the operating status of the combined inertial navigation, the load rate of the CAN bus, the load rate and packet loss rate of the SPI, and the packet loss rate and load rate of the ETH, it is determined that the operation of the intelligent driving controller is abnormal, or that the data transmission between the intelligent driving controller and the millimeter-wave radar, the laser radar, the camera, the combined inertial navigation, the body chassis sensor, and the autonomous driving system is abnormal;

[0060] Visualize abnormal information;

[0061] Receive the new version data burned by the user, and send the new version data to the intelligent driving controller, so that the intelligent driving controller performs version update.

[0062] For example, when the state data of the intelligent driving controller in the autonomous driving data is inconsistent with the normal state, the controller developer can be informed through the visualization window that there are certain problems between the intelligent driving controller and the millimeter wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system (such as abnormal data transmission), or there are abnormalities in the operation of the intelligent driving controller (such as abnormal CAN bus input and output or SPI input and output inside the intelligent driving controller), and the controller developer will troubleshoot the problem. When the controller developers find the problem, they need to modify the controller development program, and then use the version burning tool of the host computer (such as the YDU_Tool_Box tool installed in the host computer) to burn the modified controller development program into the intelligent driving controller for version iteration optimization.

[0063] For example, hardware testers can use the system monitoring module to monitor the controller's operating status in real time, troubleshoot hardware problems, optimize iterative versions, burn new versions through YDU_Tool_Box, output version logs, and feed back version logs to hardware developers and software developers at the same time. Software developers use the perception monitoring module to view data links and feedback data link or data packet loss problems to hardware developers. After that, hardware developers iterate and update versions, output version logs, and feed back to hardware developers and software developers at the same time.

[0064] Specifically, YDU_Tool_Box and algorithm developers (i.e. the aforementioned software developers) are data receivers, and ROS2 is used as the middleware to transmit various data inside the intelligent driving control controller. Therefore, the data source of YDU_Tool_Box and algorithm developers is consistent. Since the algorithm developers are not clear about the operation process of the data inside the intelligent driving control controller, it is necessary to use the YDU_Tool_Box visual interface to confirm whether there is a problem with the internal data link of the intelligent driving control controller, such as the data on the MCU side cannot be sent to the SOC side, or the data sent from the MCU side to the SOC side has packet loss, etc. If such a problem exists, it needs to be fed back to the hardware developers to troubleshoot the corresponding problems and perform version iteration after successful resolution.

[0065] As an example, YDU_Tool_Box can integrate the version burning tool in a jump mode. After the version burning is successful, YDU_Tool_Box can also automatically write information such as the historical version number, the current version number, and the updated version time.

[0066] Exemplarily, step 102 has no direct correlation with the aforementioned time synchronization of the autonomous driving data, that is, step 102 can be executed without performing time synchronization on the autonomous driving data, or step 102 can be executed after performing time synchronization on the autonomous driving data.

[0067] Step 103, determining target data corresponding to the target event from the autonomous driving data, extracting the target data and generating a target file, where the target file is used to analyze the target event through a preset algorithm and reproduce the scene corresponding to the target event.

[0068] In some embodiments, step 103 may include: if target action data is detected, recording a time period corresponding to the target action data, the target action data being vehicle action data determined based on a target event; using the data corresponding to the time period in the autonomous driving data as target data, and marking the target data as a target event; extracting the target data from the autonomous driving data; generating a first target file and a second target file from the target data, the first target file being used to analyze the target event using a preset algorithm, and the second target file being used to reproduce the scene using an autonomous driving scene simulation tool.

[0069] As an example, the scene event tag is defined by the output signal of the intelligent driving controller (for example, the scene is judged as a car cutting in by the action characteristics such as the target relative lateral speed is greater than the vehicle, the relative longitudinal speed is greater than the vehicle, the relative longitudinal distance increases, and the relative lateral distance decreases). By analogy, different signal combination inputs are collected by YDU_Tool_Box and marked as different scenes. Scene slicing is based on the scene event tag, and the data before and after the event is copied out and output in two formats, one is a data format recognizable by YDU_Tool_Box, and the other is a data format recognizable by VTD.

[0070] Specifically, during the test, the actual vehicle tester runs YDU_Tool_Box to receive the data after the perception fusion output by the intelligent driving controller. When encountering special scenes (such as the front vehicle cutting in, the vehicle overtaking, etc.), YDU_Tool_Box can automatically mark this period of time and mark the events that occurred during this period of time, and then extract the data within this period of time from the total data to generate files that can be recognized by YDU_Tool_Box and scene files that can be directly used by the autonomous driving scene simulation tool VTD (Virtual Test Drive). The software algorithm team performs algorithm analysis based on the files that can be recognized by YDU_Tool_Box, and the simulation team reproduces the scene based on the scene files that can be directly used by VTD.

[0071] Exemplarily, the autonomous driving data used in step 103 is time-synchronized data.

[0072] In some embodiments, the above-mentioned autonomous driving data processing method may further include: visualizing the autonomous driving data and printing the target information in the autonomous driving data, the target information including the position attributes and self-attributes of the target, the position attributes including at least one of azimuth, lateral and longitudinal distances, horizontal height, lane and CIPV (Closest In Path Vehicle), and the self-attributes including at least one of length, width, height, lateral and longitudinal speed and direction; taking the target information in the lidar data as the true value, and calibrating the millimeter-wave radar and camera according to the target information in the millimeter-wave radar data and the camera data.

[0073] For example, YDU_Tool_Box can provide camera internal calibration debugging function, external calibration compensation function, and external calibration compensation function for sensors such as millimeter-wave radar and Mobileye Q4. Every time the car stops at a fixed position, the same target is detected by the camera, millimeter-wave radar, lidar, and Mobileye Q4, and each outputs the target information. The target information of the lidar is taken as the true value, and the target information of the millimeter-wave radar, the camera, and the Mobileye Q4 are compared with the target information of the lidar, and the millimeter-wave radar, camera, and Mobileye Q4 are calibrated.

[0074] For example, the target information output by the lidar can be used as the true value, and the target information output by the controller's perception fusion algorithm can be used as the value to be verified. The target information output by the perception fusion algorithm is compared with the target information output by the lidar to obtain the gap between the output of the perception fusion algorithm and the true value. The perception fusion algorithm developers can continuously optimize and update the algorithm based on this visualization content and ultimately obtain the optimal algorithm parameters.

[0075] Specifically, the visualization rendering function provided by YDU_Tool_Box can visualize the target attributes and self-attributes output by the camera, millimeter-wave radar, lidar, and Mobileye Q4, and print the target information. During the calibration and parameter adjustment process, the developer compares the calibration results of the camera, millimeter-wave radar, and Mobileye Q4 with the lidar to obtain the optimal calibration parameters, and finally updates the software version and outputs the version log.

[0076] Among them, the camera intrinsic calibration requires images and videos for internal calibration. YDU_Tool_Box provides image acquisition and collection functions, such as 1080P and 720P images and videos. The collected images and videos are input into the camera intrinsic calibration module in YDU_Tool_Box to perform camera intrinsic calibration, and gradually adjust the parameters to complete the calibration.

[0077] For example, the camera internal calibration can be performed without using time-synchronized data, while the millimeter-wave radar and camera external calibration must use time-synchronized data. Because it needs to be compared with the true value system, the timestamp of the true value system data, the timestamp of the millimeter-wave radar data, and the timestamp of the camera radar data must be synchronized before the external parameter calibration can be performed. Otherwise, it is impossible to determine whether the scene occurs at the same time, and it is impossible to determine whether the data collected by all sensors is the data at the time when the scene occurs, and the calibration accuracy cannot be guaranteed.

[0078] In some embodiments, the above-mentioned autonomous driving data processing method may further include: if the scene corresponding to the target event cannot be reproduced according to the target file, data printing and visual rendering of the target data to determine the cause of the autonomous driving loss of control; optimizing the algorithm in the intelligent driving controller based on the cause of the autonomous driving loss of control; transmitting the target data back to the intelligent driving controller, so that the intelligent driving controller processes the target data through the optimized algorithm, and reproduces the scene of the processed target data again.

[0079] Among them, the autonomous driving data used in the above-mentioned scene reproduction is the autonomous driving data after the aforementioned time synchronization.

[0080] Specifically, in the process of intelligent driving research and development, perception fusion algorithms and planning control algorithms always need to be continuously updated and optimized. When certain special scenes or dangerous scenes cannot be reproduced, and the autonomous driving is out of control for some reason, it is necessary to collect the scene data at that time, and then find the cause of the autonomous driving out of control through data printing and visual rendering. After that, the algorithm is optimized, the new software version is upgraded, the software version log is output, and the data collected at that time is injected back into the controller, and then the data is replayed with YDU_Tool_Box. The degree of performance optimization is verified through rendering visualization and printing functions, and debugging is repeated. For example, the sensor data collected at that time and the chassis information are injected back into the controller. At this time, the external input remains unchanged, and the algorithm part changes. It verifies how the test car will drive under the input of the dangerous scene at that time and the optimization of the algorithm. However, one optimization is obviously not enough to make the result perfect, so it is necessary to debug the optimization algorithm repeatedly.

[0081] YDU_Tool_Box provides printing of all signals that can be output by the vehicle body and controller, and provides visual rendering of obstacles and lane lines. After data collection, the previously recorded data can be replayed repeatedly. Based on the visual rendering and printing of the internal data of the vehicle body and controller, the developer can find out the cause of the loss of control of the autonomous driving. For example: During the process of intelligent driving and following the vehicle, the front vehicle gradually brakes to a stop, but the test vehicle does not follow the front vehicle and also brakes. After this set of data is recorded, it will be replayed later. The developer can use the visual rendering of YDU_Tool_Box and the position of the target vehicle in front of the vehicle, as well as the printed body CAN data (such as brake pedal opening, steering wheel angle), and the internal data of the controller (such as whether the brake signal was triggered at the time, whether the target distance was detected to be getting closer, etc.) as a basis to find the cause of the loss of control.

[0082] In this embodiment, YDU_Tool_Box can provide chassis information chart display, lane line and obstacle visualization rendering, lane line and obstacle data printing, obstacle target trajectory prediction rendering, vehicle planned trajectory rendering, vehicle actual trajectory rendering, parking space rendering and other rendering and printing functions, which meet development needs and can improve development efficiency.

[0083] When rendering and printing data visualization, the coordinate systems of all autonomous driving data must be unified because the coordinate systems of the output data from millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor, and controller are inconsistent. Therefore, the output coordinate system must be converted into a unified coordinate system through translation and rotation in advance, such as using the center of the vehicle's rear axle as the origin of the coordinate system.

[0084] YDU_Tool_Box can perform visual rendering, printing, scene event marking, controller status monitoring, and VTD simulation scene file generation in online mode. In offline mode, data re-injection function is added compared to online mode. The data re-injection function mainly re-injects the original data output to YDU_Tool_Box by millimeter-wave radar, laser radar, camera, combined inertial navigation, body chassis sensor and automatic driving system into the intelligent driving controller. The intelligent driving controller will re-output new fusion data according to the fusion algorithm, and then receive it through YDU_Tool_Box, and perform visual rendering, printing, scene event marking, controller status monitoring, and VTD simulation scene file generation.

[0085] The data re-injection function provided by YDU_Tool_Box is designed to re-inject all the sensor and body data collected at that time into the controller. The perception fusion algorithm inside the controller analyzes and processes these data, and then outputs the fused data. The data re-injection function of YDU_Tool_Box provides data input, while the controller fusion module provides data output. The output of the fusion module is sent to the lower computer of YDU_Tool_Box in the controller via ROS2, and then sent to YDU_Tool_Box via the lower computer in the form of SOCKET to achieve a closed data loop.

[0086] In some embodiments, the above-mentioned autonomous driving data processing method may further include: printing key information in the entire process of autonomous driving data processing, and outputting a log file in TXT format, wherein the key information includes: whether the SOCKET connection is successful, whether the CAN data is received normally, whether the log printing is successful, and whether the file reading is successful. Among them, a new log file is generated each time the YDU_Tool_Box runs, and the log printing shows the operation status of the YDU_Tool_Box host computer throughout its life cycle, which is convenient for developers to understand the internal software or data link problems.

[0087] Exemplarily, whether the above-mentioned file reading is successful can be specifically: YDU_Tool_Box will read the files obtained by each sensor during data acquisition, for example, the file related to the CAN message is a ".acs" file, the file related to the Camera is a ".264" file, and the file recording the Camera timestamp is a ".txt" file, etc. If the reading is successful, the corresponding file reading success will be printed, otherwise, the reading failure will be printed.

[0088] The above-mentioned autonomous driving data processing method, after acquiring the autonomous driving data, determines whether there is any abnormality in the operation of the intelligent driving controller, or whether there is any abnormality in the data transmission between the intelligent driving controller and the millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system based on the autonomous driving data. It can also determine the target data corresponding to the target event in the autonomous driving data, extract the target data and generate a target file. Through the target file, the target event can be analyzed, and the scene corresponding to the target event can be reproduced, so as to quickly locate potential problems in the controller software and algorithm and improve the efficiency of actual vehicle testing.

[0089] In the embodiment of the present invention, the host computer software YDU_Tool_Box basically covers the entire process of intelligent driving development, from the real-time monitoring system status and version iteration of hardware development, sensor calibration used for intelligent driving, AI algorithm development, perception fusion algorithm development, simulation scene construction, to the final real vehicle testing link. In the process of intelligent driving research and development, multiple R&D teams need to cooperate with each other, from hardware to software to simulation and finally to the real vehicle. In the entire development chain, version control and data standardization will improve R&D efficiency and reduce R&D costs. YDU_Tool_Box runs through the entire development and testing process. The versions of hardware and software have strict log output, and standardized data formats and open functions of multiple sensors and controllers are provided to the R&D team, which successfully solves the above problems.

[0090] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0091] Corresponding to the autonomous driving data processing method described in the above embodiment, Figure 2 A structural block diagram of an autonomous driving data processing device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0092] See also Figure 2 The autonomous driving data processing device in the embodiment of the present invention may include an acquisition module 201, an abnormality determination module 202 and a generation module 203.

[0093] Among them, the acquisition module 201 is used to obtain the automatic driving data sent by the intelligent driving controller, and the automatic driving data includes millimeter wave radar data, lidar data, camera data, combined inertial navigation data, body chassis data, intelligent driving controller status data and automatic driving system data.

[0094] The abnormality determination module 202 is used to determine whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system.

[0095] The generation module 203 is used to determine the target data corresponding to the target event from the autonomous driving data, extract the target data and generate a target file, wherein the target file is used to analyze the target event through a preset algorithm and reproduce the scene corresponding to the target event.

[0096] Optionally, after acquiring the autonomous driving data sent by the intelligent driving controller, the autonomous driving data processing device may include: a synchronization module, configured to add a homologous timestamp to each data in the received autonomous driving data.

[0097] Optionally, the above-mentioned autonomous driving data processing device may include: an optimization module, which is used to print and visually render the target data to determine the cause of autonomous driving loss of control when the scene corresponding to the target event cannot be reproduced according to the target file; optimize the algorithm in the intelligent driving controller based on the cause of autonomous driving loss of control; and transmit the target data back to the intelligent driving controller, so that the intelligent driving controller processes the target data through the optimized algorithm and reproduces the scene of the processed target data again.

[0098] Optionally, the above-mentioned autonomous driving data processing device may include: a calibration module, used to visualize the autonomous driving data and print the target information in the autonomous driving data, the target information including the position attributes and self-attributes of the target, the position attributes including azimuth, lateral and longitudinal distances, horizontal height, lane and at least one of the nearest vehicle CIPV on the path, the self-attributes including at least one of length, width, height, lateral and longitudinal speed and direction; taking the target information in the lidar data as the true value, and calibrating the millimeter-wave radar and the camera according to the target information in the millimeter-wave radar data and the camera data.

[0099] Optionally, the abnormality determination module 202 can be specifically used to: determine that there is an abnormality in the operation of the intelligent driving controller, or there is an abnormality in the data transmission between the intelligent driving controller and the millimeter-wave radar, lidar, camera, combined inertial navigation, body chassis sensor and automatic driving system when at least one of the abnormalities occurs, such as whether the camera can take pictures normally, the operating status of the millimeter-wave radar, the operating status of the ultrasonic radar, the operating status of the combined inertial navigation, the CAN bus load rate, the SPI load rate and packet loss rate, and the ETH packet loss rate and load rate; visualize the abnormal information; receive the new version data burned by the user, and send the new version data to the intelligent driving controller, so that the intelligent driving controller performs a version update.

[0100] Optionally, the generation module 203 can be specifically used to: when target action data is detected, record a time period corresponding to the target action data, wherein the target action data is vehicle action data determined based on the target event; use the data corresponding to the time period in the autonomous driving data as the target data, and mark the target data as a target event; extract the target data from the autonomous driving data; generate a first target file and a second target file from the target data, wherein the first target file is used to analyze the target event through a preset algorithm, and the second target file is used to reproduce the scene through an autonomous driving scene simulation tool.

[0101] Optionally, the above-mentioned autonomous driving data processing device may include: a key information printing module, which is used to print key information in the entire process of autonomous driving data processing and output a log file in TXT format. The key information includes: whether the SOCKET connection is successful, whether the CAN data is received normally, whether the log is printed successfully, and whether the file is read successfully.

[0102] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0103] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] The present invention also provides a host computer, see Figure 3 The host computer 300 may include: at least one processor 310, a memory 320, and a computer program stored in the memory 320 and executable on the at least one processor 310. When the processor 310 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented, for example: Figure 1Steps 101 to 103 in the illustrated embodiment. Alternatively, when the processor 310 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 2 Functions of modules 201 to 203 are shown.

[0105] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 320, and executed by the processor 310 to complete the present invention. The one or more modules / units may be a series of computer program segments that can complete specific functions, and the program segments are used to describe the execution process of the computer program in the host computer 300.

[0106] Those skilled in the art will understand that Figure 3 It is only an example of a host computer and does not constitute a limitation on the host computer. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0107] The processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0108] The memory 320 may be an internal storage unit of the host computer, or an external storage device of the host computer, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 320 is used to store the computer program and other programs and data required by the host computer. The memory 320 may also be used to temporarily store data that has been output or is to be output.

[0109] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present invention is not limited to only one bus or one type of bus.

[0110] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in each embodiment of the above-mentioned autonomous driving data processing method.

[0111] An embodiment of the present invention provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal implements the steps in each embodiment of the above-mentioned autonomous driving data processing method when executing the computer program product.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the host computer, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0113] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for processing autonomous driving data, characterized in that: include: Acquire autonomous driving data, where the autonomous driving data includes millimeter wave radar data, lidar data, camera data, combined inertial navigation data, vehicle chassis data, intelligent driving controller status data, and autonomous driving system data; Determine whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, lidar, camera, combined inertial navigation, body chassis sensor and autonomous driving system; Target data corresponding to a target event is determined from the autonomous driving data, the target data is extracted and a target file is generated, wherein the target file is used to analyze the target event through a preset algorithm and to reproduce a scene corresponding to the target event.

2. The method for processing autonomous driving data according to claim 1, wherein: After the autonomous driving data sent by the intelligent driving controller is obtained, the autonomous driving data processing method includes: Add a homologous timestamp to each data in the received autonomous driving data.

3. The method for processing autonomous driving data according to claim 2, wherein: The autonomous driving data processing method comprises: If the scene corresponding to the target event cannot be reproduced according to the target file, the target data is printed and visually rendered to determine the cause of the loss of control of the autonomous driving; Optimizing the algorithm in the intelligent driving controller based on the cause of the loss of control of the autonomous driving; The target data is transmitted back to the intelligent driving controller, so that the intelligent driving controller processes the target data through an optimized algorithm and reproduces the scene of the processed target data again.

4. The method for processing autonomous driving data according to claim 2, wherein: The autonomous driving data processing method comprises: Visualize the autonomous driving data and print target information in the autonomous driving data, wherein the target information includes location attributes and self attributes of the target, wherein the location attributes include at least one of azimuth, lateral and longitudinal distances, horizontal height, lane, and CIPV of the nearest vehicle on the path, and the self attributes include at least one of length, width, height, lateral and longitudinal speeds, and direction; The target information in the laser radar data is taken as the true value, and the millimeter wave radar and the camera are calibrated according to the target information in the millimeter wave radar data and the camera data.

5. The method for processing autonomous driving data according to claim 2, wherein: The determining whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, the laser radar, the camera, the combined inertial navigation, the body chassis sensor and the autonomous driving system, includes: When at least one of whether the camera can capture images normally, the operating status of the millimeter-wave radar, the operating status of the ultrasonic radar, the operating status of the combined inertial navigation, the CAN bus load rate, the SPI load rate and packet loss rate, and the ETH packet loss rate and load rate is abnormal, it is determined that the operation of the intelligent driving controller is abnormal, or the data transmission between the intelligent driving controller and the millimeter-wave radar, the laser radar, the camera, the combined inertial navigation, the body chassis sensor and the automatic driving system is abnormal; Visualize abnormal information; Receive new version data burned by the user, and send the new version data to the intelligent driving controller, so that the intelligent driving controller performs version update.

6. The method for processing autonomous driving data according to claim 2, wherein: The step of determining target data corresponding to a target event from the autonomous driving data, extracting the target data and generating a target file includes: If target motion data is detected, recording a time period corresponding to the target motion data, the target motion data being vehicle motion data determined based on the target event; using data in the autonomous driving data corresponding to the time period as the target data, and marking the target data with a target event; extracting the target data from the autonomous driving data; The target data is used to generate a first target file and a second target file, wherein the first target file is used to analyze the target event through a preset algorithm, and the second target file is used to reproduce the scene through an autonomous driving scene simulation tool.

7. The method for processing autonomous driving data according to claim 2, wherein: The autonomous driving data processing method comprises: Print the key information of the entire process of autonomous driving data processing and output a log file in TXT format. The key information includes: whether the SOCKET connection is successful, whether CAN data is received, whether the log is printed successfully, and whether the file is read successfully.

8. An automatic driving data processing device, characterized in that: include: An acquisition module is used to acquire the autonomous driving data sent by the intelligent driving controller, wherein the autonomous driving data includes millimeter wave radar data, lidar data, camera data, combined inertial navigation data, vehicle chassis data, intelligent driving controller status data and autonomous driving system data; an abnormality determination module, used to determine whether there is an abnormality in the operation of the intelligent driving controller based on the autonomous driving data, and whether there is an abnormality in the data transmission between the intelligent driving controller and the millimeter wave radar, laser radar, camera, combined inertial navigation, body chassis sensor and autonomous driving system; A generation module is used to determine target data corresponding to a target event from the autonomous driving data, extract the target data and generate a target file, wherein the target file is used to analyze the target event through a preset algorithm and reproduce the scene corresponding to the target event.

9. A host computer, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images