Message data processing method and device, computer device and storage medium

Abstract

The application relates to a message data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring image messages and radar messages collected in M time frames during vehicle driving; for each time frame, analyzing the image messages in the current time frame to obtain first running data of N1 first obstacles in the current time frame; analyzing the radar messages in the current time frame to obtain second running data of N2 second obstacles in the current time frame; based on the sequence of the M time frames, combining the second running data of the N2 second obstacles in the M time frames to obtain a running data sequence of the N2 second obstacles in the M time frames; and based on the first running data of the N1 first obstacles and the running data sequence of the N2 second obstacles in the M time frames, playing back the data according to the time frames. The method can improve the visual effect of the message data.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a message data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Data is essential for the development of autonomous driving technology. Both vehicle data and sensor data play a crucial role in the development process. How to utilize this data for more efficient research and development has always been a key focus in the field of autonomous driving.

[0003] Vehicle data and sensor data are usually collected in the form of messages. Due to the large amount of data in the messages, the traditional message data is sparse after parsing and saving, making it inconvenient to view the data and resulting in poor visualization. Summary of the Invention

[0004] Therefore, it is necessary to provide a message data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems and improve the visualization effect of message data.

[0005] Firstly, this application provides a message data processing method, including:

[0006] Acquire image and radar messages collected within M time frames during vehicle operation;

[0007] For each time frame, the image message of the current time frame is parsed to obtain the first running data of the N1 first obstacles in the current time frame;

[0008] The radar message in the current time frame is parsed to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operational data of N2 second obstacles in M ​​time frames are combined to obtain the operational data sequence of N2 second obstacles in M ​​time frames.

[0009] Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, the data is replayed according to the time frames; M, N1 and N2 are all natural numbers greater than 1.

[0010] In one embodiment, the radar message in the current time frame is parsed to obtain the second operational data of N2 second obstacles in the current time frame, including:

[0011] The radar message in the current time frame is parsed to obtain the radar data corresponding to the radar message in the current time frame. The radar data includes location information and operation information.

[0012] Based on the location information, N2 second obstacles are identified;

[0013] The second operational data of the N2 second obstacles in the current time frame is determined from the operational information.

[0014] In one embodiment, based on the order of M time frames, the second running data of N2 second obstacles under the M time frames are combined to obtain the running data sequence of N2 second obstacles under the M time frames, including:

[0015] For each time frame, combine the second running data of N2 second obstacles in the current time frame; determine the order of N2 second obstacles, sort them according to the order of N2 second obstacles, and determine the running data sequence of N2 second obstacles in the current time frame.

[0016] In one embodiment, based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, data playback is performed according to the time frames, including:

[0017] For each time frame, based on the first running data of the N1 first obstacles in the current time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen in the current time frame;

[0018] Based on the running data sequence of N2 second obstacles in the current time frame, determine the position and direction of movement of each of the N2 second obstacles on the display screen in the current time frame;

[0019] The positions and directions of motion of N1 first obstacles and N2 second obstacles in the M time frames are displayed on the screen in sequence.

[0020] In one embodiment, based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, data playback is performed according to the time frames, including:

[0021] The image messages within M time frames are parsed to obtain the lane line information within the M time frames;

[0022] Based on lane line information within M time frames, first running data of N1 first obstacles, and running data sequence of N2 second obstacles, data playback is performed according to time frames.

[0023] In one embodiment, the message data processing method further includes:

[0024] Obtain the parsed timestamps corresponding to the N1 first obstacles in the current time frame;

[0025] According to the parsing timestamp order, sort the first running data of N1 first obstacles in the current time frame to obtain the sorted data sequence in the current time frame;

[0026] Write the sorted data sequence from M time frames into an image text file, line by line.

[0027] Secondly, this application also provides a message data processing apparatus, comprising:

[0028] The acquisition module is used to acquire image messages and radar messages collected within M time frames during the vehicle's operation.

[0029] The first parsing module is used to parse the image message of the current time frame for each time frame to obtain the first running data of N1 first obstacles in the current time frame;

[0030] The second parsing module is used to parse the radar message in the current time frame to obtain the second operation data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operation data of N2 second obstacles in M ​​time frames are combined to obtain the operation data sequence of N2 second obstacles in M ​​time frames.

[0031] The playback module is used to play back data according to time frames based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames; M, N1, and N2 are all natural numbers greater than 1.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0033] Acquire image and radar messages collected within M time frames during vehicle operation;

[0034] For each time frame, the image message of the current time frame is parsed to obtain the first running data of the N1 first obstacles in the current time frame;

[0035] The radar message in the current time frame is parsed to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operational data of N2 second obstacles in M ​​time frames are combined to obtain the operational data sequence of N2 second obstacles in M ​​time frames.

[0036] Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, the data is replayed according to the time frames; M, N1 and N2 are all natural numbers greater than 1.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0038] Acquire image and radar messages collected within M time frames during vehicle operation;

[0039] For each time frame, the image message of the current time frame is parsed to obtain the first running data of the N1 first obstacles in the current time frame;

[0040] The radar message in the current time frame is parsed to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operational data of N2 second obstacles in M ​​time frames are combined to obtain the operational data sequence of N2 second obstacles in M ​​time frames.

[0041] Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, the data is replayed according to the time frames; M, N1 and N2 are all natural numbers greater than 1.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0043] Acquire image and radar messages collected within M time frames during vehicle operation;

[0044] For each time frame, the image message of the current time frame is parsed to obtain the first running data of the N1 first obstacles in the current time frame;

[0045] The radar message in the current time frame is parsed to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operational data of N2 second obstacles in M ​​time frames are combined to obtain the operational data sequence of N2 second obstacles in M ​​time frames.

[0046] Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, the data is replayed according to the time frames; M, N1 and N2 are all natural numbers greater than 1.

[0047] The aforementioned message data processing method, apparatus, computer equipment, storage medium, and computer program product acquire image messages and radar messages collected within M time frames during vehicle operation. For each time frame, the image messages and radar messages under the current time frame are parsed to obtain the first running data of N1 first obstacles and the second running data of N2 second obstacles under the current time frame. Based on the order of the M time frames, the second running data of N2 second obstacles under the M time frames are combined to obtain the running data sequence of N2 second obstacles under the M time frames. Since the running data of multiple obstacles collected under each time frame are arranged in the order of the time frames, the data can be played back in the order of the time frames, improving the visualization effect of the message data. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an application environment diagram of a message data processing method in one embodiment;

[0050] Figure 2 This is a flowchart illustrating a message data processing method in one embodiment;

[0051] Figure 3 This is a flowchart illustrating a message data processing method in another embodiment;

[0052] Figure 4 This is a schematic diagram of the data acquisition process in one embodiment;

[0053] Figure 5 This is a schematic diagram of the data processing flow in one embodiment;

[0054] Figure 6 This is a schematic diagram of the data playback process in one embodiment;

[0055] Figure 7 This is a structural block diagram of a message data processing device in one embodiment;

[0056] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The message data processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with image acquisition device 104 and radar 106 installed in the vehicle. Image acquisition device 104 is used to acquire image messages during vehicle operation. Radar 106 is used to acquire radar messages during vehicle operation. Terminal 102 can be deployed independently outside the vehicle or integrated inside the vehicle; for example, terminal 102 can be an electronic control unit (ECU). The message data processing method provided in this application embodiment can be executed by terminal 102. Terminal 102 acquires image messages and radar messages collected within M time frames during vehicle operation; for each time frame, it parses the image messages of the current time frame to obtain the first running data of N1 first obstacles in the current time frame; it parses the radar messages of the current time frame to obtain the second running data of N2 second obstacles in the current time frame; based on the order of the M time frames, it combines the second running data of N2 second obstacles in the M time frames to obtain the running data sequence of N2 second obstacles in the M time frames; based on the first running data of N1 first obstacles in the M time frames and the running data sequence of N2 second obstacles in the M time frames, it performs data playback according to the time frames; M, N1, and N2 are all natural numbers greater than 1. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a message data processing method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0060] Step 202: Acquire image messages and radar messages collected within M time frames during the vehicle's operation.

[0061] Image messages are image signals acquired by image acquisition devices deployed in the vehicle. Radar messages are radar signals acquired by radar devices deployed in the vehicle. For example, the radar device can be a millimeter-wave radar device. Both image and radar messages are typically CAN messages.

[0062] In some embodiments, CANape software can also be used as the data acquisition tool. The acquired data includes vehicle-specific data and data from various sensors, such as image signals and radar signals. Before data acquisition, DBC files for each signal channel need to be obtained. Based on the DBC files, a CANape data acquisition project is created. In the project, the signal channels of the image acquisition device and the radar device are matched with the CAN channels, and the channel baud rates are set to match the image signals or radar signals, respectively.

[0063] After the CANape data acquisition project is created, image messages and radar messages will be placed into the corresponding recorders according to different signal sources.

[0064] Step 204: For each time frame, parse the image message of the current time frame to obtain the first running data of the N1 first obstacles in the current time frame.

[0065] The process involves reading image messages from the recorder and parsing them using appropriate tools depending on the recorder type. Image message parsing can be done frame-by-frame. For example, if image messages are stored in a DBC file, a DBC file parser can be used to parse the image messages frame-by-frame, obtaining the image for each time frame. Image recognition algorithms are then used to identify targets in the images, resulting in N1 first obstacles and their corresponding first operational data. This first operational data includes position data, velocity data, and direction of movement. Typically, a single image can identify the position data of different obstacles; by analyzing the position data of obstacles from multiple images, the velocity data and direction of movement of the obstacles can be determined. The types of first obstacles include vehicle types, pedestrian types, and roadblock types.

[0066] Step 206: parse the radar message in the current time frame to obtain the second operation data of N2 second obstacles in the current time frame; based on the order of M time frames, combine the second operation data of N2 second obstacles in M ​​time frames to obtain the operation data sequence of N2 second obstacles in M ​​time frames.

[0067] The process involves reading radar messages from the recorder and, depending on the recorder type, using appropriate tools to parse the radar messages. Radar message parsing can be done frame-by-frame, obtaining radar data for each time frame. The radar data includes the position data of N2 second obstacles. Based on the changes in the position data of each second obstacle across multiple time frames, the corresponding velocity data and direction of movement for each second obstacle in those time frames are determined. The second operational data includes position data, velocity data, and direction of movement.

[0068] Each second running data has a corresponding time frame and a second obstacle. The second running data are combined according to the chronological order of the time frames to obtain a running data sequence. The running data sequence is a combination sequence of the second running data for N2 second obstacles in each time frame.

[0069] Step 208: Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, perform data playback according to the time frames; M, N1 and N2 are all natural numbers greater than 1.

[0070] Data playback refers to displaying the operational data of each obstacle on the vehicle's display screen. When playing back data in time frames, the operational data of each obstacle changes with the time frames, and the display screen shows the trend of these changes. The operational data of each obstacle in each time frame includes the first operational data of N1 first obstacles and the operational data sequence of N2 second obstacles.

[0071] In some embodiments, the first running data of N1 first obstacles under M time frames can be combined according to the order of M time frames to obtain a data sequence of N1 first obstacles under M time frames. Based on the data sequence of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles, data playback is performed according to the time frames.

[0072] In the above message data processing method, image messages and radar messages collected within M time frames during vehicle operation are acquired. For each time frame, the image messages and radar messages under the current time frame are parsed to obtain the first running data of N1 first obstacles and the second running data of N2 second obstacles under the current time frame. Based on the order of the M time frames, the second running data of N2 second obstacles under the M time frames are combined to obtain the running data sequence of N2 second obstacles under the M time frames. Since the running data of multiple obstacles collected under each time frame are arranged in the order of the time frames, the data can be played back in the order of the time frames, which improves the visualization effect of the message data.

[0073] In an exemplary embodiment, parsing the radar message in the current time frame to obtain the second operational data of N2 second obstacles in the current time frame includes: parsing the radar message in the current time frame to obtain radar data corresponding to the radar message in the current time frame, the radar data including position information and operational information; determining N2 second obstacles based on the position information; and determining the second operational data of N2 second obstacles in the current time frame from the operational information.

[0074] The radar messages are stored in the recorder as CAN messages. The radar messages are parsed frame by frame according to the CAN message parsing method to obtain the radar data corresponding to the radar message in each time frame.

[0075] The radar data includes the position information of each of the N2 second obstacles. From the position information of the N2 obstacles across multiple time frames, the speed and direction of travel of each obstacle within those time frames can be determined. The travel information includes speed and direction of travel.

[0076] Obstacles to the vehicle are often located in different positions within the environment. Based on multiple positional information within the current time frame, N2 second obstacles are identified. The second operational data consists of the operational data corresponding to each of the N2 second obstacles in the operational information of the current time frame.

[0077] In this embodiment, by parsing the radar message frame by frame, the second operation data corresponding to each of the second obstacles is obtained, which is beneficial for the visualization of the message data.

[0078] In an exemplary embodiment, based on the order of M time frames, the second running data of N2 second obstacles under the M time frames are combined to obtain the running data sequence of N2 second obstacles under the M time frames, including: for each time frame, combining the second running data of N2 second obstacles under the current time frame; determining the order of N2 second obstacles, sorting them according to the order of N2 second obstacles, and determining the running data sequence of N2 second obstacles under the current time frame.

[0079] For each time frame, the second running data corresponding to each of the N2 second obstacles in the current time frame are first combined to obtain the running data combination in the current time frame.

[0080] The order of the N2 second obstacles can be their respective parsing order. Since the positions of the N2 second obstacles are different, the order of the second obstacles parsed from the radar message will be different. The running data combinations in the current time frame are sorted according to the order of the N2 second obstacles to obtain the running data sequence of the N2 second obstacles in the current time frame.

[0081] Using this method, the running data sequence of N2 second obstacles is obtained over M time frames.

[0082] In some embodiments, the running data sequence of the N2 second obstacles in the current time frame can be written into a line of a file, so that each line of the file includes the running data sequence of a time frame, which is beneficial for reading the running data arranged in the order of time frames from the file.

[0083] In this embodiment, by combining and sorting the second running data within each time frame, a running data sequence arranged in the order of time frames is obtained, which is beneficial for visually displaying the changing trend of the data in subsequent data playback.

[0084] In an exemplary embodiment, based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, data playback is performed according to the time frames, including: for each time frame, determining the position and movement direction of each of the N1 first obstacles on the display screen according to the first running data of the N1 first obstacles in the current time frame; determining the position and movement direction of each of the N2 second obstacles on the display screen according to the running data sequence of the N2 second obstacles in the current time frame; and sequentially displaying the positions and movement directions of the N1 first obstacles and the N2 second obstacles in the M time frames on the display screen according to the chronological order of the M time frames.

[0085] The first running data includes position data. For each time frame, the position data from the first running data of N1 first obstacles is converted to the display screen, obtaining the position of each of the N1 first obstacles on the display screen. From the current time frame and the positions of the N1 first obstacles on the display screen in the next adjacent time frame, the running direction of the current time frame can be obtained.

[0086] According to the order of M time frames, N1 first obstacles are marked on the display screen in each time frame, so as to display the markings of N1 first obstacles and show the position and running direction of each of the N1 first obstacles.

[0087] The same method can be used to display N2 second obstacles in M ​​time frames on the screen, as well as the positions and directions of movement of the N2 second obstacles.

[0088] In some embodiments, the first obstacle and the second obstacle can be displayed separately on the display screen. For example, they can be distinguished by color, different shapes, or other methods.

[0089] In this embodiment, by taking the running data of each obstacle for each time frame, the position and direction of movement of each obstacle on the display screen are determined, and the position and direction of movement of each obstacle are displayed on the display screen respectively. This data playback of each obstacle in the order of time frames improves the visualization effect of the message data.

[0090] In an exemplary embodiment, based on the first and second running data sequences of N1 first obstacles over M time frames, data playback is performed according to the time frames, including:

[0091] The image messages within M time frames are parsed to obtain the lane line information within the M time frames;

[0092] Based on lane line information within M time frames, first running data of N1 first obstacles, and running data sequence of N2 second obstacles, data playback is performed according to time frames.

[0093] This process involves extracting various image features from the image message, identifying lane line features, and converting these features into lane line information. The lane line information includes the number of lane lines, lane line direction, and lane line intersections.

[0094] In some embodiments, the acquired images are played back on the display screen, and for each time frame, lane lines are drawn on the display screen based on lane line information. Simultaneously, the display screen plays back the first running data sequences of N1 first obstacles and the running data sequences of N2 second obstacles.

[0095] In this embodiment, by parsing the lane line information in the image message, the lane line information, the first running data of N1 first obstacles, and the running data sequence of N2 second obstacles are simultaneously played back on the display screen, which helps to improve the visualization effect of the message data.

[0096] In one exemplary embodiment, the message data processing method further includes:

[0097] Obtain the parsed timestamps corresponding to the N1 first obstacles in the current time frame;

[0098] According to the parsing timestamp order, sort the first running data of N1 first obstacles in the current time frame to obtain the sorted data sequence in the current time frame;

[0099] Write the sorted data sequence from M time frames into an image text file, line by line.

[0100] Here, the parsing timestamp is the output timestamp corresponding to each of the N1 first obstacles when parsing the image message. In each time frame, the first running data of the N1 first obstacles are sorted according to the order of the parsing timestamps to obtain the sorted data sequence for the current time frame.

[0101] Write the sorted data sequence of the current time frame into a line in the image text file, and then write the sorted data sequences of the M time frames into the image text file line by line, according to the order of the M time frames.

[0102] In this embodiment, the first running data of the first obstacle is sorted according to the parsing timestamps of each first obstacle, so that the sorted data sequence under the same time frame is written into a line of the image text file. Since the running data after parsing the message data is written into the same text line according to the time frame, it is beneficial to improve the visualization effect of the message data.

[0103] To illustrate the message data processing method and its effects in this solution in detail, a specific embodiment is provided below:

[0104] The message data processing method is applied to scenarios involving the acquisition, processing, and playback of message data from vehicle-mounted sensors. For example... Figure 3 The diagram shown illustrates a message data processing method in one embodiment. The message data processing method includes the following steps:

[0105] Step 1: Data Collection

[0106] Data acquisition includes vehicle data acquisition and sensor data acquisition, with sensors including cameras and millimeter-wave radar. For example... Figure 4 The diagram illustrates a data acquisition process in one embodiment. The acquisition tool can be CANape (a system for measuring, calibrating, diagnosing, and developing electronic devices) software. Before data acquisition, dbc files (database files describing CAN communication information between all electronic nodes on the network) for each signal are required. Based on the dbc files, a CANape data acquisition project is created. In the project, each data channel is matched with the CAN (Controller Area Network) channel, and the channel baud rate is set to match the sensor signal or vehicle body signal.

[0107] After the project is created, the signals to be collected are selected, and these signals are placed into different recorders according to their different sources. Simultaneously, to facilitate data playback, a video acquisition channel is added to the project to capture vehicle driving scenes.

[0108] Once data acquisition begins, the acquired data will be transmitted to the terminal via Ethernet. For example, the data will be saved in the MDF (Mirror Disc File) standard format, and the video will be saved in the AVI (Audio Video Interleaved) format.

[0109] Step 2: Data Processing

[0110] like Figure 5 The diagram illustrates the data processing flow in one embodiment. Image messages captured by the camera and radar messages captured by the millimeter-wave radar are both CAN signals. These are parsed into readable signals using a dbc file. However, because the CAN messages have different transmission timestamps, each timestamp occupies a separate row, resulting in a large-dimensional and sparse data table, making data viewing inconvenient. Therefore, data simplification is required. The processed file format can be CSV (Comma-Separated Values) files. Simultaneously, the processed data can be used as input for autonomous driving software simulation to evaluate the software's quality.

[0111] (1) Processing of vehicle data and camera data

[0112] The processing of vehicle data and camera data is similar. Both types of data are sent frame by frame. Therefore, during data processing, data within the same frame time needs to be processed into the same row of data. The data is unified based on the start time of each frame to make the data easy to view.

[0113] (2) Millimeter-wave radar data processing

[0114] The principle of millimeter-wave radar transmitting data is different from that of a camera. Millimeter-wave radar detects information about multiple obstacles that share the same message. Therefore, in data processing, the information about multiple obstacles in the same radar message frame must be split and then recombined to unify the time of multiple obstacle information to the start time of each frame.

[0115] Step 3: Data Replay

[0116] like Figure 6The diagram illustrates a data playback process in one embodiment. Data playback includes two aspects: first, the playback of the acquired data; and second, the playback of the software's output data. For the acquired data, there are two playback methods: the first is to directly play back the MDF data acquired by CANape; the second is to regenerate an MDF file from the processed CSV file for playback. The first method is standalone data playback, while the second method can be combined with the software's output data, converting both into a single MDF file. During data playback, the input and output data can be compared to evaluate the completeness of the software's functionality.

[0117] During data playback, visualization is added. Multiple objects are created in the CANape project, each associated with the position coordinates of an obstacle. These objects are then displayed in a bird's-eye view, changing in real time as the data playback progresses. This allows you to see the movement of each obstacle in the real scene. Lane line information can also be added to the bird's-eye view to help software developers evaluate and estimate the actual scene.

[0118] The aforementioned message data processing method acquires image and radar messages collected within M time frames during vehicle operation. For each time frame, the image and radar messages are parsed to obtain the first operational data of N1 first obstacles and the second operational data of N2 second obstacles. Based on the order of the M time frames, the second operational data of the N2 second obstacles are combined to obtain the operational data sequence of the N2 second obstacles across the M time frames. Since the operational data of multiple obstacles collected in each time frame are arranged in time frame order, data playback can be performed sequentially, improving the visualization effect of the message data. Furthermore, remote transmission is used during data acquisition, avoiding excessive manpower and resources. The data processing reduces data complexity, allowing the data to be used as input for autonomous driving simulation software. Visualization processing is added during data playback, presenting the data more intuitively. The entire system deeply mines the value of the data, which is beneficial for the development of autonomous driving software.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] Based on the same inventive concept, this application also provides a message data processing apparatus for implementing the message data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more message data processing apparatus embodiments provided below can be found in the limitations of the message data processing method described above, and will not be repeated here.

[0121] In one exemplary embodiment, such as Figure 7 As shown, a message data processing device 100 is provided, including: an acquisition module 120, a first parsing module 140, a second parsing module 160, and a playback module 180, wherein:

[0122] The acquisition module 120 is used to acquire image messages and radar messages collected within M time frames during the vehicle's operation.

[0123] The first parsing module 140 is used to parse the image message of the current time frame for each time frame to obtain the first running data of N1 first obstacles in the current time frame;

[0124] The second parsing module 160 is used to parse the radar message in the current time frame to obtain the second operation data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operation data of N2 second obstacles in M ​​time frames are combined to obtain the operation data sequence of N2 second obstacles in M ​​time frames.

[0125] The playback module 180 is used to play back data according to time frames based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames; M, N1 and N2 are all natural numbers greater than 1.

[0126] The above-mentioned message data processing method acquires image messages and radar messages collected within M time frames during vehicle operation. For each time frame, the image messages and radar messages of the current time frame are parsed to obtain the first running data of N1 first obstacles and the second running data of N2 second obstacles in the current time frame. Based on the order of the M time frames, the second running data of N2 second obstacles in the M time frames are combined to obtain the running data sequence of N2 second obstacles in the M time frames. Since the running data of multiple obstacles collected in each time frame are arranged in the order of the time frames, the data can be played back in the order of the time frames, which improves the visualization effect of the message data.

[0127] In one embodiment, the radar message in the current time frame is parsed to obtain the second operating data of N2 second obstacles in the current time frame. The second parsing module 160 is further configured to: parse the radar message in the current time frame to obtain the radar data corresponding to the radar message in the current time frame, the radar data including position information and operating information; determine the N2 second obstacles based on the position information; and determine the second operating data of the N2 second obstacles in the current time frame from the operating information.

[0128] In one embodiment, based on the order of M time frames, the second running data of N2 second obstacles under the M time frames are combined to obtain the running data sequence of N2 second obstacles under the M time frames. The second parsing module 160 is further configured to: for each time frame, combine the second running data of N2 second obstacles under the current time frame; determine the order of N2 second obstacles, sort them according to the order of N2 second obstacles, and determine the running data sequence of N2 second obstacles under the current time frame.

[0129] In one embodiment, based on the first running data of N1 first obstacles in M ​​time frames and the running data sequence of N2 second obstacles in M ​​time frames, data playback is performed according to the time frames. The playback module 180 is further configured to: for each time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen according to the first running data of the N1 first obstacles in the current time frame; determine the position and direction of movement of each of the N2 second obstacles on the display screen according to the running data sequence of the N2 second obstacles in the current time frame; and sequentially display the position and direction of movement of the N1 first obstacles and the N2 second obstacles in the M time frames on the display screen according to the order of the M time frames.

[0130] In one embodiment, based on the first running data of N1 first obstacles and the running data sequence of N2 second obstacles under M time frames, data playback is performed according to the time frames. The playback module 180 is further configured to: parse the image messages within the M time frames respectively to obtain lane line information within the M time frames; and perform data playback according to the time frames based on the lane line information within the M time frames, the first running data of N1 first obstacles, and the running data sequence of N2 second obstacles.

[0131] In one embodiment, the message data processing device 100 further includes a writing module, which is further configured to: obtain the parsing timestamps corresponding to each of the N1 first obstacles in the current time frame; sort the first running data of the N1 first obstacles in the current time frame according to the order of the parsing timestamps to obtain the sorted data sequence in the current time frame; and write the sorted data sequence in the M time frames into an image text file line by line.

[0132] Each module in the aforementioned message data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0133] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a message data processing method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0134] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0136] Acquire image and radar messages collected within M time frames during vehicle operation; for each time frame, parse the image message of the current time frame to obtain the first operational data of N1 first obstacles in the current time frame; parse the radar message of the current time frame to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of the M time frames, combine the second operational data of N2 second obstacles in the M time frames to obtain the operational data sequence of N2 second obstacles in the M time frames; based on the first operational data of N1 first obstacles in the M time frames and the operational data sequence of N2 second obstacles in the M time frames, perform data playback according to the time frames; M, N1, and N2 are all natural numbers greater than 1.

[0137] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0138] The radar message in the current time frame is parsed to obtain the radar data corresponding to the radar message in the current time frame. The radar data includes position information and operation information. Based on the position information, N2 second obstacles are identified. In the operation information, the second operation data of the N2 second obstacles in the current time frame are identified.

[0139] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0140] For each time frame, combine the second running data of N2 second obstacles in the current time frame; determine the order of N2 second obstacles, sort them according to the order of N2 second obstacles, and determine the running data sequence of N2 second obstacles in the current time frame.

[0141] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0142] For each time frame, based on the first running data of N1 first obstacles in the current time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen in the current time frame; based on the running data sequence of N2 second obstacles in the current time frame, determine the position and direction of movement of each of the N2 second obstacles on the display screen in the current time frame; according to the chronological order of M time frames, display the positions and directions of movement of the N1 first obstacles and the N2 second obstacles in the M time frames on the display screen in sequence.

[0143] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0144] The image messages within M time frames are parsed to obtain lane line information within the M time frames. Based on the lane line information within the M time frames, the first running data of N1 first obstacles, and the running data sequence of N2 second obstacles, the data is replayed according to the time frames.

[0145] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0146] Obtain the parsing timestamps corresponding to the N1 first obstacles in the current time frame; sort the first running data of the N1 first obstacles in the current time frame according to the order of the parsing timestamps to obtain the sorted data sequence in the current time frame; write the sorted data sequence in the M time frames into the image text file line by line.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0148] Acquire image and radar messages collected within M time frames during vehicle operation; for each time frame, parse the image message of the current time frame to obtain the first operational data of N1 first obstacles in the current time frame; parse the radar message of the current time frame to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of the M time frames, combine the second operational data of N2 second obstacles in the M time frames to obtain the operational data sequence of N2 second obstacles in the M time frames; based on the first operational data of N1 first obstacles in the M time frames and the operational data sequence of N2 second obstacles in the M time frames, perform data playback according to the time frames; M, N1, and N2 are all natural numbers greater than 1.

[0149] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0150] The radar message in the current time frame is parsed to obtain the radar data corresponding to the radar message in the current time frame. The radar data includes position information and operation information. Based on the position information, N2 second obstacles are identified. In the operation information, the second operation data of the N2 second obstacles in the current time frame are identified.

[0151] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0152] For each time frame, combine the second running data of N2 second obstacles in the current time frame; determine the order of N2 second obstacles, sort them according to the order of N2 second obstacles, and determine the running data sequence of N2 second obstacles in the current time frame.

[0153] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0154] For each time frame, based on the first running data of N1 first obstacles in the current time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen in the current time frame; based on the running data sequence of N2 second obstacles in the current time frame, determine the position and direction of movement of each of the N2 second obstacles on the display screen in the current time frame; according to the chronological order of M time frames, display the positions and directions of movement of the N1 first obstacles and the N2 second obstacles in the M time frames on the display screen in sequence.

[0155] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0156] The image messages within M time frames are parsed to obtain lane line information within the M time frames. Based on the lane line information within the M time frames, the first running data of N1 first obstacles, and the running data sequence of N2 second obstacles, the data is replayed according to the time frames.

[0157] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0158] Obtain the parsing timestamps corresponding to the N1 first obstacles in the current time frame; sort the first running data of the N1 first obstacles in the current time frame according to the order of the parsing timestamps to obtain the sorted data sequence in the current time frame; write the sorted data sequence in the M time frames into the image text file line by line.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0160] Acquire image and radar messages collected within M time frames during vehicle operation; for each time frame, parse the image message of the current time frame to obtain the first operational data of N1 first obstacles in the current time frame; parse the radar message of the current time frame to obtain the second operational data of N2 second obstacles in the current time frame; based on the order of the M time frames, combine the second operational data of N2 second obstacles in the M time frames to obtain the operational data sequence of N2 second obstacles in the M time frames; based on the first operational data of N1 first obstacles in the M time frames and the operational data sequence of N2 second obstacles in the M time frames, perform data playback according to the time frames; M, N1, and N2 are all natural numbers greater than 1.

[0161] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0162] The radar message in the current time frame is parsed to obtain the radar data corresponding to the radar message in the current time frame. The radar data includes position information and operation information. Based on the position information, N2 second obstacles are identified. In the operation information, the second operation data of the N2 second obstacles in the current time frame are identified.

[0163] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0164] For each time frame, combine the second running data of N2 second obstacles in the current time frame; determine the order of N2 second obstacles, sort them according to the order of N2 second obstacles, and determine the running data sequence of N2 second obstacles in the current time frame.

[0165] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0166] For each time frame, based on the first running data of N1 first obstacles in the current time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen in the current time frame; based on the running data sequence of N2 second obstacles in the current time frame, determine the position and direction of movement of each of the N2 second obstacles on the display screen in the current time frame; according to the chronological order of M time frames, display the positions and directions of movement of the N1 first obstacles and the N2 second obstacles in the M time frames on the display screen in sequence.

[0167] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0168] The image messages within M time frames are parsed to obtain lane line information within the M time frames. Based on the lane line information within the M time frames, the first running data of N1 first obstacles, and the running data sequence of N2 second obstacles, the data is replayed according to the time frames.

[0169] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0170] Obtain the parsing timestamps corresponding to the N1 first obstacles in the current time frame; sort the first running data of the N1 first obstacles in the current time frame according to the order of the parsing timestamps to obtain the sorted data sequence in the current time frame; write the sorted data sequence in the M time frames into the image text file line by line.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0172] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A message data processing method, characterized in that, The method includes: Acquire image and radar messages collected within M time frames during vehicle operation; For each time frame, the image message of the current time frame is parsed to obtain the first running data of N1 first obstacles in the current time frame; The radar message in the current time frame is parsed to obtain the second operation data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operation data of N2 second obstacles in M ​​time frames are combined to obtain the operation data sequence of N2 second obstacles in M ​​time frames. Based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames, data playback is performed according to time frames; where M, N1, and N2 are all natural numbers greater than 1; The method of combining the second operational data of N2 second obstacles under M time frames in sequence to obtain the operational data sequence of N2 second obstacles under M time frames includes: For each time frame, the second running data of N2 second obstacles in the current time frame are combined; the order of N2 second obstacles is determined, and they are sorted according to the order of N2 second obstacles to determine the running data sequence of N2 second obstacles in the current time frame.

2. The method according to claim 1, characterized in that, The step of parsing the radar message in the current time frame to obtain the second operational data of the N2 second obstacles in the current time frame includes: The radar message in the current time frame is parsed to obtain the radar data corresponding to the radar message in the current time frame. The radar data includes location information and operation information. Based on the location information, N2 second obstacles are identified; The second operational data of the N2 second obstacles in the current time frame are determined from the operational information.

3. The method according to claim 1, characterized in that, The first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames are used for data playback according to time frames, including: For each time frame, based on the first running data of the N1 first obstacles in the current time frame, determine the position and direction of movement of each of the N1 first obstacles on the display screen in the current time frame; Based on the running data sequence of the N2 second obstacles in the current time frame, determine the position and direction of movement of each of the N2 second obstacles on the display screen in the current time frame; The positions and directions of movement of N1 first obstacles and N2 second obstacles in the M time frames are displayed sequentially on the display screen.

4. The method according to claim 1, characterized in that, The first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames are used for data playback according to time frames, including: The image messages within M time frames are parsed to obtain the lane line information within the M time frames; Based on lane line information within M time frames, first running data of N1 first obstacles, and running data sequence of N2 second obstacles, data playback is performed according to time frames.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the parsed timestamps corresponding to the N1 first obstacles in the current time frame; According to the order of the parsed timestamps, the first running data of N1 first obstacles in the current time frame are sorted to obtain the sorted data sequence in the current time frame; Write the sorted data sequence from M time frames into an image text file, line by line.

6. A message data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire image messages and radar messages collected within M time frames during the vehicle's operation. The first parsing module is used to parse the image message of the current time frame for each time frame to obtain the first running data of N1 first obstacles in the current time frame; The second parsing module is used to parse the radar message in the current time frame to obtain the second operation data of N2 second obstacles in the current time frame; based on the order of M time frames, the second operation data of N2 second obstacles in M ​​time frames are combined to obtain the operation data sequence of N2 second obstacles in M ​​time frames. The playback module is used to play back data according to time frames based on the first running data of N1 first obstacles under M time frames and the running data sequence of N2 second obstacles under M time frames; where M, N1, and N2 are all natural numbers greater than 1. The second parsing module is used to combine the second running data of N2 second obstacles in the current time frame for each time frame; determine the order of the N2 second obstacles, sort them according to the order of the N2 second obstacles, and determine the running data sequence of the N2 second obstacles in the current time frame.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images