Data processing method, device and storage medium

Abstract

Embodiments of the present application provide a data processing method and device and a storage medium. In the embodiments of the present application, the original driving environment data and the original vehicle driving route of an autonomous vehicle can be spatio-temporally aligned. According to the spatio-temporally aligned vehicle driving route, a regional map passed through by the spatio-temporally aligned vehicle driving route can be obtained. Then, according to the spatio-temporally aligned driving environment data, the driving environment at the corresponding moment can be rendered on the regional map, realizing the visualization of the regional map and the vehicle driving environment at the corresponding moment, which can provide a visual reference for user perception of multi-dimensional data quality and help the user intuitively perceive the data quality collected by the autonomous vehicle.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus and storage medium. Background Technology

[0002] Autonomous driving has become a development trend in the automotive industry, and research on autonomous driving perception, localization, planning, decision-making, and control is in full swing. To ensure the safety of autonomous vehicles, they must undergo millions of kilometers of testing and verification, thus autonomous driving is becoming increasingly reliant on simulation platforms.

[0003] Typically, the system of an autonomous vehicle can be divided into three modules: first, the perception module, which is equivalent to the human eye, collecting the status of the surrounding environment in real time through sensors; second, the planning and decision-making module, which is equivalent to the human brain, planning the driving path and converting the planned path into executable commands such as accelerator, brake, and steering; and third, the control module, which is equivalent to the human hand and foot, used to control the vehicle to perform operations such as accelerator, brake, and steering.

[0004] Existing technologies utilize real-world data for simulating, training, and validating modules of a simulation platform. The quality of this real-world data has a crucial impact on the accuracy of the simulation platform's modules. Therefore, it is essential to ensure that the quality of the data used for training the simulation platform is perceptible to the user. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and storage medium for visualizing real-world data collected by autonomous vehicles, enabling users to intuitively perceive data quality.

[0006] This application provides a data processing method, including: Acquire raw, multi-dimensional data collected by the autonomous vehicle during its operation; the raw, multi-dimensional data includes at least: raw driving environment data and raw vehicle driving route; The original multi-dimensional data is spatiotemporally aligned to obtain spatiotemporally aligned target multi-dimensional data; the target multi-dimensional data includes at least: spatiotemporally aligned driving environment data and spatiotemporally aligned vehicle driving routes. Based on the spatiotemporally aligned vehicle travel route, obtain a map of the area traversed by the spatiotemporally aligned vehicle travel route. Based on the spatiotemporally aligned driving environment data at the corresponding time, the driving environment at the corresponding time is rendered on the regional map; the corresponding time refers to the time corresponding to the timestamp of the spatiotemporally aligned vehicle driving route on the regional map.

[0007] This application also provides a data processing system, including: a data storage node, a data processing node, and a rendering engine; The data storage node is used to store map data and raw multi-dimensional data collected by autonomous vehicles; the raw multi-dimensional data includes at least: raw driving environment data and raw vehicle driving route; The data processing node is used to perform spatiotemporal alignment on the original multidimensional data to obtain spatiotemporally aligned target multidimensional data; the target multidimensional data includes at least: spatiotemporally aligned driving environment data and spatiotemporally aligned vehicle driving routes; based on the spatiotemporally aligned vehicle driving routes, a map of the areas traversed by the spatiotemporally aligned vehicle driving routes is obtained from the data storage system. The rendering engine is used to render the driving environment at a corresponding time on the regional map based on the spatiotemporally aligned driving environment data at the corresponding time; the corresponding time refers to the time corresponding to the timestamp of the spatiotemporally aligned vehicle driving route on the regional map.

[0008] This application embodiment also provides a computing device, including: a memory, a processor, and a display component; wherein, the memory is used to store computer programs; The processor is coupled to the memory and the display component and is used to execute the computer program to perform the steps in the above data processing method.

[0009] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the above-described data processing method.

[0010] In this embodiment, the original driving environment data and original vehicle driving route of the autonomous vehicle are spatiotemporally aligned; and a map of the area traversed by the spatiotemporally aligned vehicle driving route is obtained based on the spatiotemporally aligned vehicle driving route; then, the driving environment at the corresponding time can be rendered on the area map based on the spatiotemporally aligned driving environment data, realizing the visualization of the area map and the vehicle driving environment at the corresponding time, which can provide a visual reference for users to perceive the multi-dimensional data quality and help users intuitively perceive the data quality collected by the autonomous vehicle. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the structure of the data processing system provided in the embodiments of this application; Figure 2 This is a diagram illustrating the working architecture of the data processing system provided in the embodiments of this application. Figure 3 A schematic diagram illustrating the data time alignment method provided in the embodiments of this application; Figure 4 A flowchart illustrating the data processing method provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In this embodiment, in order for users to perceive the quality of the data used for training on the simulation platform, the original driving environment data and the original driving route of the autonomous vehicle can be spatiotemporally aligned; and based on the spatiotemporally aligned driving route, a map of the area traversed by the spatiotemporally aligned driving route can be obtained; then, based on the spatiotemporally aligned driving environment data, the driving environment at the corresponding time can be rendered on the area map, realizing the visualization of the area map and the vehicle driving environment at the corresponding time. This provides a visual reference for users to perceive the multi-dimensional data quality and helps users intuitively perceive the data quality collected by the autonomous vehicle.

[0014] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0015] It should be noted that the same reference numerals denote the same object in the following figures and embodiments. Therefore, once an object is defined in one figure or embodiment, it does not need to be discussed further in subsequent figures and embodiments.

[0016] Figure 1 This is a schematic diagram of the structure of the data processing system provided in an embodiment of this application. Figure 1 As shown, the data processing system mainly includes: data storage node 10, data processing node 20, and rendering engine 30.

[0017] In this embodiment, data storage node 10 refers to a device used for data storage, which can store data in the form of a database. The database can be one or more of object storage (OSS), indexed database, and relational database. "Multiple" refers to two or more types. In this embodiment, data storage node 10 mainly stores data collected by the autonomous vehicle during its operation. It should be noted that in this embodiment, the autonomous driving level of the autonomous vehicle can be any level from L1 to L5.

[0018] Autonomous vehicles are equipped with multiple sensors. "Multiple" refers to two or more types. These sensors include, but are not limited to, image acquisition devices, radar, positioning devices, inertial measurement units (IMUs), odometers, and vehicle state sensors. Image acquisition devices can be visual devices such as monocular cameras, binocular cameras, or depth cameras, while radar includes at least one type such as lidar, millimeter-wave radar, or microwave radar. Both image acquisition devices and radar can be mounted on autonomous vehicles.

[0019] Autonomous vehicles can collect data from multiple dimensions using onboard sensors during operation; this is referred to as multi-dimensional data. "Multi-dimensional" means two or more types of data. For example... Figure 2 As shown, the multi-dimensional data includes, but is not limited to, vehicle driving routes, vehicle status data, and driving environment data.

[0020] The vehicle's driving route can be represented as a time series of positioning points during the vehicle's journey. Vehicle state data refers to data reflecting the vehicle's driving state, including at least one of the following: speed data, throttle data, braking data, steering wheel data, attitude data, inertial measurement unit (IMU) data, and vehicle driving mode. IMU data includes the autonomous vehicle's attitude angles (or angular rates) and acceleration. Vehicle driving environment data refers to environmental data collected during the vehicle's journey, including but not limited to at least one of the following: image data of the environment collected by the image acquisition device on the autonomous vehicle and point cloud data of the environment collected by the radar on the autonomous vehicle. Image data can be a single frame image or continuous video frames.

[0021] Data collected by autonomous vehicles can be defined as raw data; correspondingly, multi-dimensional data can be called raw multi-dimensional data. Raw multi-dimensional data may include one or more of the following: raw driving environment data, raw vehicle state data, and raw vehicle driving route. "Multiple" refers to two or more types. For example... Figure 2 As shown, autonomous vehicles can provide the collected raw, multi-dimensional data to a data processing system. This data processing system can be deployed in the cloud, enabling data to be uploaded to the cloud in autonomous driving scenarios.

[0022] Data processing node 20 refers to a computing device with computing and communication capabilities. This computing device can be a single server device, a cloud-based server array, or a virtual machine (VM) running within a cloud-based server array. Additionally, server-side devices can also refer to other computing devices with corresponding service capabilities, such as computer terminals (running service programs).

[0023] The data storage node 10 and the data processing node 20 can be connected wirelessly or via a wired connection. The wireless connection can be a network connection. For example, the data storage node 10 and the data processing node 20 can be connected via a public or private network. Alternatively, the data storage node 10 and the data processing node 20 can also be connected via a mobile network, and the mobile network standard can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, etc.

[0024] In this embodiment, the autonomous vehicle and the data processing system are connected via network communication. For details on the network connection method, please refer to the network connection method between the data storage node 10 and the data processing node 20 described above. The autonomous vehicle can transmit the collected raw multi-dimensional data to the data processing system via the network, where the data storage node 10 stores the raw multi-dimensional data.

[0025] Data storage node 10 can store and manage raw, multi-dimensional data in a database format. Accordingly, such as... Figure 2 As shown, data processing node 20 can standardize the raw multi-dimensional data and create an index for it. Of course, data processing node 20 can also label the raw multi-dimensional data to determine its data dimensions. Accordingly, the data labels for the raw multi-dimensional data include, but are not limited to: dataset identifier, timestamp, vehicle identifier, and data dimensions.

[0026] In this embodiment, as Figure 2 As shown, in order to improve data security, data processing node 20 can also generate mirror data of the original multi-dimensional data, and data storage node 10 can also store mirror data of the original multi-dimensional data, realizing redundant backup of the original multi-dimensional data, which helps to improve the security of the original multi-dimensional data.

[0027] In practical applications, to ensure the safety of autonomous vehicles, it is necessary to simulate the planning and decision-making modules and control modules of autonomous vehicles on a simulation platform to obtain a model capable of accurately planning, making decisions, and controlling autonomous vehicles. During the actual model simulation training process, the simulation model can be trained using real-world data collected from the autonomous vehicle. Therefore, the quality and accuracy of the data used for simulation model training have a crucial impact on the training efficiency and accuracy of the simulation model. If users can perceive the quality of the data used for simulation module training in advance and promptly identify and remove substandard data from the simulation module's training data, it will undoubtedly improve the efficiency and accuracy of subsequent simulation model training.

[0028] Based on the above analysis, in this embodiment of the application, a data visualization scheme is provided to enable users to intuitively perceive the quality of the training data of the simulation module. Specifically, before visualizing the multi-dimensional data, the data processing node 20 can obtain the raw multi-dimensional data collected by the autonomous vehicle during driving from the data storage node 10. This raw multi-dimensional data includes at least: raw driving environment data and raw vehicle driving route. In some embodiments, the raw multi-dimensional data may also include: raw vehicle state data.

[0029] In this embodiment, the specific implementation of the data processing node 20 obtaining raw multi-dimensional data from the data storage node 10 is not limited. In some embodiments, the user can independently select the target dimension for visualization. For example, a human-computer interaction interface can be provided. This human-computer interaction interface includes a dimension selection option for visualization. The dimensions for visualization include various data dimensions collected by the autonomous vehicle, including but not limited to: radar point clouds, images, videos, and some or all of the dimensions such as vehicle status and vehicle driving route. The user can achieve visualization of the target dimension information by selecting the target dimension for visualization.

[0030] Accordingly, data processing node 20 can obtain the target dimension selected by the user based on the human-computer interaction interface for visualization; data processing node 20 can obtain the raw data of the target dimension from data storage node 10. The target dimension can be any dimension selected by the user. In this embodiment, the visualization method is illustrated by taking the target dimensions including: driving environment dimension, vehicle state dimension, and vehicle driving route dimension as examples.

[0031] Of course, in addition to selecting the target dimension for visualization, users can also set query conditions. In some embodiments, users can provide query conditions through a human-computer interaction interface. Optionally, users can select query conditions or input query conditions through the human-computer interaction interface. Users can set query conditions independently according to their query needs. Query conditions include, but are not limited to, one or more of the following: time conditions, vehicle identification conditions, data dimension conditions, and spatial conditions.

[0032] Accordingly, the data processing node 20 can obtain the query conditions provided by the user based on the human-computer interaction interface; and generate a query expression in a data format supported by the query engine based on the query conditions; further, the query expression can be parsed to obtain a query statement adapted to the query engine; then, the query statement can be used to query in the data storage node 10 to obtain the target dimension data that meets the query conditions.

[0033] This data query method does not require users to input complex query statements. Users only need to input query conditions, and the data processing node 20 can convert the query conditions into query statements adapted to the query engine. This reduces the user's learning cost and helps to improve the flexibility and versatility of the data processing system.

[0034] The query criteria provided by the user can be of any dimension. Accordingly, query criteria include, but are not limited to, one or more of the following: time criteria, vehicle identification criteria, sensor dimension criteria, and spatial criteria. "Multiple" refers to two or more criteria.

[0035] After acquiring the target dimension data that meets the query conditions, the data processing node 20 can classify the target dimension data by vehicle dimension to determine the target dimension data for each autonomous vehicle. The target dimension data can be any data dimension that meets the query conditions. For each autonomous vehicle's target dimension data, the rendering engine 30 can render a corresponding data dashboard. In this embodiment, a data dashboard refers to a form that displays data in an intuitive and visual way.

[0036] In this embodiment, the format of the data dashboard is not limited. Optionally, the format of the data dashboard may include at least one of the following: image dashboard, text dashboard, number dashboard, progress bar dashboard, graphical dashboard, and list dashboard; etc. Among them, graphical dashboards include at least one of the following: scatter plot dashboard, line chart dashboard, line graph dashboard, histogram dashboard, pie chart dashboard, bullet chart dashboard, area chart dashboard, and waterfall chart dashboard; but are not limited to these.

[0037] In this embodiment of the application, the visualization method is illustrated by taking the target dimension data, which includes the driving environment dimension, vehicle state dimension, and vehicle driving route dimension, as examples.

[0038] Considering that the various sensors on autonomous vehicles each have their own coordinate system, meaning that the data collected by each sensor is based on the sensor's own coordinate system and the data is in different spatial domains; on the other hand, even after hardware synchronization, different sensors still have different sampling frequencies, meaning that the data acquired by the sensors are not synchronized in terms of timestamps.

[0039] Spatiotemporal alignment is performed on the original multidimensional data to obtain spatiotemporally aligned target multidimensional data. The spatiotemporally aligned target multidimensional data includes at least: spatiotemporally aligned driving environment data, vehicle state dataset, and vehicle driving routes.

[0040] Specifically, data processing node 20 can perform time alignment on the original multi-dimensional data based on the timestamps of the original multi-dimensional data. Optionally, for any first-dimensional data and second-dimensional data in the original multi-dimensional data, a first target timestamp corresponding to the first-dimensional data and a second target timestamp corresponding to the second-dimensional data can be determined based on the timestamps of the first-dimensional data and the second-dimensional data. Specifically, the first-dimensional data has no corresponding data at the first target timestamp, but the second-dimensional data has data at the first target timestamp; the second-dimensional data has no corresponding data at the second target timestamp, but the first-dimensional data has data at the second target timestamp. For example, if the first-dimensional data is point cloud data and the second-dimensional data is image data, then at a certain timestamp T, the point cloud data has no corresponding point cloud data, but the image data has corresponding image data at that timestamp T. Therefore, timestamp T is determined to be the target timestamp corresponding to the point cloud data, i.e., timestamp T is the first target timestamp corresponding to the first-dimensional data (point cloud data).

[0041] Furthermore, the data processing node 20 can obtain first-dimensional target sampling data whose sampling time is adjacent to the first target timestamp of the first-dimensional data from the first-dimensional data; and interpolate the first-dimensional data at the first target timestamp based on the first-dimensional target sampling data. The first-dimensional target sampling data adjacent to the first target timestamp can be data from the first-dimensional data located at the previous sampling point of the first target timestamp, or data from the first-dimensional data located at the next sampling point of the first target timestamp, or data from two sampling points in the first-dimensional data located before and after the first target timestamp.

[0042] For example, such as Figure 3 As shown, assume the first dimension data are A1-A6; the second dimension data are B1-B4. According to... Figure 3The temporal sequence distribution shown indicates that: the first dimension data has no data at timestamps T3 and T7, while the second dimension data has data points B2 and B4 at timestamps T3 and T7. Therefore, the first target timestamps corresponding to the first dimension data are determined to be timestamps T3 and T7. Similarly, the second dimension data has no data at timestamps T2, T4, T6, and T8, while the first dimension data has data points A2, A3, A5, and A6 at timestamps T2, T4, T6, and T8. Therefore, the second target timestamps corresponding to the second dimension data are determined to be timestamps T2, T4, T6, and T8. Furthermore, for the first dimension data, the adjacent first dimension target sampling data at the target timestamp T3 can be data point A2, or data point A3, or data points A2 and A3.

[0043] Furthermore, in an embodiment where the first-dimensional target sampling data is the data at the sampling point preceding the first target timestamp in the first-dimensional data, the data at the sampling point preceding the target timestamp can be interpolated onto the first target timestamp corresponding to the first-dimensional data, so that the first-dimensional data and the second-dimensional data are aligned on the first target timestamp. For example, for the target timestamp T3 corresponding to the first-dimensional data, data point A2 can be interpolated onto the target timestamp T3, so that the first-dimensional data and the second-dimensional data are aligned on timestamp T3.

[0044] For an embodiment where the first-dimensional target sampling data is the data located at the sampling point following the first target timestamp in the first-dimensional data, the data at the sampling point following the first target timestamp can be interpolated onto the first target timestamp to align the first-dimensional data with the second-dimensional data on the first target timestamp. For example, Figure 3 For the target timestamp T3 corresponding to the first dimension data, the data point A3 can be interpolated onto the target timestamp T3 to align the first dimension data and the second dimension data on timestamp T3.

[0045] For the first-dimensional target sampling data, which consists of two sampling points located before and after the first target timestamp in the first-dimensional data, the mean of the two sampling points can be calculated, and this mean can be interpolated onto the first target timestamp corresponding to the first-dimensional data to align the first-dimensional data with the second-dimensional data at the first target timestamp. For example, Figure 3 The mean values ​​of data points A2 and A3 are interpolated onto the target timestamp T3 to align the first and second dimension data at timestamp T3. This interpolation method is merely illustrative and does not constitute a limitation.

[0046] Similarly, data processing node 20 can also obtain second-dimensional target sampling data whose sampling time is adjacent to the second target timestamp of the second-dimensional data from the second-dimensional data; and interpolate the second-dimensional data on the second target timestamp based on the second-dimensional target sampling data, thereby achieving temporal alignment between the first-dimensional data and the second-dimensional data. For a description of the second-dimensional target sampling data and a detailed implementation method for interpolating the second-dimensional data on the corresponding second target timestamp based on the second-dimensional target sampling data, please refer to the relevant content on the interpolation processing of the first-dimensional data described above, which will not be repeated here.

[0047] In some embodiments, in addition to interpolating the original multi-dimensional data to achieve temporal alignment, as described above, interpolation can also be performed on the time-aligned multi-dimensional data according to the same target sampling period. The target sampling period is shorter than the sampling period of the time-aligned multi-dimensional data. This allows for data alignment at a finer temporal granularity. In this application embodiment, the specific value of the target sampling period is not limited. Optionally, the target sampling period can be a millisecond-level, nanosecond-level, or even finer-grained sampling period.

[0048] In addition to temporal alignment, data processing node 20 also needs to spatially align the multi-dimensional data, transforming it into the same spatial domain. Accordingly, data processing node 20 can transform the original multi-dimensional data into the same coordinate system based on the mapping relationship between the coordinate systems where the original multi-dimensional data resides, thus achieving spatial alignment of the multi-dimensional data. After achieving temporal and spatial alignment of the multi-dimensional data, the spatiotemporally aligned target multi-dimensional data can be obtained.

[0049] The researchers in this application discovered that the headers of data collected by some sensors contain the coordinate information of those sensors. Based on this, the data processing node 20 can obtain the coordinate system data corresponding to each of the original multi-dimensional data. The coordinate system data corresponding to each dimension of the original data refers to the coordinate system information of the original data in that dimension. Furthermore, based on the coordinate system data corresponding to each of the original multi-dimensional data, the mapping relationship between the coordinates of the original multi-dimensional data can be determined. Furthermore, based on the mapping relationship between the coordinate systems of the original multi-dimensional data, the original multi-dimensional data can be transformed to the same coordinate system, achieving spatial alignment of the multi-dimensional data.

[0050] In some embodiments of this application, the original multi-dimensional data can be transformed to the coordinate system where the vehicle attitude data is located. The coordinate system where the vehicle attitude data is located is a coordinate system established with the vehicle's center point as the origin. Alternatively, the original multi-dimensional data can be transformed to the world coordinate system to achieve spatial alignment of the multi-dimensional data.

[0051] For spatiotemporally aligned target multidimensional data, data processing node 20 can store the spatiotemporally aligned target multidimensional data in data storage node 10, so that the simulation platform can be used for simulation training using the spatiotemporally aligned target multidimensional data. Before using the spatiotemporally aligned target multidimensional data to train the simulation platform, in this embodiment of the application, the spatiotemporally aligned target multidimensional data can be visualized so that users can have an intuitive understanding of the quality of the target multidimensional data.

[0052] For the aforementioned embodiments including spatiotemporally aligned driving environment data, vehicle status data, and vehicle driving routes, to enable users to intuitively perceive the quality of the target's multi-dimensional data, a map of the area traversed by the spatiotemporally aligned vehicle driving route can be obtained based on the spatiotemporally aligned vehicle driving route. The map data can be stored in data storage node 10. In this embodiment, the map data can be base map data or Point of Interest (POI) map data, etc.

[0053] In this embodiment, the location of the positioning point in the spatiotemporally aligned vehicle driving route can be queried in the data storage node 10 to obtain a regional map containing the location of the positioning point in the spatiotemporally aligned vehicle driving route.

[0054] Furthermore, data processing node 20 can provide regional map data and spatiotemporally aligned target multi-dimensional data to rendering engine 30. Accordingly, such as... Figure 1 As shown, the rendering engine 30 can render a region map on the front-end interface; based on the spatiotemporally aligned driving environment data at the corresponding time, it can render the driving environment at the corresponding time on the region map. In this embodiment, the corresponding time refers to the time corresponding to the timestamp used to determine the spatiotemporally aligned vehicle driving route of the region map. In some embodiments, the original multi-dimensional data of the above-mentioned autonomous vehicle may also include: vehicle state data. For a description of the vehicle state data, please refer to the relevant content of the above embodiments, which will not be repeated here. Correspondingly, the spatiotemporally aligned target multi-dimensional data may include: spatiotemporally aligned vehicle state data. In this embodiment, the rendering engine 30 can also render vehicle icons reflecting the vehicle state and vehicle position at the corresponding time on the region map based on the spatiotemporally aligned vehicle driving route and the spatiotemporally aligned vehicle state data at the corresponding time.

[0055] In some embodiments, to ensure the temporal order of multi-dimensional data visualization and prevent data from arriving at the rendering engine 30 before data from the previous moment, the data processing node 20 can also arrange the spatiotemporally aligned target multi-dimensional data according to time sequence to obtain a time series of the target multi-dimensional data. During multi-dimensional data visualization, the rendering engine 30 can render the target multi-dimensional data according to the time sequence.

[0056] Furthermore, to improve rendering speed, data processing node 20 can start multiple threads to download the time series to rendering engine 30 in chronological order. Rendering engine 30 can render the target multi-dimensional data contained in the time series in chronological order. For specific implementation details regarding the rendering engine 30's rendering of the target multi-dimensional data contained in the time series, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0057] To further improve rendering efficiency, a cache queue can be set up to cache target multi-dimensional data downloaded to the rendering engine 30. This reduces the number of repeated downloads and the number of input / output (IO) operations.

[0058] In some embodiments, because the sampling period of the spatiotemporally aligned target multidimensional data is small and the temporal granularity is fine, displaying it frame-by-frame results in low rendering efficiency. To improve the visualization rendering speed, before rendering the target multidimensional data, frames can be extracted from the spatiotemporally aligned target multidimensional data according to a set frame extraction period to obtain the extracted target multidimensional data. The frame extraction period is less than or equal to the user's visual persistence time, but greater than the sampling period of the spatiotemporally aligned target multidimensional data. For example, the sampling period of the time-aligned target multidimensional data can be on the order of nanoseconds, and the frame extraction period can be on the order of millimeters, such as 50 milliseconds.

[0059] Furthermore, the multi-dimensional data of the target after frame extraction can be visualized. Specifically, the data processing node 20 can render vehicle icons reflecting vehicle status and position on the regional map based on the spatiotemporally aligned vehicle driving route after frame extraction at the corresponding time and the spatiotemporally aligned vehicle status data after frame extraction at the corresponding time; and render the driving environment at the corresponding time on the regional map based on the spatiotemporally aligned driving environment data after frame extraction at the corresponding time.

[0060] In the above embodiments, the original driving environment data and original vehicle driving route of the autonomous vehicle can be spatiotemporally aligned; and based on the spatiotemporally aligned vehicle driving route, a map of the area traversed by the spatiotemporally aligned vehicle driving route can be obtained; then, based on the spatiotemporally aligned driving environment data, the driving environment at the corresponding time can be rendered on the regional map, realizing the visualization of the regional map and the vehicle driving environment at the corresponding time. This provides a visual reference for users to perceive multi-dimensional data quality, helping users to intuitively perceive the data quality collected by the autonomous vehicle. For example, through the visualized regional map and the vehicle driving environment at the corresponding time, users can perceive whether the vehicle driving environment at the corresponding time is compatible with the regional map, thereby perceiving the quality of the driving environment data at the corresponding time. For example, if the regional map shows an elevated bridge, while the vehicle driving environment at the corresponding time is a tunnel, it can be determined that the vehicle driving route and vehicle driving environment data at that time are not compatible, that is, the quality of the vehicle driving route and vehicle driving environment data at that time is unqualified, etc.

[0061] In some embodiments, the driving environment data of an autonomous vehicle may include driving environment point cloud data and driving environment image data. Accordingly, the rendering engine 30 can render the point cloud driving environment and driving environment image at a corresponding time on a regional map based on the driving environment point cloud data and driving environment image data, thereby visualizing the point cloud driving environment and driving environment image, providing a reference for users to intuitively perceive the quality of the vehicle's driving environment. For users, the visualized point cloud driving environment and image driving environment can be used to determine whether the driving environment reflected by the two types of driving environment data is the same. If they are the same, it indicates that the quality of the driving environment point cloud data and driving environment image data at the corresponding time is acceptable; if they are different, it indicates that the quality of the driving environment point cloud data and driving environment image data at the corresponding time is unacceptable. For example, if the driving environment point cloud data reflects the presence of obstacle X in the driving environment, but the driving environment image data at the same time reflects the absence of obstacle X at the same location, it indicates that the driving environment reflected by the driving environment point cloud data and driving environment image data at that time is different, and it can be determined that the quality of the driving environment point cloud data and driving environment image data at that time is unacceptable.

[0062] The above-mentioned method renders vehicle icons reflecting vehicle status and location on a regional map, visualizing vehicle status; it also renders the driving environment at a corresponding moment on the regional map, visualizing the driving environment. For users, the visualized vehicle status and driving environment allow them to determine whether the vehicle status and driving environment at a given moment are compatible, thus intuitively perceiving the quality of the multi-dimensional data at that moment. If the vehicle status and driving environment are compatible, the quality of the multi-dimensional data at that moment is considered acceptable. If they are incompatible, the quality of the multi-dimensional data at that moment is considered unacceptable. For example, if the driving environment indicates a straight road, but the vehicle status data indicates the vehicle is turning, then the vehicle status data is incompatible with the driving environment, and therefore, the vehicle status data and driving environment data at that moment are considered unacceptable.

[0063] For target multi-dimensional data whose quality is confirmed by the user, the user can confirm the data quality through a human-computer interaction interface. Accordingly, in response to the quality confirmation operation of the spatiotemporally aligned target multi-dimensional data, the data processing node 20 can use the spatiotemporally aligned target multi-dimensional data to train the simulation module in the autonomous driving simulation platform; and use the trained simulation module to perform driving control of the autonomous vehicle.

[0064] Alternatively, for users to visualize and verify the quality of target multi-dimensional data, data processing node 20 can store the quality-compliant target multi-dimensional data to data storage node 10, so that the simulation platform can use the quality-compliant target multi-dimensional data for simulation model training. Specifically, such as Figure 2 As shown, the simulation module of the autonomous driving simulation platform can be trained using verified target multi-dimensional data. Since the quality of the source data used to train the simulation module directly affects the accuracy of the trained simulation module, the target multi-dimensional data used for training the simulation module in this embodiment is spatiotemporally aligned and verified multi-dimensional data. Therefore, the source data used for training the simulation module is of high quality, which helps to improve the accuracy of the trained simulation module.

[0065] Furthermore, the trained simulation modules can be used to control the driving of autonomous vehicles, such as driving route planning and vehicle state control.

[0066] In addition to the data processing system provided in the above embodiments, this application also provides a data processing method, which will be described exemplarily below.

[0067] Figure 4 This is a schematic flowchart illustrating the data processing method provided in an embodiment of this application. Figure 4 As shown, this data processing method mainly includes the following steps. 401. Obtain raw multi-dimensional data collected by autonomous vehicles during driving; the raw multi-dimensional data shall include at least: raw driving environment data and raw vehicle driving route.

[0068] 402. Perform spatiotemporal alignment on the original multidimensional data to obtain spatiotemporally aligned target multidimensional data; the target multidimensional data shall include at least: spatiotemporally aligned driving environment data and spatiotemporally aligned vehicle driving routes.

[0069] 403. Based on the spatiotemporally aligned vehicle travel routes, obtain a map of the areas traversed by the spatiotemporally aligned vehicle travel routes.

[0070] 404. Based on the spatiotemporally aligned driving environment data at the corresponding time, render the driving environment at the corresponding time on the regional map. The corresponding time refers to the time corresponding to the timestamp used to determine the spatiotemporally aligned vehicle driving route of the regional map.

[0071] In this embodiment of the application, a data visualization scheme is provided to enable users to intuitively perceive the quality of the data trained by the simulation module. Specifically, in step 401, raw multi-dimensional data collected by the autonomous vehicle during its driving process can be obtained from the data storage node. This raw multi-dimensional data includes at least: raw driving environment data and raw vehicle driving route.

[0072] In this application embodiment, the specific implementation form of obtaining raw multi-dimensional data from data storage nodes is not limited. In some embodiments, the user can independently select the target dimension for visualization. For example, a human-computer interaction interface can be provided. This human-computer interaction interface includes a dimension selection option for visualization. The dimensions for visualization include various data dimensions collected by autonomous vehicles, including but not limited to: radar point clouds, images, videos, and some or all of the dimensions such as vehicle status and vehicle driving routes. Users can achieve visualization of target dimension information by selecting the target dimension for visualization.

[0073] Accordingly, the target dimension selected by the user based on the human-computer interaction interface for visualization can be obtained; the raw data of the target dimension can be obtained from the data storage node. The target dimension can be any dimension selected by the user. In this embodiment, the visualization method is illustrated by taking the following as examples: the target dimension includes the driving environment dimension, the vehicle state dimension, and the vehicle driving route dimension.

[0074] Of course, in addition to selecting the target dimension for visualization, users can also set query conditions. In some embodiments, users can provide query conditions through a human-computer interaction interface. Optionally, users can select query conditions or input query conditions through the human-computer interaction interface. Users can set query conditions independently according to their query needs. Query conditions include, but are not limited to, one or more of the following: time conditions, vehicle identification conditions, data dimension conditions, and spatial conditions.

[0075] Accordingly, the query conditions provided by the user based on the human-computer interaction interface can be obtained; and a query expression in a data format supported by the query engine can be generated based on the query conditions; further, the query expression can be parsed to obtain a query statement adapted to the query engine; then, the query statement can be used to query in the data storage node to obtain the target dimension data that meets the query conditions.

[0076] This data query method does not require users to input complex query statements. Users only need to input query conditions, which can be converted into query statements adapted to the query engine. This reduces the user's learning cost and helps to improve the flexibility and versatility of the data processing system.

[0077] The query criteria provided by the user can be of any dimension. Accordingly, query criteria include, but are not limited to, one or more of the following: time criteria, vehicle identification criteria, sensor dimension criteria, and spatial criteria. "Multiple" refers to two or more criteria.

[0078] Optionally, after obtaining the target dimension data that meets the query conditions, the target dimension data can be categorized by vehicle dimension to determine the target dimension data for each autonomous vehicle. The target dimension data can be any data dimension that meets the query conditions. For the target dimension data of each autonomous vehicle, a corresponding data dashboard can be rendered.

[0079] In this embodiment of the application, the visualization method is illustrated by taking the target dimension data, which includes the driving environment dimension, vehicle state dimension, and vehicle driving route dimension, as examples.

[0080] Considering that the various sensors on autonomous vehicles each have their own coordinate system, meaning that the data collected by each sensor is based on the sensor's own coordinate system and the data is in different spatial domains; on the other hand, even after hardware synchronization, different sensors still have different sampling frequencies, meaning that the data acquired by the sensors are not synchronized in terms of timestamps.

[0081] Based on this, in step 402, the original multi-dimensional data can be spatiotemporally aligned to obtain spatiotemporally aligned target multi-dimensional data. The spatiotemporally aligned target multi-dimensional data includes at least: spatiotemporally aligned driving environment data and spatiotemporally aligned vehicle driving routes, etc.

[0082] Specifically, the original multi-dimensional data can be time-aligned based on its timestamps. Optionally, for any first-dimensional and second-dimensional data in the original multi-dimensional data, a first target timestamp corresponding to the first-dimensional data and a second target timestamp corresponding to the second-dimensional data can be determined based on the timestamps of the first-dimensional data and the second-dimensional data. Specifically, the first-dimensional data has no corresponding data at the first target timestamp, but the second-dimensional data has data at the first target timestamp; the second-dimensional data has no corresponding data at the second target timestamp, but the second-dimensional data has data at the second target timestamp.

[0083] Furthermore, first-dimensional target sampling data adjacent to the first target timestamp can be obtained from the first-dimensional data; and interpolation is performed on the first-dimensional data at the target timestamp corresponding to the first-dimensional data based on the first-dimensional target sampling data. Specifically, the first-dimensional target sampling data adjacent to the first target timestamp of the first-dimensional data can be data from the first-dimensional data located at the previous sampling point of the first target timestamp, or data from the first-dimensional data located at the next sampling point of the first target timestamp, or data from two sampling points in the first-dimensional data located before and after the first target timestamp.

[0084] Furthermore, in an embodiment where the first dimension target sampling data is the data of the sampling point preceding the first target timestamp in the first dimension data, the data of the sampling point preceding the first target timestamp can be interpolated onto the target timestamp corresponding to the first dimension data, so that the first dimension data and the second dimension data are aligned on the first target timestamp.

[0085] In an embodiment where the first dimension target sampling data is the data located at the next sampling point after the first target timestamp in the first dimension data, the data located at the next sampling point after the first target timestamp can be interpolated onto the first target timestamp corresponding to the first dimension data, so that the first dimension data and the second dimension data are aligned on the first target timestamp.

[0086] For the first dimension target sampling data, which is the data of two sampling points before and after the first target timestamp in the first dimension data, the mean of the two sampling points before and after the first target timestamp can be calculated, and the mean can be interpolated onto the first target timestamp corresponding to the first dimension data, so that the first dimension data and the second dimension data are aligned with the first target timestamp.

[0087] Similarly, second-dimensional target sampling data whose sampling time is adjacent to the second target timestamp of the second-dimensional data can be obtained from the second-dimensional data; and based on the second-dimensional target sampling data, interpolation is performed on the second-dimensional data at the corresponding second target timestamp, thereby achieving temporal alignment between the first-dimensional data and the second-dimensional data. For a description of the second-dimensional target sampling data and a detailed implementation method for interpolating the second-dimensional data at the corresponding second target timestamp, please refer to the relevant content on the interpolation processing of the first-dimensional data described above, which will not be repeated here.

[0088] In some embodiments, in addition to interpolating the original multi-dimensional data to achieve temporal alignment, as described above, interpolation can also be performed on the time-aligned multi-dimensional data according to the same target sampling period. The target sampling period is shorter than the sampling period of the time-aligned multi-dimensional data. This allows for data alignment at a finer temporal granularity. In this application embodiment, the specific value of the target sampling period is not limited. Optionally, the target sampling period can be a millisecond-level, nanosecond-level, or even finer-grained sampling period.

[0089] In addition to temporal alignment, multi-dimensional data also needs spatial alignment, meaning it needs to be transformed into the same spatial domain. Accordingly, based on the mapping relationship between the original coordinate systems of the multi-dimensional data, the original multi-dimensional data can be transformed into the same coordinate system, achieving spatial alignment. After achieving temporal and spatial alignment of the multi-dimensional data, the spatiotemporally aligned target multi-dimensional data can be obtained.

[0090] The researchers in this application discovered that the headers of data collected by some sensors contain the coordinate information of those sensors. Based on this, the coordinate system data corresponding to each of the original multi-dimensional data can be obtained. The coordinate system data corresponding to each dimension of the original data refers to the coordinate system information of the original data in that dimension. Furthermore, based on the coordinate system data corresponding to each of the original multi-dimensional data, the mapping relationship between the coordinates of the original multi-dimensional data can be determined. Furthermore, based on the mapping relationship between the coordinate systems of the original multi-dimensional data, the original multi-dimensional data can be transformed to the same coordinate system, achieving spatial alignment of the multi-dimensional data.

[0091] In some embodiments of this application, the original multi-dimensional data can be transformed to the coordinate system where the vehicle attitude data is located. The coordinate system where the vehicle attitude data is located is a coordinate system established with the vehicle's center point as the origin. Alternatively, the original multi-dimensional data can be transformed to the world coordinate system to achieve spatial alignment of the multi-dimensional data.

[0092] For spatiotemporally aligned target multidimensional data, the spatiotemporally aligned target multidimensional data can be stored in a data storage node for subsequent simulation training of the simulation platform. Before using the spatiotemporally aligned target multidimensional data to train the simulation platform, in this embodiment, the spatiotemporally aligned target multidimensional data can be visualized to give users an intuitive understanding of the quality of the target multidimensional data.

[0093] For the above embodiments including spatiotemporally aligned driving environment data, vehicle status data, and vehicle driving routes, in order to enable users to intuitively perceive the quality of the target's multi-dimensional data, in step 403, a map of the area traversed by the spatiotemporally aligned vehicle driving route is obtained based on the spatiotemporally aligned vehicle driving route. The map data can be stored in a data storage node. In this embodiment, the map data can be base map data or POI map data, etc.

[0094] In this embodiment, the location of the positioning point in the spatiotemporally aligned vehicle driving route can be queried in the data storage node to obtain a regional map containing the location of the positioning point in the spatiotemporally aligned vehicle driving route.

[0095] Furthermore, in step 404, the driving environment at the corresponding moment can be rendered on the regional map based on the spatiotemporally aligned driving environment data at the corresponding moment. In this embodiment, the corresponding moment refers to the moment corresponding to the timestamp used to determine the spatiotemporally aligned vehicle driving route of the regional map.

[0096] In some embodiments, the original multi-dimensional data of the aforementioned autonomous vehicle may further include: vehicle state data. For a description of the vehicle state data, please refer to the relevant content in the above embodiments, which will not be repeated here. Correspondingly, the spatiotemporally aligned target multi-dimensional data may include: spatiotemporally aligned vehicle state data. In this embodiment, based on the spatiotemporally aligned vehicle driving route and the spatiotemporally aligned vehicle state data at the corresponding time, vehicle icons reflecting the vehicle state and position at the corresponding time can also be rendered on the regional map.

[0097] In some embodiments, to ensure the temporal order of multi-dimensional data visualization and prevent data from arriving at the rendering engine before data from the previous moment, the spatiotemporally aligned target multi-dimensional data can be arranged in chronological order to obtain a time series of the target multi-dimensional data. During multi-dimensional data visualization, the target multi-dimensional data can be rendered according to the chronological order in the time series.

[0098] Furthermore, to improve rendering speed, multiple threads can be started to download the time series to the rendering engine in chronological order. The rendering engine can then render the target multi-dimensional data contained in the time series in chronological order. For details on the specific implementation of the rendering engine's rendering of the target multi-dimensional data contained in the time series, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0099] To further improve rendering efficiency, a cache queue can be set up to cache the target multi-dimensional data downloaded to the rendering engine, which can reduce the number of repeated downloads and I / O operations.

[0100] In some embodiments, because the sampling period of the spatiotemporally aligned target multidimensional data is small and the temporal granularity is fine, displaying it frame-by-frame results in low rendering efficiency. To improve the visualization rendering speed, before rendering the target multidimensional data, frames can be extracted from the spatiotemporally aligned target multidimensional data according to a set frame extraction period to obtain the extracted target multidimensional data. The frame extraction period is less than or equal to the user's visual persistence time, but greater than the sampling period of the spatiotemporally aligned target multidimensional data. For example, the sampling period of the time-aligned target multidimensional data can be on the order of nanoseconds, and the frame extraction period can be on the order of millimeters, such as 50 milliseconds.

[0101] Furthermore, the multi-dimensional data of the target after frame extraction can be visualized. Specifically, based on the spatiotemporally aligned vehicle driving route and the spatiotemporally aligned vehicle status data after frame extraction at the corresponding time, vehicle icons reflecting the vehicle status and position can be rendered on the regional map; and based on the spatiotemporally aligned driving environment data after frame extraction at the corresponding time, the driving environment at the corresponding time can be rendered on the regional map.

[0102] In the above embodiments, the original driving environment data and original vehicle driving route of the autonomous vehicle can be spatiotemporally aligned; and based on the spatiotemporally aligned vehicle driving route, a map of the area traversed by the spatiotemporally aligned vehicle driving route can be obtained; then, based on the spatiotemporally aligned driving environment data, the driving environment at the corresponding time can be rendered on the regional map, realizing the visualization of the regional map and the vehicle driving environment at the corresponding time. This provides a visual reference for users to perceive multi-dimensional data quality, helping users to intuitively perceive the data quality collected by the autonomous vehicle. For example, through the visualized regional map and the vehicle driving environment at the corresponding time, users can perceive whether the vehicle driving environment at the corresponding time is compatible with the regional map, thereby perceiving the quality of the driving environment data at the corresponding time. For example, if the regional map shows an elevated bridge, while the vehicle driving environment at the corresponding time is a tunnel, it can be determined that the vehicle driving route and vehicle driving environment data at that time are not compatible, that is, the quality of the vehicle driving route and vehicle driving environment data at that time is unqualified, etc.

[0103] The driving environment data for autonomous vehicles can include driving environment point cloud data and driving environment image data. Accordingly, based on the driving environment point cloud data and driving environment image data, the point cloud driving environment and driving environment image at a corresponding moment can be rendered on a regional map, realizing the visualization of the point cloud driving environment and driving environment image, providing users with a reference for intuitively perceiving the quality of the vehicle's driving environment. For users, the visualized point cloud driving environment and image driving environment can be used to determine whether the driving environment reflected by the two types of driving environment data is the same. If they are the same, it indicates that the quality of the driving environment point cloud data and driving environment image data at the corresponding moment is acceptable; if they are different, it indicates that the quality of the driving environment point cloud data and driving environment image data at the corresponding moment is unacceptable. For example, if the driving environment point cloud data indicates the presence of obstacle X in the driving environment, but the driving environment image data at the same moment indicates the absence of obstacle X at the same location, then it indicates that the driving environment reflected by the driving environment point cloud data and driving environment image data at that moment is different, and it can be determined that the quality of the driving environment point cloud data and driving environment image data at that moment is unacceptable.

[0104] The above-mentioned method renders vehicle icons reflecting vehicle status and location on a regional map, visualizing vehicle status; it also renders the driving environment at a corresponding moment on the regional map, visualizing the driving environment. For users, the visualized vehicle status and driving environment allow them to determine whether the vehicle status and driving environment at a given moment are compatible, thus intuitively perceiving the quality of the multi-dimensional data at that moment. If the vehicle status and driving environment are compatible, the quality of the multi-dimensional data at that moment is considered acceptable. If the vehicle status and its environment are incompatible, the quality of the multi-dimensional data at that moment is considered unacceptable. For example, if the driving environment indicates a straight road, but the vehicle status data indicates the vehicle is turning, then the vehicle status data is incompatible with the driving environment, and therefore, the vehicle status data and driving environment data at that moment are considered unacceptable.

[0105] For target multi-dimensional data whose quality is confirmed by the user, the user can confirm the data quality through the human-computer interaction interface. Accordingly, in response to the quality confirmation operation for spatiotemporally aligned target multi-dimensional data, the simulation module in the autonomous driving simulation platform can be trained using the spatiotemporally aligned target multi-dimensional data; and the trained simulation module can be used to control the driving of the autonomous vehicle.

[0106] Alternatively, for users to visualize and verify qualified target multi-dimensional data, the qualified target multi-dimensional data can be stored in a data storage node for subsequent simulation platform training of simulation models. Specifically, the verified target multi-dimensional data can be used to train the simulation module of the autonomous driving simulation platform. Since the quality of the source data used to train the simulation module directly affects the accuracy of the trained simulation module, the target multi-dimensional data used for training the simulation module in this embodiment is spatiotemporally aligned and verified multi-dimensional data. Therefore, the quality of the source data used for training the simulation module is high, which helps to improve the accuracy of the trained simulation module.

[0107] Furthermore, the trained simulation modules can be used to control the driving of autonomous vehicles, such as driving route planning and vehicle state control.

[0108] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 401 and 402 can be device A; or the execution subject of step 401 can be device A, and the execution subject of step 402 can be device B; and so on.

[0109] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 401, 402, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0110] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the above-described data processing method.

[0111] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 5 As shown, the computing device includes: a memory 50a, a processor 50b, and a display component 50c; wherein, the memory 50a is used to store computer programs.

[0112] The processor 50b is coupled to the memory 50a and the display component 50c to execute a computer program for performing the following: acquiring raw multi-dimensional data collected by the autonomous vehicle during driving; the raw multi-dimensional data includes at least: raw driving environment data and raw vehicle driving route; performing spatiotemporal alignment on the raw multi-dimensional data to obtain spatiotemporally aligned target multi-dimensional data; the target multi-dimensional data includes at least: spatiotemporally aligned driving environment data and spatiotemporally aligned vehicle driving route; acquiring a map of the area traversed by the spatiotemporally aligned vehicle driving route based on the spatiotemporally aligned vehicle driving route; and rendering the driving environment at the corresponding moment on the area map through the display component 50c based on the spatiotemporally aligned driving environment data at the corresponding moment; the corresponding moment refers to the time corresponding to the timestamp used to determine the spatiotemporally aligned vehicle driving route of the area map.

[0113] In some embodiments, the original multidimensional data further includes: original vehicle state data; the target multidimensional data further includes: spatiotemporally aligned vehicle-road state data. The processor 50b is also configured to: render vehicle icons reflecting the vehicle state and vehicle position at a given time on the area map, based on the spatiotemporally aligned vehicle travel route and the spatiotemporally aligned vehicle state data at the given time.

[0114] Optionally, the processor 50b is also configured to: in response to a quality compliance verification operation for spatiotemporally aligned target multidimensional data, train a simulation module in an autonomous driving simulation platform using the spatiotemporally aligned target multidimensional data; and use the trained simulation module to perform driving control on an autonomous vehicle.

[0115] Optionally, when performing spatiotemporal alignment on the original multidimensional data, the processor 50b specifically performs: time alignment on the original multidimensional data based on the timestamps of the original multidimensional data; and transforms the original multidimensional data to the same coordinate system based on the mapping relationship between the coordinate systems in which the original multidimensional data is located, so as to obtain the target multidimensional data.

[0116] Furthermore, when processor 50b performs time alignment on the original multi-dimensional data based on the timestamps of the original multi-dimensional data, it specifically performs the following: for any first-dimensional data and second-dimensional data in the original multi-dimensional data, it determines a first target timestamp corresponding to the first-dimensional data and a target timestamp corresponding to the second-dimensional data based on the timestamps of the first-dimensional data and the second-dimensional data; wherein, the first-dimensional data has no corresponding data at the first target timestamp, and the second-dimensional data has data at the first target timestamp; the second-dimensional data has no corresponding data at the second target timestamp, and the first-dimensional data has data at the second target timestamp; furthermore, it can obtain first-dimensional target sampled data whose sampling time is adjacent to the target timestamp of the first-dimensional data from the first-dimensional data; obtain second-dimensional target sampled data whose sampling time is adjacent to the target timestamp of the second-dimensional data from the second-dimensional data; perform interpolation on the target timestamp of the first-dimensional data based on the first-dimensional target sampled data, and perform interpolation on the target timestamp of the second-dimensional data based on the second-dimensional target sampled data, so as to align the first-dimensional data and the second-dimensional data in time.

[0117] Optionally, when processor 50b performs time alignment on the original multi-dimensional data based on the timestamps of the original multi-dimensional data, it is also used to: interpolate the time-aligned multi-dimensional data according to the same target sampling period, wherein the target sampling period is less than the sampling period of the time-aligned multi-dimensional data.

[0118] Optionally, the processor 50b is further configured to: before transforming the original multi-dimensional data to the same coordinate system based on the mapping relationship between the coordinate systems of the original multi-dimensional data, obtain the coordinate system data corresponding to each of the original multi-dimensional data from the original multi-dimensional data; and determine the mapping relationship between the coordinate systems of the original multi-dimensional data based on the coordinate system data corresponding to each of the original multi-dimensional data.

[0119] In some embodiments, the processor 50b is further configured to: perform frame extraction on the spatiotemporally aligned target multidimensional data according to a set frame extraction period to obtain the frame-extracted target multidimensional data; the frame extraction period is less than or equal to the user's visual persistence time. Accordingly, when the processor 50b renders vehicle icons reflecting vehicle status and vehicle position on the area map based on the spatiotemporally aligned vehicle driving route and the spatiotemporally aligned vehicle status data at the corresponding time, it is specifically configured to: render vehicle icons reflecting vehicle status and vehicle position on the area map based on the frame-extracted spatiotemporally aligned vehicle driving route and the frame-extracted spatiotemporally aligned vehicle status data at the corresponding time.

[0120] Accordingly, when the processor 50b renders the driving environment at the corresponding moment on the regional map based on the spatiotemporally aligned driving environment data at the corresponding moment, it is specifically used to: render the driving environment at the corresponding moment on the regional map based on the spatiotemporally aligned driving environment data after frame extraction at the corresponding moment.

[0121] Optionally, the processor 50b is also used to: acquire the target dimension selected by the user based on the human-computer interaction interface for visualization; acquire the spatiotemporal alignment data of the target dimension; and display the data dashboard corresponding to the spatiotemporal alignment data of the target dimension at the corresponding time on the regional map.

[0122] Optionally, the processor 50b is also used to: arrange the spatiotemporally aligned target multidimensional data in chronological order to obtain a temporal sequence of the target multidimensional data; and start multiple threads to download the temporal sequence to the rendering engine in chronological order so that the rendering engine can render the target multidimensional data in chronological order.

[0123] Optionally, the processor 50b is also configured to: obtain query conditions provided by the user based on the human-computer interaction interface; generate a query expression in a data format supported by the query engine based on the query conditions; perform syntax parsing on the query expression to obtain a query statement adapted to the query engine; and use the query statement to perform a query in the database storing data of autonomous vehicles to obtain target dimension data that meets the query conditions.

[0124] Optionally, the processor 50b is also configured to: perform spatiotemporal alignment on the target dimension data to obtain spatiotemporally aligned target dimension data; classify the spatiotemporally aligned target dimension data by vehicle dimension to determine the target dimension data for each autonomous vehicle; and display the data dashboard corresponding to the target dimension data at the corresponding time through the display component 50c.

[0125] In some alternative implementations, such as Figure 5 As shown, the computing device may also include optional components such as a communication component 50d, a power supply component 50e, and an audio component 50f. Figure 5 The diagram only shows some components and does not mean that the computing device must contain them. Figure 5 The inclusion of all components does not imply that a computing device can only include... Figure 5 The components shown.

[0126] The computing device provided in this embodiment can perform spatiotemporal alignment of the original driving environment data and the original driving route of the autonomous vehicle; and obtain a map of the area traversed by the spatiotemporally aligned driving route based on the spatiotemporally aligned driving route; then, it can render the driving environment at the corresponding time on the area map based on the spatiotemporally aligned driving environment data, realizing the visualization of the area map and the vehicle driving environment at the corresponding time, which can provide a visual reference for users to perceive the multi-dimensional data quality, and help users intuitively perceive the data quality collected by the autonomous vehicle.

[0127] In this embodiment, the memory is used to store computer programs and can be configured to store various other data to support operation on its host device. The processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0128] In the embodiments of this application, the processor can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a programmable device such as a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), or a complex programmable logic device (CPLD); or it can be an advanced reduced instruction set (RISC) processor (ARM) or a system on chip (SOC), etc., but is not limited thereto.

[0129] In this embodiment, the communication component is configured to facilitate wired or wireless communication between its host device and other devices. The device housing the communication component can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In another exemplary embodiment, the communication component may also be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Bluetooth (BT), or other technologies.

[0130] In embodiments of this application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0131] In this embodiment, a power supply component is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0132] In embodiments of this application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, in devices with voice interaction capabilities, voice interaction with the user can be achieved through the audio component.

[0133] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0140] Computer storage media are readable storage media, also known as removable media. Removable and non-removable media can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Acquire raw, multi-dimensional data collected by autonomous vehicles during operation; The original multi-dimensional data includes at least: original driving environment data, original vehicle driving route, and original vehicle status data; the original driving environment data refers to the environmental data collected by the autonomous vehicle during driving, including: image data of the environment collected by the image acquisition device on the autonomous vehicle and point cloud data of the environment collected by the radar on the autonomous vehicle. The original multi-dimensional data is spatiotemporally aligned to obtain spatiotemporally aligned target multi-dimensional data; the target multi-dimensional data includes at least: spatiotemporally aligned driving environment data, spatiotemporally aligned vehicle driving routes, and spatiotemporally aligned vehicle-road status data. Based on the spatiotemporally aligned vehicle travel route, obtain a map of the area traversed by the spatiotemporally aligned vehicle travel route. Based on the spatiotemporally aligned driving environment data at the corresponding moment, the point cloud driving environment and driving environment image at the corresponding moment are rendered on the regional map so that the user can perceive whether the driving environment at the corresponding moment is compatible with the regional map and whether the driving environment reflected by the point cloud driving environment and the driving environment image is the same; the corresponding moment refers to the time corresponding to the timestamp used to determine the spatiotemporally aligned vehicle driving route of the regional map. Based on the spatiotemporally aligned vehicle driving route and the spatiotemporally aligned vehicle status data at the corresponding time, vehicle icons reflecting the vehicle status and vehicle position at the corresponding time are rendered on the regional map. Based on the spatiotemporally aligned driving environment data at the corresponding time, the driving environment at the corresponding time is rendered on the regional map so that the user can perceive whether the vehicle status at the corresponding time is compatible with the driving environment. If the driving environment at the corresponding time is compatible with the regional map, and the driving environment reflected by the point cloud driving environment and the driving environment image is the same, and the vehicle status at the corresponding time is compatible with the driving environment, then the quality of the spatiotemporally aligned driving environment data, the spatiotemporally aligned vehicle driving route, and the spatiotemporally aligned vehicle-road status data is determined to be qualified. In response to the quality acceptance verification operation of the spatiotemporally aligned target multidimensional data, the simulation module in the autonomous driving simulation platform is trained using the spatiotemporally aligned target multidimensional data. The trained simulation module is used to control the driving of the autonomous vehicle.

2. The method according to claim 1, characterized in that, The process of performing spatiotemporal alignment on the original multi-dimensional data to obtain spatiotemporally aligned target multi-dimensional data includes: Based on the timestamps of the original multi-dimensional data, the original multi-dimensional data is time-aligned. Based on the mapping relationship between the coordinate systems of the original multidimensional data, the original multidimensional data is transformed to the same coordinate system to obtain the target multidimensional data.

3. The method according to claim 2, characterized in that, The step of aligning the original multi-dimensional data according to its timestamp includes: For any first-dimensional data and second-dimensional data in the original multi-dimensional data, a first target timestamp corresponding to the first-dimensional data and a second target timestamp corresponding to the second-dimensional data are determined based on the timestamps of the first-dimensional data and the second-dimensional data; wherein, the first-dimensional data has no corresponding data at the first target timestamp, and the second-dimensional data has data at the first target timestamp; and the second-dimensional data has no corresponding data at the second target timestamp, and the first-dimensional data has data at the second target timestamp. Obtain the first dimension target sampling data whose sampling time is adjacent to the first target timestamp from the first dimension data; Obtain second-dimensional target sampling data whose sampling time is adjacent to the second target timestamp from the second-dimensional data; Based on the first dimension target sampling data, the first dimension data is interpolated at the first target timestamp, and based on the second dimension target sampling data, the second dimension data is interpolated at the second target timestamp, so as to align the first dimension data and the second dimension data in time.

4. The method according to claim 3, characterized in that, The step of performing time alignment on the original multi-dimensional data based on the timestamps of the original multi-dimensional data further includes: Interpolate time-aligned multi-dimensional data according to the same target sampling period, wherein the target sampling period is smaller than the sampling period of the time-aligned multi-dimensional data.

5. The method according to claim 2, characterized in that, Before transforming the original multi-dimensional data to the same coordinate system based on the mapping relationship between the coordinate systems where the original multi-dimensional data is located, the process further includes: From the original multi-dimensional data, obtain the coordinate system data corresponding to each of the original multi-dimensional data; Based on the coordinate system data corresponding to the original multidimensional data, determine the mapping relationship between the coordinate systems where the original multidimensional data is located.

6. The method according to any one of claims 1-5, characterized in that, Also includes: The spatiotemporally aligned target multidimensional data is framed according to the set frame extraction period to obtain the frame-extracted target multidimensional data. The frame extraction period is less than or equal to the user's visual persistence time; The step of rendering vehicle icons reflecting the vehicle status and position at the corresponding time on the regional map based on the spatiotemporally aligned vehicle driving routes and the spatiotemporally aligned vehicle status data at the corresponding time includes: Based on the spatiotemporally aligned vehicle driving routes after frame extraction at the corresponding time and the spatiotemporally aligned vehicle status data after frame extraction at the corresponding time, vehicle icons reflecting vehicle status and vehicle position are rendered on the area map. The step of rendering the driving environment at the corresponding moment on the regional map based on the spatiotemporally aligned driving environment data at the corresponding moment includes: Based on the spatiotemporally aligned driving environment data after frame extraction at the corresponding time, the driving environment at the corresponding time is rendered on the regional map.

7. The method according to any one of claims 1-5, characterized in that, Also includes: Obtain the target dimension selected by the user based on the human-computer interaction interface for visualization; Obtain the spatiotemporally aligned data for the target dimension; A data dashboard is displayed on the regional map showing the spatiotemporally aligned data of the target dimension at the corresponding time.

8. The method according to any one of claims 1-5, characterized in that, Also includes: The spatiotemporally aligned target multidimensional data are arranged in chronological order to obtain a temporal sequence of the target multidimensional data. Multiple threads are started to download the time sequence to the rendering engine in chronological order, so that the rendering engine can render the target multi-dimensional data in chronological order.

9. The method according to any one of claims 1-5, characterized in that, Also includes: Obtain the query conditions provided by the user based on the human-computer interaction interface; Based on the query conditions, generate a query expression in a data format supported by the query engine; The query expression is parsed to obtain a query statement adapted to the query engine; The query statement is used to query the database storing data of autonomous vehicles to obtain target dimension data that meets the query conditions.

10. The method according to claim 9, characterized in that, Also includes: The target dimension data is spatiotemporally aligned to obtain spatiotemporally aligned target dimension data; The target dimension data for spatiotemporal alignment is classified according to the vehicle dimension in order to determine the target dimension data for each autonomous vehicle. Display the data dashboard corresponding to the target dimension data at the given time.

11. A computing device, characterized in that, include: A memory, a processor, and a display component; wherein the memory is used to store computer programs; The processor is coupled to the memory and the display component for executing the computer program to perform the steps of the method according to any one of claims 1-10.

12. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1-10.

Images