Data processing method, apparatus, device, medium, and product

Abstract

The application discloses a data processing method and device, equipment, medium and product, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring obstacle information of a vehicle in the case that the vehicle is in a first data processing mode, wherein the obstacle information comprises at least one obstacle; in the case that the obstacle belongs to a target type and the distance between the obstacle and the vehicle is less than a preset distance threshold, acquiring N frames of dynamic data before a first timestamp corresponding to the obstacle information and closest to the first timestamp based on the first timestamp, wherein N is a positive integer, and the dynamic data comprises at least one of an obstacle image frame, an obstacle point cloud frame, an obstacle detection position and vehicle pose data; and storing the dynamic data with the same second timestamp in an associated manner based on the second timestamp of the dynamic data. According to the application, the utilization rate of the overall data can be improved.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a data processing method, apparatus, device, medium and product. Background Technology

[0002] Autonomous driving technology is an innovative achievement resulting from the deep integration of the automotive industry with cutting-edge fields such as sensor and communication technologies. It plays a crucial role in enhancing road traffic safety and improving traffic efficiency. Data acquisition is a core component in the research and development of autonomous driving technology; the efficient collection, processing, and application of data are the cornerstones for realizing autonomous driving functions.

[0003] Currently, vehicles are equipped with various sensors, such as cameras, millimeter-wave radar, lidar, and positioning devices, which can collect massive amounts of data in real time, including environmental information and vehicle status information during the vehicle's operation.

[0004] However, a large amount of invalid data is collected by these sensors, resulting in low data utilization. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, medium, and product that can improve the overall data utilization rate.

[0006] A first aspect of this application provides a data processing method, the method comprising:

[0007] When the vehicle is in the first data processing mode, obstacle information of the vehicle is acquired, including at least one obstacle.

[0008] When the obstacle is a target type and the distance between it and the vehicle is less than a preset distance threshold, based on the first timestamp corresponding to the obstacle information, obtain N frames of dynamic data that are closest to the first timestamp and before the first timestamp. N is a positive integer. The dynamic data includes at least one of obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data.

[0009] Based on the second timestamp of dynamic data, dynamic data with the same second timestamp are associated and stored.

[0010] A second aspect of this application provides a data processing apparatus, the apparatus comprising:

[0011] An obstacle acquisition module is used to acquire obstacle information of the vehicle when the vehicle is in a first data processing mode. The obstacle information includes at least one obstacle.

[0012] The data acquisition module is used to acquire N frames of dynamic data that are closest to the first time stamp and before the first time stamp, based on the first time stamp corresponding to the obstacle information, when the obstacle belongs to the target type and the distance between it and the vehicle is less than a preset distance threshold. N is a positive integer. The dynamic data includes at least one of obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data.

[0013] The data storage module is used to associate and store dynamic data with the same second timestamp based on the second timestamp of the dynamic data.

[0014] A third aspect of the embodiments of this application provides an electronic device, the device comprising: a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, they implement the data processing method provided in any of the embodiments of this application described above.

[0015] A fourth aspect of the embodiments of this application provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the data processing method provided by any aspect of the embodiments of this application described above.

[0016] A fifth aspect of the embodiments of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a data processing method as provided in any of the embodiments of this application described above.

[0017] In the data processing method provided in this application embodiment, when the vehicle is in a first data processing mode, obstacle information of the vehicle is acquired. Then, when the obstacle in the obstacle information belongs to the target type and the distance between it and the vehicle is less than a preset distance threshold, N frames of dynamic data closest to the first timestamp are acquired and saved. Thus, this application sets a corresponding first data processing mode according to test requirements. Then, after the required obstacle is detected within the preset distance threshold through the first data processing mode, the corresponding dynamic data is acquired based on the corresponding timestamp and stored. There is no need to store each dynamic data collected by the sensor in real time, thereby reducing the collection of invalid data and improving the overall data utilization rate. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic flowchart of a first data processing method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of obstacle detection location matching provided in one embodiment of this application.

[0021] Figure 3 This is a flowchart illustrating a second data processing method provided in one embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this application. Detailed Implementation

[0024] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0025] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0026] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0027] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0028] Autonomous driving technology, as an innovative achievement resulting from the deep integration of the automotive industry with multiple cutting-edge technologies, plays a crucial role in improving road traffic safety and efficiency. Data acquisition and processing are key aspects of realizing autonomous driving functions. However, current vehicle sensors generate a large amount of invalid data when collecting environmental and vehicle status information in real time, leading to low data utilization.

[0029] Specifically, this problem is particularly prominent in practical applications. For example, in a typical urban road scenario, autonomous vehicles are equipped with various sensors such as high-definition cameras, millimeter-wave radar, and lidar. These sensors generate massive amounts of data every second, including image data of surrounding vehicles, pedestrians, traffic signs, and point cloud data such as distance and speed. However, during normal driving, most of the collected data may not contain information that substantially affects autonomous driving decisions. For example, when driving straight on a wide road, distant buildings or stationary objects on the roadside may be repeatedly captured, but this data does not significantly help the vehicle's immediate decision-making.

[0030] Consequently, a large amount of redundant data not only occupies valuable storage space but also increases data processing time and complexity. In emergency situations requiring rapid response, this data redundancy can slow down system reaction speed, affecting the safety and reliability of autonomous driving. Furthermore, during data analysis and machine learning model training, excessive irrelevant data can also reduce the learning efficiency and accuracy of algorithms.

[0031] The purpose of this application is to provide a data processing method, apparatus, device, medium, and product. In the data processing method provided by this application, when the vehicle is in a first data processing mode, obstacle information of the vehicle is acquired. Then, when the obstacle in the obstacle information belongs to the target type and the distance between it and the vehicle is less than a preset distance threshold, N frames of dynamic data closest to the first timestamp are acquired and saved. Thus, this application sets a corresponding first data processing mode according to testing requirements. Then, after detecting the required obstacle within the preset distance threshold using the first data processing mode, the corresponding dynamic data is acquired based on the corresponding timestamp and stored. This eliminates the need for real-time storage of all dynamic data collected by sensors, thereby reducing the collection of invalid data and improving the overall data utilization rate.

[0032] The following describes specific embodiments of the data processing method, apparatus, device, medium, and product provided in this application. The data processing method will be described first.

[0033] Figure 1 A flowchart of a data processing method is provided. This data processing method can be applied to a server and may include the following steps S101 to S103.

[0034] S101, when the vehicle is in the first data processing mode, acquire obstacle information of the vehicle, the obstacle information including at least one obstacle;

[0035] In this embodiment, the first data processing mode refers to the data acquisition mode of the vehicle under a specific driving state. Specifically, it can be implemented by automatically switching based on factors such as vehicle speed and driving environment, or by manual setting. The first data processing mode is a data processing mode that performs single-frame acquisition or continuous frame acquisition.

[0036] Obstacle information refers to data such as the position, speed, and type of surrounding objects. Specifically, it can be obtained by fusing the detection results of LiDAR, camera, etc. through the perception fusion function program.

[0037] Specifically, the vehicle first enters the first data processing mode. This mode can be automatically triggered based on factors such as vehicle speed and driving environment, or it can be manually set by the driver. In this mode, the onboard sensors are activated and begin to continuously collect information about the surrounding environment.

[0038] Then, the server acquires information about obstacles around the vehicle. Specifically, this step is accomplished through the collaboration of multiple sensors, such as cameras capturing image information, and millimeter-wave radar and lidar acquiring distance and speed information. After preliminary processing, this raw data is transformed into structured data containing information such as obstacle location, speed, and type.

[0039] S102, when the obstacle is a target type and the distance between it and the vehicle is less than a preset distance threshold, based on the first timestamp corresponding to the obstacle information, obtain N frames of dynamic data that are closest to the first timestamp and before the first timestamp, where N is a positive integer, and the dynamic data includes at least one of obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data.

[0040] In this embodiment, dynamic data refers to vehicle environment information that changes over time, which can be represented by continuously acquired obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data.

[0041] Specifically, the server filters the acquired obstacle information. The filtering criteria include two aspects: obstacle type and distance between the obstacle and the vehicle. Subsequent data processing is only triggered when the obstacle belongs to a predefined target type (e.g., pedestrian, vehicle) and its distance from the vehicle is less than a preset distance threshold. The purpose of this step is to filter out distant or non-critical obstacles that have little impact on autonomous driving decisions, thereby reducing the amount of data that needs to be processed.

[0042] Then, for obstacles that meet the conditions, the server searches backward and retrieves the nearest dynamic data frames (N frames) based on the first timestamp corresponding to the obstacle information. Here, N is a positive integer representing the number of historical data frames to be retrieved. The dynamic data includes various types such as obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data, all originating from different sensors.

[0043] In addition, when the vehicle is in the first data processing mode, the server will store all static data that does not change over time, such as the scene map, LiDAR external parameters, and camera internal and external parameters.

[0044] S103, based on the second timestamp of the dynamic data, associate and store dynamic data with the same second timestamp.

[0045] In this embodiment, associated storage refers to associating and saving different types of dynamic data with the same timestamp. This can be achieved by creating an index or associating database tables.

[0046] Specifically, the server associates and stores the acquired dynamic data according to the second timestamp. In other words, the server groups different types of dynamic data with the same second timestamp together to form a complete data packet. This associated storage method ensures time synchronization between data from different sources, facilitating subsequent data analysis and processing.

[0047] As a preferred implementation, this application can be used in autonomous driving scenarios on urban roads. Specifically, suppose an autonomous vehicle equipped with a high-definition camera and LiDAR is driving on a main urban road. When the vehicle enters the vicinity of an intersection, it automatically switches to a first data processing mode.

[0048] In the first data processing mode, the server first obtains information about surrounding obstacles. For example, if a camera detects a pedestrian crossing the road 50 meters ahead, the first timestamp of this pedestrian is T1. The server determines that the obstacle belongs to the target type (pedestrian) and that the distance is less than a preset distance threshold (e.g., 100 meters), thus triggering the subsequent data processing flow.

[0049] Next, the server searches backwards and retrieves the most recent 5 frames (N=5) of dynamic data based on timestamp T1. This dynamic data includes: 5 frames of camera images, 5 frames of point cloud data, 5 sets of obstacle detection locations, and 5 sets of vehicle pose data.

[0050] These dynamic data sets each have their own second timestamp, and the server associates data with the same second timestamp together. For example, image frames, point cloud frames, obstacle detection locations, and vehicle pose data with timestamps T1-0.1s are combined into a single data packet. In this way, the server ultimately obtains five complete, temporally consecutive data packets.

[0051] In the data processing method provided in this embodiment, when the vehicle is in a first data processing mode, obstacle information of the vehicle is acquired. Then, when the obstacle in the obstacle information belongs to the target type and the distance between it and the vehicle is less than a preset distance threshold, the N frames of dynamic data closest to the first timestamp are acquired and saved. Thus, this application sets the corresponding first data processing mode according to the test requirements. Then, after the required obstacle is detected within the preset distance threshold through the first data processing mode, the corresponding dynamic data is acquired based on the corresponding timestamp and stored. There is no need to store each dynamic data collected by the sensor in real time, thereby reducing the collection of invalid data and improving the overall data utilization rate.

[0052] As an optional embodiment, S102 may specifically include:

[0053] Obtain the second timestamp of the N frames of dynamic data that are closest to the first timestamp and preceding the first timestamp;

[0054] For each dynamic data, dynamic data is obtained from the dynamic data stream when the second timestamp and the first timestamp meet the corresponding time alignment conditions. The dynamic data stream is a data stream composed of dynamic data collected in real time by the vehicle's sensors.

[0055] In this embodiment, the dynamic data stream is a data stream composed of dynamic data collected in real time by various sensors of the vehicle.

[0056] Specifically, the server first obtains the second timestamps of the N frames of dynamic data that are closest to the first timestamp and preceding it. This step can be achieved by traversing the dynamic data stream, comparing the difference between the timestamp of each frame and the first timestamp, and selecting the timestamp of the N frames with the smallest difference as the second timestamp.

[0057] Secondly, for each dynamic data point, it is determined whether the second timestamp and the first timestamp meet the corresponding time alignment conditions. For example, the time alignment condition for the dynamic data closest to the first timestamp can be that the time difference between the acquisition time of the obstacle image frame, the acquisition time of the obstacle point cloud frame, the acquisition time of the obstacle detection position, and the acquisition time of the vehicle pose data and the first timestamp is less than 100 milliseconds.

[0058] Finally, if the time alignment condition is not met, the corresponding dynamic data is discarded; if the time alignment condition is met, the corresponding dynamic data is obtained from the dynamic data stream. Here, the dynamic data stream refers to the data stream composed of dynamic data collected in real time by the vehicle's sensors, including but not limited to obstacle image frames, obstacle point cloud frames, obstacle detection locations, and vehicle pose data.

[0059] This embodiment acquires the N frames of dynamic data closest to the first timestamp of the obstacle information and performs time alignment, ensuring that the most relevant dynamic data to the obstacle information is obtained. This not only improves the efficiency of data acquisition but also guarantees the temporal relevance of the acquired data, thus providing a more accurate and valuable information foundation for subsequent data analysis and processing.

[0060] As an optional embodiment, the dynamic data includes the vehicle pose data;

[0061] After obtaining the second timestamp of the N frames of dynamic data preceding and closest to the first timestamp, the data processing method further includes:

[0062] If the second timestamp and the first timestamp do not meet the time alignment condition, obtain the first pose data and the second pose data that are closest to the target time before and after the target time. The target time is the time corresponding to the vehicle pose data to be obtained.

[0063] Interpolation compensation is performed on the first pose data and the second pose data to obtain the vehicle pose data at the target time.

[0064] In this embodiment, the target time is the time corresponding to the vehicle pose data that needs to be acquired. For example, if the dynamic data to be acquired is one frame, then the target time is the time corresponding to the first timestamp.

[0065] The first pose data is the vehicle pose data closest to the target time before the target time; the second pose data is the vehicle pose data closest to the target time after the target time.

[0066] Specifically, the server first determines whether the second timestamp of the vehicle pose data meets the time alignment condition with the first timestamp. The time alignment condition can be set according to actual needs. For example, a time threshold can be set, and if the time difference between the second timestamp and the first timestamp is less than the threshold, the time alignment condition is considered to be met.

[0067] If the time alignment condition is not met, then the first and second pose data closest to the target time are acquired. The target time is the time corresponding to the vehicle pose data to be acquired, usually the time corresponding to the first timestamp. The first and second pose data represent the vehicle's position and attitude information at different times.

[0068] Then, interpolation compensation is performed on the first and second pose data. Interpolation compensation can employ methods such as linear interpolation, polynomial interpolation, or spline interpolation. Through interpolation compensation, the vehicle pose data at the target time can be estimated.

[0069] Finally, the vehicle pose data at the target time is obtained. This vehicle pose data is obtained through interpolation compensation processing, which can more accurately reflect the vehicle's position and attitude information at the target time.

[0070] As an example, vehicle pose data includes pose position and quaternions. Specifically, the pose position at the target time after interpolation compensation can be determined using the following formula 1:

[0071]

[0072] In Formula 1, pos s The time values ​​are used to represent the pose position at the target time. `pos1` represents the pose position in the first pose data, and `pos2` represents the pose position in the second pose data. `t1` represents the time corresponding to the first pose data, and `t2` represents the time corresponding to the second pose data. s Used to characterize the target time.

[0073] The quaternion at the target time after interpolation compensation can be determined using the following formula 2:

[0074]

[0075] In Formula 2, quat s Quaternions are used to represent the target time. `quat1` represents the quaternion in the first pose data, and `quat2` represents the quaternion in the second pose data. `t1` represents the time corresponding to the first pose data, and `t2` represents the time corresponding to the second pose data. sUsed to characterize the target time, slerp is used to characterize the spherical linear interpolation function.

[0076] This embodiment demonstrates that even if the second timestamp and the first timestamp do not meet the time alignment condition, vehicle pose data at the target time can still be obtained through interpolation compensation. This improves the real-time performance and accuracy of data processing, thereby enhancing overall data processing efficiency.

[0077] As an optional embodiment, prior to S101, the data processing method may further include:

[0078] When the vehicle is in the second data processing mode, the first obstacle detection position of the target obstacle image frame and the second obstacle detection position of the target obstacle point cloud frame are obtained. The target obstacle image frame and the target obstacle point cloud frame are the latest obstacle image frame and obstacle point cloud frame.

[0079] The first obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position;

[0080] Based on the third timestamp of the target obstacle image frame and the fourth timestamp of the target obstacle point cloud frame, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position.

[0081] The fourth obstacle detection location is matched with the second obstacle detection location to obtain the location matching result;

[0082] If the location matching result meets the preset matching conditions, the target obstacle image frame and the target obstacle point cloud frame are stored.

[0083] In this embodiment, the target obstacle image frame is the latest obstacle image frame in the dynamic data stream, and the target obstacle point cloud frame is the latest obstacle point cloud frame in the dynamic data stream.

[0084] The first obstacle detection location is the obstacle position detected in the target obstacle image frame, and the second obstacle detection location is the obstacle position detected in the target obstacle point cloud frame. Specifically, the first obstacle detection location is the obstacle position in the camera coordinate system, and the second obstacle detection location is the obstacle position in the vehicle coordinate system.

[0085] The second data processing mode is a data processing mode that intelligently collects data from areas with weaker detection performance.

[0086] Specifically, when the vehicle is in the second data processing mode, the server first acquires the latest target obstacle image frames and target obstacle point cloud frames. These two types of data come from different sensors such as cameras and LiDAR, so coordinate system transformation and time synchronization are required.

[0087] Next, this application transforms the location of the first obstacle detected by the camera to the vehicle coordinate system. This step involves transformations between multiple coordinate systems, including from the camera coordinate system to the radar coordinate system, and then to the vehicle coordinate system. This transformation unifies the data from different sensors into the same coordinate system, facilitating subsequent comparison and matching.

[0088] Since there may be slight differences in the data acquisition time of different sensors, this application also introduces a time compensation mechanism. Based on the timestamps of the target obstacle image frame and the target obstacle point cloud frame, time compensation is performed on the converted obstacle detection position to obtain a more accurate fourth obstacle detection position. This step can eliminate the position error caused by time differences and further improve the accuracy of the data.

[0089] After completing coordinate transformation and time compensation, this application matches the fourth obstacle detection position with the second obstacle detection position detected by the lidar to obtain a position matching result. For example... Figure 2 The diagram illustrates the matching of obstacle detection positions. When the distance between the center points of the fourth obstacle detection position A and the second obstacle detection position B is less than a set threshold, the fourth obstacle detection position A and the second obstacle detection position B are considered to be successfully matched. When the distance between the center points of the fourth obstacle detection position A and the second obstacle detection position C is greater than the set threshold, the fourth obstacle detection position A and the second obstacle detection position C are considered to be unmatched.

[0090] Finally, if the location matching result meets the preset matching conditions, the target obstacle image frame and the target obstacle point cloud frame are stored. Specifically, there will be multiple sets of first obstacle detection positions in the target obstacle image frame and second obstacle detection positions in the target obstacle point cloud frame. If the proportion of the number of successfully matched detection position groups to the total number of detection position groups is less than a set threshold, it is determined that the preset matching conditions are met at this time.

[0091] This embodiment achieves two key improvements. First, by employing coordinate system transformation and time compensation, precise alignment of data from different sensors is realized, enhancing data consistency and usability. Second, by setting matching conditions for selective storage, storage space usage is significantly reduced while maintaining data quality.

[0092] As an optional embodiment, the first obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position, which may specifically include:

[0093] Based on the first extrinsic parameter matrix, the first obstacle detection position is transformed to the radar coordinate system to obtain the fifth obstacle detection position. The first extrinsic parameter matrix is ​​the transformation matrix between the camera coordinate system and the radar coordinate system.

[0094] Based on the second extrinsic parameter matrix, the fifth obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position. The second extrinsic parameter matrix is ​​the transformation matrix between the radar coordinate system and the vehicle coordinate system.

[0095] In this embodiment, the first extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the camera coordinate system and the radar coordinate system, and the second extrinsic parameter matrix is ​​used to describe the spatial transformation relationship between the radar coordinate system and the vehicle coordinate system.

[0096] Specifically, the server first obtains the first extrinsic parameter matrix and the second extrinsic parameter matrix. The first and second extrinsic parameter matrices can be obtained through calibration during vehicle system initialization.

[0097] Then, the first obstacle detection position is transformed to the radar coordinate system using the following formula 3 to obtain the fifth obstacle detection position:

[0098] P'cam_detect=Tcamera_to_lidar·Pcam_detect Formula 3

[0099] In Formula 3, P'cam_detect is used to characterize the detection position of the fifth obstacle, Tcamera_to_lidar is used to characterize the first extrinsic parameter matrix, and Pcam_detect is used to characterize the detection position of the first obstacle.

[0100] Then, the fifth obstacle detection position is transformed to the vehicle coordinate system using the following formula 4 to obtain the third obstacle detection position:

[0101] P”cam_detect = Tlidar_to_vehicle · P'cam_detect Formula 4

[0102] In Formula 4, P”cam_detect is used to characterize the detection position of the third obstacle, Tlidar_to_vehicle is used to characterize the second extrinsic parameter matrix, and P'cam_detect is used to characterize the detection position of the fifth obstacle.

[0103] In this embodiment, a transformation matrix between the camera coordinate system and the radar coordinate system is used to transform the obstacle position from the camera coordinate system to the radar coordinate system. Then, a transformation matrix between the radar coordinate system and the vehicle coordinate system is used to transform the obstacle position from the radar coordinate system to the final vehicle coordinate system. In this way, an accurate transformation from the camera coordinate system to the vehicle coordinate system can be achieved, which helps to match the obstacle detection position in the subsequent unified coordinate system.

[0104] As an optional embodiment, based on the third timestamp of the target obstacle image frame and the fourth timestamp of the target obstacle point cloud frame, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position, which may specifically include:

[0105] Based on the first rotation matrix of the camera and the second rotation matrix of the radar, determine the rotation transformation matrix from the camera to the radar;

[0106] Based on the rotation transformation matrix, a time transformation matrix is ​​constructed between the third and fourth timestamps;

[0107] Based on the time transformation matrix, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position.

[0108] In this embodiment, firstly, based on the first rotation matrix of the camera and the second rotation matrix of the radar, the rotation transformation matrix from the camera to the radar is determined. This step takes into account the installation position and orientation of the camera and radar on the vehicle, and establishes the spatial transformation relationship between them by calculating the rotational relationship between the two sensors.

[0109] Secondly, based on the rotation transformation matrix, a time transformation matrix is ​​constructed between the third and fourth timestamps. This step combines spatial transformation relationships with temporal factors, creating a matrix capable of transformation between different time points. The time transformation matrix considers the possible movements of the vehicle between two time points, thus enabling more accurate prediction of the obstacle's position at different time points.

[0110] Finally, based on the time transformation matrix, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position. By applying the time transformation matrix, this application can transform the obstacle position detected by the camera from its original time point to a position consistent with the radar detection time point, achieving time synchronization of data from different sensors.

[0111] Specifically, the rotation transformation matrix from camera to radar can be determined using the following formula 5:

[0112]

[0113] In Formula 5, Rlidar_to_camera R is used to characterize the rotation transformation matrix from camera to radar. lidar The second rotation matrix used to characterize the radar The transpose of the first rotation matrix used to characterize the camera.

[0114] The time transformation matrix between the third and fourth timestamps can be determined using the following formula 6:

[0115]

[0116] In Formula 6, T t_camera_to_t_lidar R is used to characterize the time transformation matrix between the third and fourth timestamps. lidar_to_camera The rotation transformation matrix t is used to characterize the transformation from camera to radar. lidar_to_camera Used to characterize the translation vector from the camera to the radar.

[0117] The translation vector from the camera to the radar can be determined using the following formula 7:

[0118]

[0119] In Formula 7, t lidar_to_camera R is used to characterize the translation vector from the camera to the radar. lidar_to_camera Used to characterize the rotation transformation matrix from camera to radar. p is the transpose of the first rotation matrix used to characterize the camera. lidar p is used to characterize the position vector of the lidar. camera A vector used to represent the position of the camera.

[0120] Finally, the location of the fourth obstacle detection can be determined using the following formula 8:

[0121] Pt_lidarcam_detect = Tt_camera_to_t_lidar · P” cam_detect Formula 8

[0122] In Formula 8, Pt_lidarcam_detect is used to characterize the detection location of the fourth obstacle, T t_camera_to_t_lidar The time transformation matrix is ​​used to characterize the time between the third and fourth timestamps, and P”cam_detect is used to characterize the location of the third obstacle detection.

[0123] This embodiment introduces rotation and time transformation matrices to achieve time alignment and position compensation for data collected from different sensors. This method considers the spatial relationship and temporal differences between the camera and radar, thus enabling more accurate transformation of obstacle detection positions from one point in time to another.

[0124] As an optional embodiment, prior to S101, the data processing method further includes:

[0125] Obtain obstacle information for the vehicle when it is in the third data processing mode;

[0126] If the distance between the obstacle and the vehicle in the obstacle information is less than a preset distance threshold, acquire dynamic data corresponding to the first timestamp of the obstacle information;

[0127] Store dynamic data corresponding to the first timestamp of the obstacle information.

[0128] In this embodiment, the third data processing mode is a mode that stores all dynamic data in the scene within a preset distance threshold range. Compared with the first data processing mode, the third data processing mode does not need to consider obstacle types; dynamic data of all obstacle types in the scene within the preset distance threshold range will be stored.

[0129] Specifically, first, the vehicle enters a third data processing mode. This mode may be automatically triggered by the onboard system or manually selected by the driver. Simultaneously, the vehicle's sensors continuously acquire obstacle information from the surrounding environment. This information may include the type, location, and speed of obstacles.

[0130] The server then determines whether the distance between the detected obstacle and the vehicle is less than a preset distance threshold. This threshold can be dynamically adjusted based on factors such as vehicle speed and road conditions to ensure safety. If the distance between the obstacle and the vehicle is less than the preset distance threshold, the server immediately retrieves dynamic data corresponding to the timestamp of the obstacle information.

[0131] Finally, the server stores this dynamic data. One method is to associate and store dynamic data with the same second timestamp.

[0132] This embodiment selectively stores dynamic data based on the distance between obstacles and vehicles, effectively improving the efficiency and quality of data storage. Specifically, when the distance between an obstacle and a vehicle is detected to be less than a preset distance threshold, it indicates that the obstacle may affect the vehicle's driving safety. In this case, acquiring and storing dynamic data corresponding to the timestamp of the obstacle information ensures the capture of critical environmental information.

[0133] As an optional embodiment, prior to S101, the data processing method may further include:

[0134] Obtain data processing configuration information;

[0135] Based on the data processing configuration information, the vehicle's data processing mode is determined.

[0136] In this embodiment, the data processing configuration information used to characterize the data processing mode can be obtained in various ways. For example, the data processing configuration information can be input through a human-machine interface, or it can be set in a parameter file on the vehicle side.

[0137] After obtaining the data processing configuration information, the server determines the vehicle's data processing mode based on this information. This process involves parsing the configuration information, matching preset mode types, or dynamically generating a data processing mode based on configuration parameters. The determined data processing mode will dictate the specific strategies and procedures for subsequent data processing.

[0138] In one preferred implementation, the data processing configuration information may include multiple parameters, such as data acquisition frequency and storage strategy. The server can comprehensively determine the data processing mode based on these parameters.

[0139] Furthermore, this application also enables dynamic adjustment. During vehicle operation, the data processing configuration information can be updated based on real-time conditions, thereby achieving dynamic switching of data processing modes. This dynamic adjustment capability allows the vehicle to better adapt to changing environments and demands.

[0140] This embodiment acquires data processing configuration information and determines the vehicle's data processing mode based on this information, enabling flexible configuration of the vehicle's data processing mode. Thus, this method can dynamically adjust the vehicle's data processing mode according to different scenarios and needs, improving adaptability and efficiency.

[0141] As an optional embodiment, obtaining data processing configuration information includes any of the following:

[0142] The configuration interface is displayed, which includes an information input box for inputting data processing configuration information;

[0143] In response to information input operations on the configuration interface, data processing configuration information is obtained;

[0144] Or receive data processing configuration information sent by the server.

[0145] In this embodiment, by displaying a configuration interface and providing information input boxes, users can directly configure the system on the vehicle. The system responds to input operations on the configuration interface and can obtain users' configuration requirements in real time. By receiving configuration information sent by the server, configuration and unified management can be achieved through a remote human-machine interface. These methods complement each other and adapt to different usage scenarios and needs.

[0146] Specifically, the configuration interface can be displayed on an in-vehicle display screen. The configuration interface can include multiple options, such as data processing mode selection and data acquisition frequency settings. Information input fields can be in the form of text input boxes, drop-down menus, or sliders, allowing users to easily input or select the required configuration parameters.

[0147] Responding to input on the configuration interface provides immediate feedback to user interaction. For example, when a user selects a data processing mode on the configuration interface, the server immediately records this data processing configuration information and applies it to subsequent data processing steps. This approach improves configuration efficiency and reduces operation latency.

[0148] Receiving configuration information sent by the server is suitable for scenarios requiring unified management or remote control. For example, a fleet management system can send unified data processing configuration information to multiple vehicles simultaneously through the server to achieve batch configuration.

[0149] As an example of a scenario, such as Figure 3 The diagram illustrates a data processing method. First, the server obtains data processing configuration information from a remote server via S301, or from the vehicle's configuration interface via S302. Then, in S303, the server selects the data processing mode based on the configuration information.

[0150] Then, for static data, it is stored directly through S304; for dynamic data, if the selected data processing mode is the first data processing mode, the dynamic data is stored using the first data processing mode through S305; if the selected data processing mode is the first data processing mode, the dynamic data is stored using the second data processing mode through S306; if the selected data processing mode is the third data processing mode, the dynamic data is stored using the third data processing mode through S307.

[0151] This embodiment provides multiple methods for obtaining configuration information, retaining the convenience of local configuration while introducing the flexibility of remote configuration. This diversified configuration approach can adapt to different use cases and needs.

[0152] Based on the data processing method provided in this application, correspondingly, this application also provides specific embodiments of a data processing apparatus.

[0153] like Figure 4 As shown, the data processing device 400 provided in this application embodiment includes an obstacle acquisition module 410, a data acquisition module 420, and a data storage module 430.

[0154] The obstacle acquisition module 410 is used to acquire obstacle information of the vehicle when the vehicle is in a first data processing mode, the obstacle information including at least one obstacle.

[0155] The data acquisition module 420 is used to acquire N frames of dynamic data that are closest to the first time stamp and before the first time stamp, based on the first time stamp corresponding to the obstacle information, when the obstacle belongs to the target type and the distance between it and the vehicle is less than a preset distance threshold. N is a positive integer. The dynamic data includes at least one of obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data.

[0156] The data storage module 430 is used to associate and store dynamic data with the same second timestamp based on the second timestamp of the dynamic data.

[0157] As an optional embodiment, the data storage module 430 is specifically used for:

[0158] Obtain the second timestamp of the N frames of dynamic data that are closest to the first timestamp and preceding the first timestamp;

[0159] For each dynamic data, dynamic data is obtained from the dynamic data stream when the second timestamp and the first timestamp meet the corresponding time alignment conditions. The dynamic data stream is a data stream composed of dynamic data collected in real time by the vehicle's sensors.

[0160] As an optional embodiment, the dynamic data includes the vehicle pose data;

[0161] The data storage module 430 is further used to: obtain the second timestamp of the N frames of dynamic data that are closest to the first timestamp and preceding the first timestamp;

[0162] If the second timestamp and the first timestamp do not meet the time alignment condition, obtain the first pose data and the second pose data that are closest to the target time before and after the target time. The target time is the time corresponding to the vehicle pose data to be obtained.

[0163] Interpolation compensation is performed on the first pose data and the second pose data to obtain the vehicle pose data at the target time.

[0164] As an optional embodiment, before acquiring obstacle information about the vehicle while the vehicle is in the first data processing mode, the data processing device 400 further includes the following modules:

[0165] The location acquisition module is used to acquire the first obstacle detection position of the target obstacle image frame and the second obstacle detection position of the target obstacle point cloud frame when the vehicle is in the second data processing mode. The target obstacle image frame and the target obstacle point cloud frame are the latest obstacle image frame and obstacle point cloud frame.

[0166] The position transformation module is used to transform the first obstacle detection position to the vehicle coordinate system to obtain the third obstacle detection position;

[0167] The position compensation module is used to perform time compensation on the third obstacle detection position based on the third timestamp of the target obstacle image frame and the fourth timestamp of the target obstacle point cloud frame, so as to obtain the fourth obstacle detection position.

[0168] The position matching module is used to match the fourth obstacle detection position with the second obstacle detection position to obtain the position matching result;

[0169] The data storage module 430 is also used to store the target obstacle image frame and the target obstacle point cloud frame when the position matching result meets the preset matching conditions.

[0170] As an optional embodiment, the position conversion module is specifically used for:

[0171] Based on the first extrinsic parameter matrix, the first obstacle detection position is transformed to the radar coordinate system to obtain the fifth obstacle detection position. The first extrinsic parameter matrix is ​​the transformation matrix between the camera coordinate system and the radar coordinate system.

[0172] Based on the second extrinsic parameter matrix, the fifth obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position. The second extrinsic parameter matrix is ​​the transformation matrix between the radar coordinate system and the vehicle coordinate system.

[0173] As an optional embodiment, the position compensation module is specifically used for:

[0174] Based on the first rotation matrix of the camera and the second rotation matrix of the radar, determine the rotation transformation matrix from the camera to the radar;

[0175] Based on the rotation transformation matrix, a time transformation matrix is ​​constructed between the third and fourth timestamps;

[0176] Based on the time transformation matrix, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position.

[0177] As an optional embodiment, before acquiring obstacle information about the vehicle while the vehicle is in the first data processing mode, the data processing device 400 further includes the following modules:

[0178] The obstacle acquisition module 410 is also used to acquire obstacle information of the vehicle when the vehicle is in the third data processing mode;

[0179] The data acquisition module 420 is also used to acquire dynamic data corresponding to the first timestamp of the obstacle information when the distance between the obstacle and the vehicle in the obstacle information is less than a preset distance threshold.

[0180] The data storage module 430 is also used to store dynamic data corresponding to the first timestamp of the obstacle information.

[0181] As an optional embodiment, before acquiring obstacle information about the vehicle while the vehicle is in the first data processing mode, the data processing device 400 further includes the following modules:

[0182] The information acquisition module is used to acquire data processing configuration information;

[0183] The mode determination module is used to determine the vehicle's data processing mode based on the data processing configuration information.

[0184] As an optional embodiment, the information acquisition module is specifically used for:

[0185] The configuration interface is displayed, which includes an information input box for inputting data processing configuration information;

[0186] In response to information input operations on the configuration interface, data processing configuration information is obtained;

[0187] Or receive data processing configuration information sent by the server.

[0188] Figure 5 A schematic diagram of the hardware structure of the data processing device provided in an embodiment of this application is shown.

[0189] The data processing device may include a processor 501 and a memory 502 storing computer program instructions.

[0190] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0191] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0192] The processor 501 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0193] In one example, the data processing device may further include a communication interface 503 and a bus 510. Wherein, as... Figure 3 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0194] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0195] Bus 510 includes hardware, software, or both, that couples components of a data processing device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0196] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments.

[0197] In addition, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs the data processing method provided by any aspect of the above embodiments of this application.

[0198] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0199] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0200] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0201] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0202] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: When the vehicle is in a first data processing mode, obstacle information of the vehicle is acquired, and the obstacle information includes at least one obstacle. When the obstacle is a target type and the distance between it and the vehicle is less than a preset distance threshold, based on the first timestamp corresponding to the obstacle information, obtain N frames of dynamic data that are closest to the first timestamp and before the first timestamp, where N is a positive integer, and the dynamic data includes at least one of obstacle image frames, obstacle point cloud frames, obstacle detection positions, and vehicle pose data. Based on the second timestamp of the dynamic data, the dynamic data with the same second timestamp are associated and stored.

2. The method according to claim 1, characterized in that, The step of obtaining N frames of dynamic data that are closest to the first timestamp before the first timestamp and based on the first timestamp corresponding to the obstacle information includes: Obtain the second timestamp of the dynamic data in the N frames preceding the first timestamp and closest to the first timestamp; For each of the aforementioned dynamic data, the dynamic data is obtained from the dynamic data stream when the second timestamp and the first timestamp satisfy the corresponding time alignment conditions. The dynamic data stream is a data stream composed of dynamic data collected in real time by the vehicle's sensors.

3. The method according to claim 2, characterized in that, The dynamic data includes the vehicle pose data; After obtaining the second timestamp of the dynamic data in the N frames preceding the first timestamp and closest to the first timestamp, the method further includes: If the second timestamp and the first timestamp do not meet the time alignment condition, the first pose data and the second pose data that are closest to the target time before and after the target time are obtained, where the target time is the time corresponding to the vehicle pose data to be obtained. Interpolation compensation is performed on the first pose data and the second pose data to obtain the vehicle pose data at the target time.

4. The method according to claim 1, characterized in that, Before acquiring obstacle information of the vehicle while the vehicle is in the first data processing mode, the method further includes: When the vehicle is in the second data processing mode, the first obstacle detection position of the target obstacle image frame and the second obstacle detection position of the target obstacle point cloud frame are obtained, wherein the target obstacle image frame and the target obstacle point cloud frame are the latest obstacle image frame and the obstacle point cloud frame. The first obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position; Based on the third timestamp of the target obstacle image frame and the fourth timestamp of the target obstacle point cloud frame, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position. The fourth obstacle detection position is matched with the second obstacle detection position to obtain a position matching result; If the location matching result meets the preset matching conditions, the target obstacle image frame and the target obstacle point cloud frame are stored.

5. The method according to claim 4, characterized in that, The step of transforming the first obstacle detection position to the vehicle coordinate system to obtain the third obstacle detection position includes: Based on the first extrinsic parameter matrix, the first obstacle detection position is transformed to the radar coordinate system to obtain the fifth obstacle detection position. The first extrinsic parameter matrix is ​​the transformation matrix between the camera coordinate system and the radar coordinate system. Based on the second extrinsic parameter matrix, the fifth obstacle detection position is transformed to the vehicle coordinate system to obtain the third obstacle detection position. The second extrinsic parameter matrix is ​​the transformation matrix between the radar coordinate system and the vehicle coordinate system.

6. The method according to claim 4, characterized in that, The step of performing time compensation on the third obstacle detection position based on the third timestamp of the target obstacle image frame and the fourth timestamp of the target obstacle point cloud frame to obtain the fourth obstacle detection position includes: Based on the first rotation matrix of the camera and the second rotation matrix of the radar, determine the rotation transformation matrix from the camera to the radar; Based on the rotation transformation matrix, a time transformation matrix is ​​constructed between the third timestamp and the fourth timestamp; Based on the time transformation matrix, time compensation is performed on the third obstacle detection position to obtain the fourth obstacle detection position.

7. The method according to claim 1, characterized in that, Before acquiring obstacle information of the vehicle while the vehicle is in the first data processing mode, the method further includes: When the vehicle is in the third data processing mode, the obstacle information of the vehicle is acquired; If the distance between the obstacle and the vehicle in the obstacle information is less than a preset distance threshold, dynamic data corresponding to the first timestamp of the obstacle information is acquired; The dynamic data corresponding to the first timestamp of the obstacle information is stored.

8. The method according to any one of claims 1-7, characterized in that, Before acquiring obstacle information of the vehicle while the vehicle is in the first data processing mode, the method further includes: Obtain data processing configuration information; Based on the data processing configuration information, the data processing mode of the vehicle is determined.

9. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-8.

Images