Vehicle data processing method and device, electronic equipment, vehicle and medium

Abstract

The embodiment of the application provides a kind of vehicle data processing method, device, electronic equipment, vehicle and medium, the method comprises: according to the radar scanning period of the vehicle-mounted radar of target vehicle and the image delivery time length of vehicle-mounted shooting device, determine the image release timestamp of vehicle-mounted shooting device;According to image release timestamp and image delivery time length determine scene fusion time, and obtain target image data and target point cloud data of scene fusion time;Target image data and target point cloud data are fused to scene, and the synchronous scene of scene fusion time is obtained;Scene motion offset processing is carried out to synchronous scene, and target scene data is obtained.The application improves the accuracy of synchronization time, avoids image data over motion compensation, improves the precision of synchronous scene;Through scene motion offset processing, the accuracy of vehicle perception surrounding environment is improved, without model motion compensation, reduce the time consumption of model inference.

Description

Technical Field

[0001] This application relates to the field of vehicle data processing, and more particularly to a vehicle data processing method, apparatus, electronic device, vehicle, and medium. Background Technology

[0002] In autonomous driving technology, the environment in which vehicles operate is often very complex. However, in existing technologies, due to differences in the environmental data acquisition and transmission times between different devices on the vehicle, the fused scene data has significant errors, resulting in low accuracy in the vehicle's perception of its surroundings. Summary of the Invention

[0003] This application provides a vehicle data processing method, apparatus, electronic device, vehicle, and medium to solve the technical problem in the prior art where the difference in environmental acquisition time and data transmission time between different devices on a vehicle leads to large errors in the fused scene data.

[0004] A vehicle data processing method, comprising: The image release timestamp of the vehicle-mounted camera is determined based on the radar scanning cycle of the vehicle's onboard radar and the image transmission time of the onboard camera. The scene fusion time is determined based on the image release timestamp and the image transmission duration, and the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time, are obtained. Scene fusion is performed on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion; The synchronous scene is processed by scene motion offset to obtain the target scene data of the target vehicle.

[0005] In the vehicle data processing method, the image release timestamp of the onboard imaging device is calculated by using the radar scanning cycle of the target vehicle's onboard radar and the image transmission time of the onboard imaging device. This allows for the determination of the scene fusion moment, thereby improving the accuracy of synchronization time. Based on the target image data corresponding to the onboard imaging device at the scene fusion moment and the target point cloud data corresponding to the onboard radar at the scene fusion moment, synchronization of the target image data and target point cloud data at the scene fusion moment is achieved. This enables the fusion of two-dimensional image data and three-dimensional point cloud data, thus avoiding over-motion compensation of image data and improving the accuracy of scene synchronization. Then, by performing scene motion offset processing on the synchronized scene, prediction of the target scene data is achieved, improving the accuracy of the vehicle's perception of the surrounding environment. Since motion compensation is not required by the model, the inference time of the model is reduced.

[0006] Further, the step of performing scene motion offset processing on the synchronized scene to obtain target scene data for the target vehicle includes: Obtain the prediction inference time corresponding to the intelligent driving model, wherein the prediction inference time is the predicted time required for the intelligent driving model to generate control signals; The synchronous scene is processed by scene motion offsetting based on scene fusion time, time deviation, and prediction inference duration to obtain target scene data of the target vehicle; the time deviation is the deviation between the preset release timestamp and the scene fusion time.

[0007] This embodiment performs scene motion offset processing on the synchronous scene by considering the scene fusion time, time deviation, and prediction inference duration. This achieves motion compensation for the synchronous scene and acquisition of target scene data, thereby enabling the prediction of scene data at the time when the intelligent driving model outputs control signals, and thus improving the accuracy of the target scene data.

[0008] Furthermore, the step of performing scene motion offset processing on the synchronized scene using scene fusion time, time deviation, and prediction inference duration to obtain target scene data for the target vehicle includes: The scene fusion time, time deviation and prediction inference time are summed to obtain the estimated time, which is the time when the predicted intelligent driving model outputs the control signal; The synchronous scene at the time of scene fusion is extended to the estimated time using the dead reckoning algorithm to obtain the target scene data corresponding to the estimated time.

[0009] This embodiment calculates the estimated time by summing the scene fusion time, time deviation, and prediction inference time, thereby enabling the prediction of when the intelligent driving model outputs control signals. By extending the synchronous scene of the scene fusion time to the estimated time through dead reckoning, it provides scene data for predicting when the intelligent driving model outputs control signals, avoiding motion compensation by the intelligent driving model and reducing the inference time of the intelligent driving model.

[0010] Furthermore, obtaining the prediction inference time corresponding to the intelligent driving model includes: Obtain multiple historical reasoning durations corresponding to the intelligent driving model, wherein the historical reasoning duration is the time required for the intelligent driving model to generate historical control information; The predicted inference time of the intelligent driving model is obtained by using the least squares method to predict the inference time of all the historical inference times.

[0011] This embodiment uses the least squares method to predict the duration of all historical inferences, thereby predicting the duration of control signals generated by the intelligent driving model. This avoids large errors caused by scene shifts due to different inference durations each time, and thus improves the accuracy of predicting the inference duration.

[0012] Further, acquiring the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion moment includes: Acquire all point cloud data and their corresponding point cloud timestamps obtained by the vehicle-mounted radar scanning the surrounding environment within the radar scanning cycle. Based on the scene fusion time, the point cloud data corresponding to each point cloud timestamp is offset by point cloud motion to obtain the target point cloud data corresponding to the scene fusion time.

[0013] This embodiment performs point cloud motion offset on the point cloud data corresponding to each point cloud timestamp at the scene fusion time, realizing the offset of the point cloud data at each timestamp, so that all point cloud data are at the scene fusion time, thereby realizing the acquisition of the target point cloud data at the scene fusion time, and thus improving the accuracy of subsequent scene synchronization.

[0014] Further, the step of performing point cloud motion offset on the point cloud data corresponding to each point cloud timestamp based on the scene fusion time to obtain the target point cloud data corresponding to the scene fusion time includes: By using the dead reckoning algorithm, the point cloud data corresponding to each point cloud timestamp is extended and predicted to the scene fusion time, thus obtaining the target point cloud data at the scene fusion time.

[0015] This embodiment uses the dead reckoning algorithm to extend and predict the point cloud data corresponding to each point cloud timestamp to the scene fusion time, which improves the accuracy of point cloud data offset and ensures that all point cloud data are at the scene fusion time, thereby improving the accuracy of subsequent scene synchronization.

[0016] Furthermore, after performing scene motion offset processing on the synchronized scene to obtain the target scene data of the target vehicle, the process further includes: The intelligent driving model generates control signals from the target scene data to obtain the direction control signals for the target vehicle.

[0017] This embodiment uses the intelligent driving model to infer control signals from the target scene data, avoiding motion compensation of the scene data by the intelligent driving model, reducing the inference time of the intelligent driving model, and improving the accuracy of the intelligent driving model's inference.

[0018] A vehicle data processing device, comprising: The timestamp publishing module is used to determine the image publishing timestamp of the vehicle-mounted camera based on the radar scanning cycle of the vehicle's onboard radar and the image transmission duration of the onboard camera. The time data acquisition module is used to determine the scene fusion time based on the image release timestamp and the image transmission duration, and to acquire the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time; The scene data fusion module is used to perform scene fusion on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion. The scene motion offset module is used to perform scene motion offset processing on the synchronized scene at the scene fusion time to obtain the target scene data of the target vehicle.

[0019] An electronic device includes a controller and a memory, wherein, Memory, used to store computer programs; The controller is used to execute the program stored in the memory to implement the above-mentioned vehicle data processing method.

[0020] A vehicle includes an onboard radar, an onboard camera, and the aforementioned electronic equipment.

[0021] A computer-readable storage medium storing a computer program, which, when executed by a controller, implements the above-described vehicle data processing method. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a vehicle data processing method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating step S12 of a vehicle data processing method provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating step S14 of a vehicle data processing method provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating step S141 of a vehicle data processing method provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating step S142 of a vehicle data processing method provided in an embodiment of the present invention; Figure 6This is a schematic diagram of the structure of a vehicle data processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In one embodiment, such as Figure 1 As shown, a vehicle data processing method is provided, which is applied to, for example... Figure 7 The controller includes the following steps S11-S14: S11. Determine the image release timestamp of the vehicle-mounted camera based on the radar scanning cycle of the target vehicle's onboard radar and the image transmission duration of the onboard camera.

[0027] S12. Determine the scene fusion time based on the image release timestamp and the image transmission duration, and obtain the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time.

[0028] S13. Perform scene fusion on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion.

[0029] S14. Perform scene motion offset processing on the synchronized scene to obtain the target scene data of the target vehicle.

[0030] The target vehicle can refer to a gasoline-powered vehicle, a natural gas-powered vehicle, an electric vehicle, or a hybrid vehicle. For example, the vehicle could be an electric bus or a gasoline-powered sedan. The target vehicle is equipped with electronic devices, intelligent driving models, vehicle-mounted radar, and vehicle-mounted cameras. The vehicle-mounted radar can be lidar, millimeter-wave radar, etc., and there can be one or more radars. The radar scan cycle of the vehicle-mounted radar is the time required for the radar to scan one frame. Different types of vehicle-mounted radar have different radar scan cycles; for example, the radar scan cycle for an MX galvanometer to scan one frame of point cloud is approximately 80ms, while the radar scan cycle for an AT128 galvanometer to scan one frame of point cloud is approximately 55ms. The vehicle-mounted camera can be a camera or other photographic equipment, such as a panoramic camera, a surround-view camera, or a fisheye camera, and there can be one or more such cameras. Image transmission duration can be defined as the time it takes for the onboard camera to transmit captured image data to the controller, or it can be the total time taken for the image data to be transmitted from the onboard camera to the controller and then processed within the controller. For example, after the onboard camera captures surrounding data through exposure, the time it takes to transmit the data to the controller via Low Voltage Differential Signaling (LVDS) is approximately 5 milliseconds. The controller then performs image processing (ISP), distortion correction, cropping, and scaling, transforming the data into usable input data for the intelligent driving model, which takes approximately 10 milliseconds. Thus, the image transmission duration is approximately 15 milliseconds. Image release timestamp refers to the moment the onboard camera transmits image data to the controller. Specifically, when the preset release timestamps (pre-set timestamps corresponding to the onboard radar) are 100 milliseconds, 200 milliseconds, and 300 milliseconds, and the radar scan cycle is 80 milliseconds with an image transmission duration of 15 milliseconds, the image release timestamps are 95 milliseconds, 195 milliseconds, and 295 milliseconds, respectively.

[0031] The scene fusion time is used to fuse data from at least one radar and at least one imaging device at that moment. For example, since the radar scanning cycle of an automotive radar is relatively long, usually longer than the image transmission time, the scene fusion time can be set to the moment when the automotive radar completes its scan. When there are multiple automotive radars, the moment when the radar with the longest scanning time completes its scan is used as the scene fusion time. Target image data is environmental information captured by the automotive imaging device at the scene fusion time. Target point cloud data is new point cloud data obtained by predicting the point cloud data at each time point (i.e., point cloud timestamps, such as the first millisecond, the second millisecond, etc.) within the radar scanning cycle to the scene fusion time. Here, "expansion" refers to performing trajectory expansion prediction on the point cloud data corresponding to the point cloud timestamp according to certain prediction rules, thereby predicting the new point cloud data that will change and form at the scene fusion time. Point cloud data is a set of discrete points generated by the automotive radar scanning the surrounding environment every millisecond within the radar scanning cycle. The point cloud timestamp is the scanning time corresponding to each point cloud data, such as the first millisecond and its point cloud data, the second millisecond and its point cloud data, etc.

[0032] The synchronized scene refers to the scene in which the target vehicle is located at the moment of scene fusion, obtained by fusing target image data and target point cloud data. The target scene data is new scene data obtained by extending the prediction to subsequent time steps based on the synchronized scene. The extended prediction can be an operation at the millisecond level.

[0033] As an example, in step S11, after the target vehicle starts or during operation, the controller acquires relevant information about the vehicle-mounted radar and vehicle-mounted imaging equipment installed on the target vehicle, and determines the radar scanning cycle of the vehicle-mounted radar and the image transmission duration of the vehicle-mounted imaging equipment. Then, the controller calculates the sum of the radar scanning cycle of the vehicle-mounted radar and the image transmission duration of the vehicle-mounted imaging equipment to obtain the image release timestamp of the vehicle-mounted imaging equipment. For example, if the radar scanning cycle of the vehicle-mounted radar is 80 milliseconds and the image transmission duration of the vehicle-mounted imaging equipment is 15 milliseconds, then the image release timestamp is the 95th millisecond.

[0034] As an example, in step S12, the controller infers the scene fusion time using the image release timestamp. Specifically, it subtracts the image transmission time from the image release timestamp to obtain the scene fusion time. For instance, if the controller receives image data at 95 milliseconds, it needs to subtract the 15 milliseconds of image transmission time, thus obtaining a scene fusion time of 80 milliseconds. This means the image data needs to be captured at 80 milliseconds. Next, the controller controls the onboard camera to capture the surrounding environment at the scene fusion time, thereby obtaining the target image data and the target point cloud data for the scene fusion time after processing all point cloud data.

[0035] As an example, in step S13, after acquiring the target image data and target point cloud data at the scene fusion time, the controller performs scene fusion on the target image data and target point cloud data, that is, acquires a scene fusion model, and inputs the target image data and target point cloud data into the scene fusion model. The scene fusion model extracts features from the two-dimensional target image data and the three-dimensional target point cloud data, extracting the two-dimensional image features corresponding to the target image data and the three-dimensional point cloud features corresponding to the target point cloud data. Then, the feature fusion layer performs feature fusion on the two-dimensional image features and the three-dimensional point cloud features, and uses a self-attention mechanism to capture the dependency of the fused scene, so as to capture the inherent correlation and dependency relationship between the two-dimensional image features and the three-dimensional point cloud features, so that the scene fusion model can better fuse multimodal features, thereby obtaining the synchronous scene of the target vehicle at the scene fusion time. For example, the image data captured by the vehicle-mounted camera at the 80th millisecond and the point cloud data scanned by the vehicle-mounted radar are fused through the scene fusion model to obtain the scene in which the target vehicle is located at the 80th millisecond.

[0036] In another example, to reduce the amount of data and make scene fusion easier, the controller filters intermediate image data and intermediate point cloud data at a certain height (which can be road surface data, i.e., zero height) from the target image data and target point cloud data respectively. Then, the intermediate image data and intermediate point cloud data are fused using a scene fusion model to obtain the synchronous scene of the target vehicle at the moment of scene fusion.

[0037] As an example, in step S14, after obtaining the synchronized scene at the scene fusion time, the controller performs scene motion offset processing on the synchronized scene, that is, it extends the prediction backward on the synchronized scene at the scene fusion time, that is, it predicts the scene for a future period of time based on the synchronized scene, thereby obtaining the target scene data of the target vehicle, so that the reference information is more accurate when the intelligent driving model generates control signals. Preferably, the prediction backward on the synchronized scene at the scene fusion time is extended to the scene at the time when the intelligent driving model outputs the control signal.

[0038] In this embodiment, the vehicle data processing method calculates the image release timestamp of the vehicle-mounted imaging device by using the radar scanning cycle of the target vehicle's onboard radar and the image transmission duration of the onboard imaging device. This determines the scene fusion time and improves the accuracy of synchronization time. Based on the target image data corresponding to the onboard imaging device at the scene fusion time and the target point cloud data corresponding to the onboard radar at the scene fusion time, synchronization of the target image data and target point cloud data at the scene fusion time is achieved. This enables the fusion of two-dimensional image data and three-dimensional point cloud data, avoiding excessive motion compensation of image data and improving the accuracy of scene synchronization. Then, by performing scene motion offset processing on the synchronized scene, prediction of the target scene data is achieved, improving the accuracy of the vehicle's perception of the surrounding environment. No motion compensation is required from the model, reducing the time consumed by model inference.

[0039] In one embodiment, acquiring the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion moment includes: S121. Obtain all point cloud data and their corresponding point cloud timestamps obtained by the vehicle-mounted radar scanning the surrounding environment within the radar scanning cycle.

[0040] S122. Perform point cloud motion offset on the point cloud data corresponding to each point cloud timestamp according to the scene fusion time to obtain the target point cloud data corresponding to the scene fusion time.

[0041] Point cloud data consists of discrete point sets generated by the vehicle-mounted radar every millisecond during its scan cycle, scanning the surrounding environment. Point cloud timestamps represent the scan time corresponding to each point cloud data point, such as the first millisecond and its point cloud data, the second millisecond and its point cloud data, and so on. Point cloud motion offset is the operation of backward expansion and prediction of the point cloud data at each point cloud timestamp.

[0042] As an example, in step S121, after determining the scene fusion time, all point cloud data and their corresponding point cloud timestamps obtained by the vehicle-mounted radar scanning the surrounding environment within the radar scan cycle are acquired. Specifically, scanning begins and the first point cloud timestamp is recorded as the first millisecond. The surrounding environment data collected between 0 and 1 millisecond is identified as the point cloud data corresponding to the first point cloud timestamp. The second millisecond is then recorded as the second point cloud timestamp, and the surrounding environment data collected between 1 and 2 milliseconds is identified as the point cloud data corresponding to the second point cloud timestamp. This process continues until all point cloud data and their corresponding point cloud timestamps are acquired after the vehicle-mounted radar has completed its radar scan cycle. For example, when the radar scan cycle is 80 milliseconds, all point cloud data actually consists of point cloud data scanned from 0 to 80 milliseconds, and the number of point cloud data is typically around 5 points.

[0043] As an example, in step S122, the controller performs point cloud motion offset on the point cloud data corresponding to each point cloud timestamp according to the scene fusion time. That is, the point cloud data corresponding to each point cloud timestamp is extended and predicted to the scene fusion time, that is, the point cloud data corresponding to the first millisecond is extended and predicted to the scene fusion time, and the point cloud data corresponding to the second millisecond is extended and predicted to the scene fusion time, until the point cloud data corresponding to the millisecond before the vehicle radar completes scanning is extended and predicted to the scene fusion time, thereby obtaining the target point cloud data corresponding to the vehicle radar at the scene fusion time. For example, when the preset release timestamp corresponding to the vehicle radar is reached (such as the 100th millisecond, 200th millisecond, etc.), after the controller receives the point cloud data corresponding to each point cloud timestamp, it predicts the point cloud data of the first millisecond, thereby predicting the new point cloud data that has changed in the first millisecond at the 80th millisecond (scene fusion time), until the point cloud data of the 79th millisecond is predicted to the new point cloud data that has changed in the 79th millisecond at the 80th millisecond (scene fusion time). In this way, the target point cloud data at the moment of scene fusion can be obtained.

[0044] This embodiment performs point cloud motion offset on the point cloud data corresponding to each point cloud timestamp at the scene fusion time, realizing the offset of the point cloud data at each timestamp, so that all point cloud data are at the scene fusion time, thereby realizing the acquisition of the target point cloud data at the scene fusion time, and thus improving the accuracy of subsequent scene synchronization.

[0045] In one embodiment, the step of performing point cloud motion offset on the point cloud data corresponding to each point cloud timestamp based on the scene fusion time to obtain the target point cloud data corresponding to the scene fusion time includes: S1221. Using the dead reckoning algorithm, the point cloud data corresponding to each point cloud timestamp is extended and predicted to the scene fusion time to obtain the target point cloud data at the scene fusion time.

[0046] The dead reckoning algorithm is a navigation and positioning method that calculates the target's position by measuring the distance traveled and its orientation. Specifically, it combines information from multiple sensors, including vehicle speed, direction of travel, vehicle acceleration on inclines and declines, and air pressure, to calculate the vehicle's trajectory using mathematical models and algorithms. In this embodiment, the dead reckoning algorithm is used to perform the operation of extending and predicting point cloud data up to the scene fusion time.

[0047] As an example, in step S1221, after acquiring the point cloud data corresponding to each point cloud timestamp, the controller invokes the dead reckoning algorithm. Then, the controller uses the dead reckoning algorithm to extend and predict the point cloud data corresponding to each point cloud timestamp to the scene fusion time. That is, the dead reckoning algorithm predicts the motion trajectory of all point cloud data corresponding to each point cloud timestamp to predict the point cloud position at the scene fusion time, thereby obtaining the target point cloud data at the scene fusion time. For example, upon reaching the preset release timestamp corresponding to the vehicle radar (such as the 100th millisecond, 200th millisecond, etc.), after receiving the point cloud data corresponding to each point cloud timestamp, the controller uses the dead reckoning algorithm to predict the point cloud data of the first millisecond, thereby predicting the new point cloud data that changes at the 80th millisecond (scene fusion time), until the new point cloud data that changes at the 80th millisecond (scene fusion time) changes from the point cloud data of the 79th millisecond. In this way, the target point cloud data at the scene fusion time can be obtained. Among them, target point cloud data refers to all point cloud data within the radar scanning cycle, that is, a number of approximately 50,000 point clouds.

[0048] In another example, the controller retrieves a point cloud offset model (trained using a motion trajectory prediction algorithm) and inputs all point cloud data corresponding to each point cloud timestamp into the point cloud offset model. Simultaneously, the scene fusion time is input into the point cloud offset model to determine the scene fusion time to be predicted for each point cloud timestamp. Then, the point cloud offset model expands and predicts the point cloud data corresponding to each point cloud timestamp to the scene fusion time; that is, by performing motion trajectory prediction on the point cloud data corresponding to each point cloud timestamp, the target point cloud data for the scene fusion time can be obtained.

[0049] This embodiment uses the dead reckoning algorithm to extend and predict the point cloud data corresponding to each point cloud timestamp to the scene fusion time, which improves the accuracy of point cloud data offset and ensures that all point cloud data are at the scene fusion time, thereby improving the accuracy of subsequent scene synchronization.

[0050] In one embodiment, performing scene motion offset processing on the synchronized scene to obtain target scene data for the target vehicle includes: S141. Obtain the prediction inference duration corresponding to the intelligent driving model, wherein the prediction inference duration is the predicted duration required for the intelligent driving model to generate control signals.

[0051] S142. The synchronous scene is processed by scene motion offset processing through scene fusion time, time deviation and prediction inference time to obtain target scene data of target vehicle; the time deviation is the deviation between preset release timestamp and scene fusion time.

[0052] The intelligent driving model is a model used to generate directional control signals for the target vehicle, and it is part of the intelligent driving system. The prediction inference duration is the time required for the predicted intelligent driving model to generate the control signals. The time deviation is the deviation between the preset release timestamp and the scene fusion time. The preset release timestamp is the pre-set time when the point cloud data of the vehicle radar is transmitted to the controller. Specifically, the preset release timestamp is after the vehicle radar scan is completed. For example, when the radar scan cycle is 80 milliseconds, the preset release timestamp can be 90 milliseconds, 100 milliseconds, 200 milliseconds, etc. When the preset release timestamp is less than the radar scan cycle, it must wait for the radar scan to complete before transmitting the data to the controller. For example, the preset release timestamp can be 50 milliseconds, 150 milliseconds, 250 milliseconds, etc.

[0053] As an example, in step S141, after obtaining the synchronized scene at the scene fusion moment, the prediction inference duration corresponding to the intelligent driving model is obtained, that is, the controller predicts the duration required for the intelligent driving model to generate control signals through multiple historical inference durations.

[0054] As an example, in step S142, scene motion offset processing is performed on the synchronous scene using the scene fusion time, time deviation, and prediction inference duration. Specifically, the controller first obtains the preset release timestamp of the vehicle radar to calculate the time deviation between the scene fusion time and the preset release timestamp. Then, it calculates the backward prediction duration using the time deviation and prediction inference duration to determine the estimated time the controller needs to predict. Next, the controller performs a time offset on the synchronous scene at the scene fusion time using a scene offset model. This allows the scene offset model to predict the motion trajectory based on the synchronous scene to obtain the scene information at the estimated time, thereby obtaining the target scene data of the target vehicle at the estimated time. For example, when the preset release timestamp is 200 milliseconds, the scene fusion time is 180 milliseconds, and the prediction inference duration is 45 milliseconds, the time deviation between the preset release timestamp and the scene fusion time is first calculated to be 20 milliseconds. Then, the sum of the scene fusion time, time deviation, and prediction inference duration is calculated to be 245 milliseconds, meaning the estimated time is 245 milliseconds. The calculation of time deviation is to determine how much time needs to be predicted forward from the 180th millisecond.

[0055] This embodiment performs scene motion offset processing on the synchronous scene by considering the scene fusion time, time deviation, and prediction inference duration. This achieves motion compensation for the synchronous scene and acquisition of target scene data, thereby enabling the prediction of scene data at the time when the intelligent driving model outputs control signals, and thus improving the accuracy of the target scene data.

[0056] In one embodiment, obtaining the prediction inference time corresponding to the intelligent driving model includes: S1411. Obtain multiple historical reasoning durations corresponding to the intelligent driving model, wherein the historical reasoning duration is the duration required for the intelligent driving model to generate historical control information.

[0057] S1412. The model inference prediction time of the intelligent driving model is obtained by using the least squares method to predict the inference time of all the historical inference times.

[0058] The historical inference duration refers to the time required for the intelligent driving model to generate historical control information. For example, the time taken for five consecutive inferences is 45 milliseconds, 56 milliseconds, 52 milliseconds, 49 milliseconds, and 51 milliseconds, respectively. The least squares method is a mathematical optimization method that finds the best-fit data by minimizing the sum of squared errors. In this embodiment, the least squares method is used to fit multiple historical inference durations to predict the time required for the intelligent driving model to generate the control signal in this instance.

[0059] As an example, in step S1411, the controller retrieves multiple historical inference durations corresponding to the intelligent driving model, that is, it obtains all historical inference durations within adjacent time periods from the recorded historical inference durations of each inference by the intelligent driving model. The selection of historical inference durations within adjacent time periods is to ensure the validity of the data.

[0060] As an example, in step S1412, the least squares method is used to predict the inference time of all historical inferences. That is, the controller uses the least squares method to fit the data of all historical inference times. First, a suitable fitting function (such as a linear fitting function) is selected, then an error function is constructed, and a normal equation is constructed to solve the parameters so that the error function is minimized, thus obtaining the fitting curve. Then, the controller uses the fitting curve to make inference predictions, thereby obtaining the predicted inference time of the intelligent driving model.

[0061] In another example, the predicted inference time of the intelligent driving model is obtained by averaging all historical inference times within all adjacent time periods. Alternatively, the predicted inference time of the intelligent driving model is obtained by weighted summation of all historical inference times within all adjacent time periods.

[0062] This embodiment uses the least squares method to predict the duration of all historical inferences, thereby predicting the duration of control signals generated by the intelligent driving model. This avoids large errors caused by scene shifts due to different inference durations each time, and thus improves the accuracy of predicting the inference duration.

[0063] In one embodiment, the step of performing scene motion offset processing on the synchronized scene using scene fusion time, time deviation, and prediction inference duration to obtain target scene data for the target vehicle includes: S1421. The scene fusion time, time deviation and prediction inference time are summed to obtain the estimated time, which is the time when the predicted intelligent driving model outputs the control signal.

[0064] S1422. Extend the synchronous scene at the scene fusion time to the estimated time using the dead reckoning algorithm to obtain the target scene data corresponding to the estimated time.

[0065] The estimated time is the predicted time when the intelligent driving model outputs the control signal.

[0066] As an example, in step S1411, after obtaining the prediction inference duration, the controller first obtains the preset release timestamp of the vehicle radar and determines the difference between the preset release timestamp and the scene fusion time as the time deviation. Then, the controller sums the scene fusion time, the time deviation, and the prediction inference duration. That is, it first sums the time deviation and the prediction inference duration to determine the duration to be predicted from the scene fusion time, and then sums the duration to be predicted from the scene fusion time to predict the estimated time when the intelligent driving model outputs the control signal. For example, when the preset release timestamp is 100 milliseconds, the scene fusion time is 80 milliseconds, and the prediction inference duration is 50 milliseconds, the time deviation between the preset release timestamp and the scene fusion time is first calculated to be 20 milliseconds, and then the sum of the scene fusion time, the time deviation, and the prediction inference duration is calculated to be 150 milliseconds, that is, the estimated time is 150 milliseconds. The calculation of the time deviation is to determine how much time needs to be predicted from 80 milliseconds onwards. In another example, the sum of the preset release timestamp and the prediction inference duration can be calculated directly. That is, 50 milliseconds are added forward from the 100th millisecond to calculate the estimated time as the 150th millisecond. In other words, the prediction is directly made up to the 150th millisecond, without considering how much time to predict forward from the 80th millisecond.

[0067] As an example, in step S1412, the controller extends the synchronous scene at the scene fusion time to the estimated time through the dead reckoning algorithm. That is, the motion trajectory of the synchronous scene at the scene fusion time is predicted by the dead reckoning algorithm. In other words, the scene data at the estimated time is predicted based on the synchronous scene by the dead reckoning algorithm and determined as the target scene data.

[0068] This embodiment calculates the estimated time by summing the scene fusion time, time deviation, and prediction inference time, thereby enabling the prediction of when the intelligent driving model outputs control signals. By extending the synchronous scene of the scene fusion time to the estimated time through dead reckoning, it provides scene data for predicting when the intelligent driving model outputs control signals, avoiding motion compensation by the intelligent driving model and reducing the inference time of the intelligent driving model.

[0069] In one embodiment, after performing scene motion offset processing on the synchronized scene to obtain the target scene data of the target vehicle, the method further includes: S15. The intelligent driving model generates control signals from the target scene data to obtain the direction control signals of the target vehicle.

[0070] The intelligent driving model is a model used to generate directional control signals for the target vehicle, and it is part of the intelligent driving system. The directional control signal is a signal used to control the driving direction of the target vehicle. This direction can be forward or backward; for example, during autonomous driving, it's a directional control signal controlling the target vehicle to change lanes to the left; during automatic parking, it's a directional control signal controlling the target vehicle to reverse to the left.

[0071] Specifically, the system can record the time when target scene data is input into the intelligent driving model and the time when the intelligent driving model outputs control signals, thereby calculating the inference duration of the intelligent driving model. This inference duration can be used as the inference duration of the next intelligent driving model iteration. Alternatively, the earliest historical inference duration can be deleted, and the current inference duration can be fitted with the remaining historical inference durations using the least squares method to predict the predicted inference duration of the next intelligent driving model iteration.

[0072] As an example, in step S15, after obtaining the target scene data, the controller inputs the target scene data at the estimated time into the intelligent driving model. The intelligent driving model then generates control signals based on the target scene data. That is, the intelligent agent in the intelligent driving model analyzes all the information in the target scene to determine the driving direction of the target vehicle and simultaneously generates a directional control signal corresponding to the driving direction. For example, when driving on a highway, if the intelligent driving model identifies road construction ahead from the target scene data and identifies the lane where the road is under construction, the lane where the road is not under construction, and the following traffic situation in the lane where the road is not under construction, then it determines the driving direction to the lane where the road is not under construction and simultaneously generates directional control information to the lane where the road is not under construction.

[0073] In another example, after the controller inputs the target scene data at the estimated time into the intelligent driving model, it uses the reasoning ability and generation learned by the intelligent driving model during training to determine the driving direction of the target vehicle based on all the information in the target scene data, and at the same time generates a directional control signal corresponding to the driving direction.

[0074] This embodiment uses the intelligent driving model to infer control signals from the target scene data, avoiding motion compensation of the scene data by the intelligent driving model, reducing the inference time of the intelligent driving model, and improving the accuracy of the intelligent driving model's inference.

[0075] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0076] This application also provides a vehicle data processing device 60, please refer to... Figure 6 ,include: The publishing timestamp module 601 is used to determine the image publishing timestamp of the vehicle-mounted camera based on the radar scanning cycle of the vehicle-mounted radar of the target vehicle and the image transmission duration of the vehicle-mounted camera. The time data acquisition module 602 is used to determine the scene fusion time based on the image release timestamp and the image transmission duration, and to acquire the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time. The scene data fusion module 603 is used to perform scene fusion on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion. The scene motion offset module 604 is used to perform scene motion offset processing on the synchronous scene at the scene fusion time to obtain the target scene data of the target vehicle.

[0077] This application also provides an electronic device 70, please refer to... Figure 7 It includes a memory 701 and a controller 702, wherein the memory 701 is used to store computer programs; and the controller 702 is used to execute the programs stored in the memory 701 to implement the vehicle data processing method described in any embodiment of this application.

[0078] In one embodiment, the present invention provides a vehicle including an onboard radar, an onboard camera, and the aforementioned electronic device 70.

[0079] This application also provides a computer-readable storage medium storing a computer program that, when executed by a controller, implements the vehicle data processing method described in any embodiment of this application.

[0080] In this application, "multiple" refers to two or more.

[0081] The terms “first,” “second,” “third,” “fourth,” etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0082] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0083] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0084] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A vehicle data processing method, characterized by, include: The image release timestamp of the vehicle-mounted imaging device is determined based on the radar scanning cycle of the vehicle's onboard radar and the image transmission duration of the onboard imaging device. The image transmission duration is the time it takes for the onboard imaging device to transmit the captured image data to the controller, or the total time it takes for the onboard imaging device to transmit the captured image data to the controller and then process the image in the controller. The image release timestamp refers to the moment when the onboard imaging device transmits the image data to the controller. The scene fusion time is determined based on the image release timestamp and the image transmission duration, and the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time, are obtained. Scene fusion is performed on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion; The synchronized scene is subjected to scene motion offset processing to obtain the target scene data of the target vehicle; wherein, the scene motion offset is an operation of backward expansion prediction of the synchronized scene at the scene fusion moment; The process of performing scene motion offset processing on the synchronized scene to obtain target scene data for the target vehicle includes: Obtain the prediction inference time corresponding to the intelligent driving model, wherein the prediction inference time is the predicted time required for the intelligent driving model to generate control signals; The synchronous scene is processed by scene motion offset processing using scene fusion time, time deviation and prediction inference time to obtain target scene data of the target vehicle; the time deviation is the deviation between the preset release timestamp and the scene fusion time; wherein, the preset release timestamp is the time when the point cloud data of the vehicle radar is transmitted to the controller.

2. The vehicle data processing method of claim 1, wherein, The process of performing scene motion offset processing on the synchronized scene using scene fusion time, time deviation, and prediction inference duration to obtain target scene data for the target vehicle includes: The scene fusion time, time deviation and prediction inference time are summed to obtain the estimated time, which is the time when the predicted intelligent driving model outputs the control signal; The synchronous scene at the time of scene fusion is extended to the estimated time using the dead reckoning algorithm to obtain the target scene data corresponding to the estimated time.

3. The vehicle data processing method of claim 1, wherein, The acquisition of the prediction and inference time corresponding to the intelligent driving model includes: Obtain multiple historical reasoning durations corresponding to the intelligent driving model, wherein the historical reasoning duration is the time required for the intelligent driving model to generate historical control information; The predicted inference time of the intelligent driving model is obtained by using the least squares method to predict the inference time of all the historical inference times.

4. The vehicle data processing method as described in claim 1, characterized in that, Acquiring target point cloud data corresponding to the vehicle-mounted radar at the scene fusion moment includes: Acquire all point cloud data and their corresponding point cloud timestamps obtained by the vehicle-mounted radar scanning the surrounding environment within the radar scanning cycle. Based on the scene fusion time, point cloud motion offset is performed on the point cloud data corresponding to each point cloud timestamp to obtain the target point cloud data corresponding to the scene fusion time; wherein, the point cloud motion offset is the operation of forward expansion prediction of the point cloud data at each point cloud timestamp.

5. The vehicle data processing method as described in claim 4, characterized in that, The step of performing point cloud motion offset on the point cloud data corresponding to each point cloud timestamp based on the scene fusion time to obtain the target point cloud data corresponding to the scene fusion time includes: By using the dead reckoning algorithm, the point cloud data corresponding to each point cloud timestamp is extended and predicted to the scene fusion time, thus obtaining the target point cloud data at the scene fusion time.

6. The vehicle data processing method as described in claim 1, characterized in that, After performing scene motion offset processing on the synchronized scene to obtain the target scene data of the target vehicle, the process further includes: The intelligent driving model generates control signals from the target scene data to obtain the direction control signals for the target vehicle.

7. A vehicle data processing device, characterized in that, include: The timestamp publishing module is used to determine the image publishing timestamp of the vehicle-mounted imaging device based on the radar scanning cycle of the target vehicle's onboard radar and the image transmission duration of the onboard imaging device; wherein, the image transmission duration is the time it takes for the onboard imaging device to transmit the captured image data to the controller, or the total time after the onboard imaging device transmits the captured image data to the controller and then processes the image in the controller; the image publishing timestamp refers to the moment when the onboard imaging device transmits the image data to the controller; The time data acquisition module is used to determine the scene fusion time based on the image release timestamp and the image transmission duration, and to acquire the target image data corresponding to the vehicle-mounted shooting device at the scene fusion time, as well as the target point cloud data corresponding to the vehicle-mounted radar at the scene fusion time; The scene data fusion module is used to perform scene fusion on the target image data and the target point cloud data to obtain the synchronous scene of the target vehicle at the moment of scene fusion. The scene motion offset module is used to perform scene motion offset processing on the synchronous scene at the scene fusion time to obtain the target scene data of the target vehicle; wherein, the scene motion offset is an operation of backward expansion prediction of the synchronous scene at the scene fusion time; The scene motion offset module is also used for: Obtain the prediction inference time corresponding to the intelligent driving model, wherein the prediction inference time is the predicted time required for the intelligent driving model to generate control signals; The synchronous scene is processed by scene motion offset processing using scene fusion time, time deviation and prediction inference time to obtain target scene data of the target vehicle; the time deviation is the deviation between the preset release timestamp and the scene fusion time; wherein, the preset release timestamp is the time when the point cloud data of the vehicle radar is transmitted to the controller.

8. An electronic device, characterized in that, Includes controller and memory, among which, Memory, used to store computer programs; A controller is used to execute a program stored in a memory to implement the vehicle data processing method according to any one of claims 1 to 6.

9. A vehicle, characterized in that, This includes vehicle-mounted radar, vehicle-mounted camera equipment, and the electronic equipment as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by the controller, implements the vehicle data processing method according to any one of claims 1 to 6.

Images