Data processing method and device and vehicle

Abstract

The invention provides a data processing method and device and a vehicle, which can be applied to the field of intelligent driving. The data processing method comprises the following steps: acquiring first environment information around a vehicle and first state information of the vehicle; sending the first environment information and the first state information to a cloud server; first time delay optimization configuration information sent by the cloud server is received, and the cloud server stores a mapping relation among the first environment information, the first state information and the first time delay optimization configuration information; the first time delay optimization configuration information comprises a first time delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; and processing first data acquired by the sensor according to the first time delay optimization parameter. The embodiment of the invention can be applied to an intelligent automobile or a new energy automobile, the end-to-end time delay can be reduced, and the driving safety of a user can be ensured.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving, and more specifically, to a data processing method, device and vehicle. Background Art

[0002] In recent years, with the rapid development of autonomous driving artificial intelligence (AI) algorithms and technologies, a large number of traditional car manufacturers and new forces in autonomous driving have entered the market to build high-level fully autonomous driving systems and applications for L4. However, people have certain concerns about the implementation of autonomous driving technology, and one of the most concerned issues is whether it can ensure the safety of other vehicles and pedestrians. A very important indicator for measuring safety is the end-to-end latency of the data flow link of the autonomous driving system, such as the latency from the acquisition of raw signals by sensors such as lidar and cameras to the issuance of a brake command to the chassis domain controller. A lower end-to-end latency (such as less than 100ms) can greatly improve the safety factor of vehicles and pedestrians, and is one of the key goals for the later optimization of systems and applications.

[0003] Therefore, how to reduce end-to-end latency has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data processing method, device and vehicle, which help to reduce end-to-end latency, thereby helping to ensure the driving safety of users.

[0005] In a first aspect, the present application provides a data processing method, the method comprising: obtaining first environmental information around a vehicle and first status information of the vehicle; sending the first environmental information and the first status information to a cloud server; receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information comprising a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; and processing the first data collected by the sensor according to the first delay optimization parameter.

[0006] Based on the above technical solution, the cloud server can send corresponding delay optimization configuration information to the vehicle based on the environmental information around the vehicle, the vehicle's status information and the mapping relationship. In this way, the vehicle can obtain appropriate delay optimization configuration information for the current scenario from the cloud server, which helps to reduce the delay of the vehicle's processing unit when processing data collected by the sensor, thereby helping to reduce end-to-end delay, thereby helping to ensure the user's driving safety.

[0007] The mapping relationship stored in the above cloud server may be a mapping relationship between environment information, state information and latency optimization configuration information in different scenarios.

[0008] When the front-end to end delay optimization configuration is basically configured offline according to the experience of engineers for some universal environments, it is difficult to make adaptive dynamic adjustments for different scenarios during vehicle driving, and the current end-to-end delay optimization configuration adjustment process is very time-consuming. In an embodiment of the present application, by deploying delay optimization configuration information for different scenarios in advance on the cloud server, the vehicle can obtain appropriate delay optimization configuration information in different scenarios. At the same time, the cloud server determines the appropriate delay optimization configuration information for the current scenario of the vehicle through a mapping relationship, which can also reduce the waiting time for the vehicle to obtain the delay optimization configuration information.

[0009] In some possible implementations, the first latency optimization parameter includes one or more of a priority parameter of a task in a processing unit, a core binding parameter, and a scheduling parameter of a scheduler in an operating system.

[0010] In some possible implementations, the first delay optimization configuration information includes end-to-end delay optimization parameters, where the end-to-end delay is the end-to-end delay of the data flow link of the autonomous driving system in the vehicle, such as the delay from the collection of raw data by sensors (e.g., cameras, lidars, etc.) to the issuance of vehicle control commands to the vehicle controller (e.g., chassis controller).

[0011] The above end-to-end delay may include the delay of the above processing unit processing the data collected by the sensor.

[0012] In some possible implementations, the first environmental information includes at least one of time, geographic location, climate, and road conditions.

[0013] In some possible implementations, the first status information includes at least one of the vehicle's hardware specifications, system status, and autonomous driving status.

[0014] In some possible implementations, the vehicle is a vehicle in an autonomous driving state.

[0015] In combination with the first aspect, in a possible implementation of the first aspect, the method also includes: when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold, obtaining second environmental information around the vehicle and second status information of the vehicle; sending the second environmental information and the second status information to the cloud server; receiving second delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the second environmental information, the second status information and the second delay optimization configuration information, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; and processing the second data collected by the sensor according to the second delay optimization parameter.

[0016] Based on the above technical solution, during the driving process of the vehicle, when the delay of the processing unit processing the data collected by the sensor is greater than or equal to the delay threshold, the vehicle can send the second environment information around it and the second state information of the vehicle to the cloud server. The cloud server can send updated delay optimization configuration information to the vehicle based on the mapping relationship. In this way, the vehicle can adaptively adjust the delay optimization configuration information during driving to achieve the effect of automatic online performance optimization, which helps to reduce end-to-end delay, thereby helping to ensure the driving safety of users. At the same time, when the vehicle adjusts the deployed delay optimization configuration information, there is no need for the participation of existing autonomous driving application developers, such as modifying code or recompiling.

[0017] The vehicle can dynamically perceive the performance degradation caused by scene changes during driving by extracting environmental information and vehicle status information within the current time window. The cloud server determines the delay optimization configuration information suitable for the changed scene based on the changed scene of the vehicle.

[0018] In some possible implementations, when the delay of the processing unit in processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the first delay threshold or less than or equal to the second delay threshold, second environmental information around the vehicle and second state information of the vehicle are obtained, wherein the second delay threshold is less than the first delay threshold.

[0019] Alternatively, when the delay of the processing unit in processing the data collected by the sensor according to the first delay optimization parameter is not within the delay range, second environmental information around the vehicle and second state information of the vehicle are obtained.

[0020] In some possible implementations, the first delay optimization configuration information includes the delay threshold or the delay range.

[0021] Based on the above technical solution, the vehicle can obtain the delay range through the first delay optimization configuration information sent by the cloud server, so as to monitor in real time the relationship between the delay of the processing unit processing the data collected by the sensor and the delay range. When the delay is not within the delay range, the cloud server can be requested for new delay optimization configuration information again, so as to avoid the safety hazards caused by the deterioration of end-to-end delay.

[0022] In combination with the first aspect, in a possible implementation of the first aspect, the processing unit includes a main processing unit and a backup processing unit, and processes the first data collected by the sensor according to the first delay optimization parameter, including: deploying the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0023] Based on the above technical solution, by first deploying the first delay optimization parameter to the backup processing unit, the validity of the first delay optimization parameter can be verified in advance without affecting the normal driving of the vehicle.

[0024] In combination with the first aspect, in a possible implementation of the first aspect, the first delay optimization parameter is deployed before the main processing unit, and the method also includes: within the first time period, determining that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit.

[0025] Based on the above technical solution, before deploying the first delay optimization parameters to the main processing unit, it can be determined that the vehicle control instructions output by the backup processing unit within the first time period are the same as the vehicle control instructions output by the main processing unit, which helps to improve the effectiveness of verifying the first delay optimization configuration information.

[0026] In combination with the first aspect, in a possible implementation of the first aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at the multiple moments in the future.

[0027] Based on the above technical solution, the vehicle can obtain the delay configuration parameters at multiple moments in the future in advance. This avoids the frequent interactions between the vehicle and the cloud server in the future, and helps reduce the waiting time for the vehicle to obtain delay optimization configuration information.

[0028] In some possible implementations, the method includes: the vehicle sends navigation information to a cloud server, and the cloud server determines the vehicle's environmental information and status information at multiple moments in the future based on the navigation information, thereby sending to the vehicle delay optimization parameters for the processing unit to process data collected by the sensor at multiple moments in the future.

[0029] In a second aspect, a data processing method is provided, the method comprising: obtaining first environmental information around a vehicle and first state information of the vehicle; determining first delay optimization configuration information based on the first environmental information, the first state information and a mapping relationship, the mapping relationship comprising a mapping relationship between the first environmental information, the first state information and the first delay optimization configuration information, the first delay optimization configuration information comprising a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; and processing the first data collected by the sensor based on the first delay optimization parameter.

[0030] Based on the above technical solution, the vehicle can determine the appropriate delay optimization configuration information for the current scenario according to the mapping relationship saved in the vehicle, which helps to reduce the delay of the vehicle's processing unit when processing data collected by sensors, thereby helping to reduce end-to-end delay, thereby helping to ensure the user's driving safety.

[0031] In combination with the second aspect, in a possible implementation of the second aspect, the method also includes: when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold, obtaining second environmental information around the vehicle and second state information of the vehicle; determining second delay optimization configuration information according to the second environmental information, the second state information and the mapping relationship, the mapping relationship including a mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; and processing the second data collected by the sensor according to the second delay optimization parameter.

[0032] Based on the above technical solution, when the delay of the processing unit processing the data collected by the sensor is greater than or equal to the delay threshold during the driving process of the vehicle, the vehicle can re-determine the updated delay optimization configuration information based on the second environmental information around the vehicle, the second state information of the vehicle and the mapping relationship. In this way, the vehicle can adaptively adjust the delay optimization configuration information during driving to achieve the effect of automatic online performance optimization, thereby helping to reduce end-to-end delay, thereby helping to ensure the driving safety of users.

[0033] In combination with the second aspect, in a possible implementation of the second aspect, the processing unit includes a main processing unit and a backup processing unit, and processes the first data collected by the sensor according to the first delay optimization parameter, including: deploying the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0034] In combination with the second aspect, in a possible implementation of the second aspect, the first delay optimization parameter is deployed before the main processing unit, and the method also includes: within the first time period, determining that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit.

[0035] In combination with the second aspect, in a possible implementation of the second aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0036] In a third aspect, the present application provides a data processing method, the method comprising: obtaining first environmental information around a vehicle and first status information of the vehicle; sending first delay optimization configuration information to the vehicle based on the first environmental information, the first status information and a mapping relationship, the mapping relationship comprising a mapping relationship between environmental information, status information and delay optimization configuration information, the first delay optimization configuration information comprising a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor.

[0037] Based on the above technical solution, the cloud server can send corresponding delay optimization configuration information to the vehicle based on the mapping relationship between the environmental information around the vehicle and the vehicle's status information. In this way, the vehicle can obtain appropriate delay optimization configuration information for the current scenario from the cloud server, which helps to reduce the delay of the vehicle's processing unit when processing data collected by the sensor, thereby helping to reduce end-to-end delay, thereby helping to ensure the user's driving safety.

[0038] In combination with the third aspect, in a possible implementation of the third aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0039] In combination with the third aspect, in a possible implementation of the third aspect, the method further includes: according to the first environment information and the first state information, using a hierarchical navigable small worldgraphs (HNSW) algorithm to retrieve the mapping relationship to obtain the first delay optimization configuration information.

[0040] Based on the above technical solution, the cloud server can retrieve the mapping relationship based on the HNSW algorithm to obtain the first delay optimization configuration information, which helps to reduce the delay when the cloud server searches for the corresponding delay optimization configuration information through the information sent by the vehicle.

[0041] In combination with the third aspect, in a possible implementation of the third aspect, before obtaining the first environmental information around the vehicle and the first state information of the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; determining the first delay when processing data collected by the sensor according to the third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; updating the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; and storing the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0042] Based on the above technical solution, the initial delay optimization parameters are set through the offline simulation system, the end-to-end delay during actual operation on the offline simulation system is checked, and then optimized. The end-to-end delay calculation method obtained in this way is more accurate and does not require additional simulation verification.

[0043] In combination with the third aspect, in a possible implementation of the third aspect, before obtaining the first environmental information around the vehicle and the first state information of the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; extracting delay model data based on the third environmental information and the third state information data, the delay model data including one or more of the number of application threads, periodic data of the threads, time probability distribution of thread running, and dependency relationship of thread data; inputting the delay model data into a delay simulator to obtain a second delay; determining a fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; and storing the correspondence between the third environmental information, the third state information, and the fourth delay optimization parameter in the mapping relationship.

[0044] Based on the above technical solution, the end-to-end delay can be simulated by a delay simulator, which can reduce the overhead of simulation delay estimation and perform large-scale parallel optimization with fast convergence speed.

[0045] In some possible implementations, before receiving the first environmental information and the first state information sent by the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; inputting the third environmental information and the third state information into a prediction model to obtain a fourth delay optimization parameter, wherein the prediction model is trained by a training data set, and the training data set includes sample environmental information, sample state information, and the sample delay optimization parameter; and storing the correspondence between the third environmental information, the third state information, and the fourth delay optimization parameter in the mapping relationship.

[0046] In some possible implementations, obtaining third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle includes: receiving data collected by sensors of the autonomous driving vehicle and the third state information; indexing the data collected by the sensors and the third state information according to time and location to obtain the third environment information and the third state information.

[0047] In some possible implementations, the data collected by the sensor and the third state information are indexed according to time and location to obtain the third environment information and the third state information, including: through a crowdsourcing algorithm or a group intelligence perception algorithm, the data collected by the sensor and the third state information are indexed according to time and location to obtain the third environment information and the third state information.

[0048] Based on the above technical solutions, targeted scheduling strategies can be provided for massive autonomous driving scenarios (i.e., the combination of the vehicle's environmental information and the vehicle's status information). By using crowdsourcing algorithms or group intelligence perception algorithms, there is no need to devote a large amount of manpower and material resources to solve the delay optimization configuration of different extreme scenarios (or extreme situations). Each time the version is updated, only the mapping relationship in the cloud server needs to be updated, without the need to trouble users to make modifications.

[0049] In a fourth aspect, the present application provides a data processing device, which includes: an acquisition unit, used to acquire first environmental information around a vehicle and first status information of the vehicle; a sending unit, used to send the first environmental information and the first status information to a cloud server; a receiving unit, used to receive first delay optimization configuration information sent by the cloud server, the cloud server saves a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by a sensor; the processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

[0050] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the acquisition unit is used to obtain the second environmental information around the vehicle and the second state information of the vehicle when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the sending unit is used to send the second environmental information and the second state information to the cloud server; the receiving unit is used to receive the second delay optimization configuration information sent by the cloud server, the cloud server saves the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information includes the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit is used to process the second data collected by the sensor according to the second delay optimization parameter.

[0051] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0052] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the device also includes: a determination unit, used to determine, within the first time period, that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0053] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at the multiple moments in the future.

[0054] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the first environmental information includes at least one of time, geographic location, climate and road conditions; and / or, the first status information includes at least one of the hardware specifications, system status, and automatic driving status of the vehicle.

[0055] In a fifth aspect, the present application provides a data processing device, which includes: an acquisition unit for acquiring first environmental information around a vehicle and first state information of the vehicle; a determination unit for determining first delay optimization configuration information based on the first environmental information, the first state information and a mapping relationship, the mapping relationship including a mapping relationship between the first environmental information, the first state information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; the processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

[0056] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the acquisition unit is also used to obtain second environmental information around the vehicle and second state information of the vehicle when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the determination unit is also used to determine second delay optimization configuration information based on the second environmental information, the second state information and the mapping relationship, the mapping relationship including the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit is also used to process the second data collected by the sensor according to the second delay optimization parameter.

[0057] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0058] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the determination unit is further used to determine, within the first time period, that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0059] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0060] In a sixth aspect, the present application provides a data processing device, comprising: an acquisition unit, used to acquire first environmental information around a vehicle and first status information of the vehicle; a determination unit, used to determine first delay optimization configuration information based on the first environmental information, the first status information and a mapping relationship, the mapping relationship including a mapping relationship between environmental information, status information and delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; a sending unit, used to send the first delay optimization configuration information to the vehicle.

[0061] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0062] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes: an information retrieval unit, used to retrieve the mapping relationship according to the first environment information and the first state information using a hierarchical navigable small world HNSW algorithm to obtain the first delay optimization configuration information.

[0063] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes a parameter updating unit and a storage unit, the acquisition unit is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and the first state information sent by the vehicle; the determination unit is further used to determine the first delay when processing data collected by the sensor according to the third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; the parameter updating unit is used to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0064] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes a delay simulation unit, and the acquisition unit is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and the first state information sent by the vehicle; the determination unit is further used to extract delay model data based on the third environmental information and the third state information data, and the delay model data includes one or more of the number of threads of the application, the periodic data of the threads, the time probability distribution of the thread running, and the dependency relationship of the thread data; the delay simulation unit is used to input the delay model data into a delay simulator to obtain a second delay; the determination unit is further used to determine a fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0065] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the acquisition unit is used to: receive data collected from the sensor of the autonomous driving vehicle and the third state information; index the data collected by the sensor and the third state information according to time and location to obtain the third environment information and the third state information.

[0066] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the acquisition unit is used to: index the data collected by the sensor and the third state information according to time and location through a crowdsourcing algorithm or a group perception algorithm to obtain the third environment information and the third state information.

[0067] In the seventh aspect, the present application provides a data processing device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device performs any possible data processing method in the first aspect or the second aspect.

[0068] In an eighth aspect, the present application provides a data processing device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device executes any possible data processing method in the third aspect.

[0069] In a ninth aspect, the present application provides a data processing system, which control system includes a sensor and a computing platform, and the computing platform includes any possible data processing device in the fourth aspect, or includes any possible data processing device in the fifth aspect, or includes any possible data processing device in the seventh aspect.

[0070] In the tenth aspect, the present application provides a vehicle, which includes any possible data processing device in the fourth aspect, or includes any possible data processing device in the fifth aspect, or includes the data processing device described in the seventh aspect, or includes the data processing system described in the ninth aspect.

[0071] In an eleventh aspect, the present application provides a server, which includes any possible data processing device in the sixth aspect, or includes the data processing device described in the eighth aspect.

[0072] In a twelfth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any possible data processing method in the first aspect or the second aspect above.

[0073] In a thirteenth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any possible data processing method in the third aspect above.

[0074] It should be noted that the above-mentioned computer program code can be stored in whole or in part on the first storage medium, wherein the first storage medium can be packaged together with the processor or separately packaged with the processor, and the embodiments of the present application do not specifically limit this.

[0075] In a fourteenth aspect, the present application provides a computer-readable medium storing a program code. When the computer program code runs on a computer, the computer executes any possible data processing method in the first aspect or the second aspect.

[0076] In a fifteenth aspect, the present application provides a computer-readable medium storing a program code. When the computer program code runs on a computer, the computer executes any possible data processing method in the third aspect.

[0077] In a sixteenth aspect, the present application provides a chip, comprising a circuit for executing any possible data processing method in the first aspect or the second aspect.

[0078] In a seventeenth aspect, the present application provides a chip, comprising a circuit for executing any possible data processing method in the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a functional block diagram of a vehicle provided in an embodiment of the present application.

[0080] Figure 2 It is a schematic diagram of the relationship between application nodes in the robot operating system ROS.

[0081] Figure 3 It is a schematic flow chart of the data processing method provided in the embodiment of the present application.

[0082] Figure 4 It is a schematic diagram of the system architecture provided in the embodiment of the present application.

[0083] Figure 5 It is another schematic diagram of the system architecture provided in an embodiment of the present application.

[0084] Figure 6 It is another schematic flow chart of the data processing method provided in an embodiment of the present application.

[0085] Figure 7 This is an example diagram of the delay feature vector fast retrieval layer data structure provided in an embodiment of the present application.

[0086] Figure 8 It is another schematic flow chart of the data processing method provided in an embodiment of the present application.

[0087] Fig. 9 It is a schematic block diagram of a data processing device provided in an embodiment of the present application.

[0088] Fig.10 It is another schematic block diagram of the data processing device provided in an embodiment of the present application.

[0089] Fig.11 It is another schematic block diagram of the data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. For example, "at least one of A and B" is similar to "A and / or B", describing the association relationship of associated objects, indicating that three relationships can exist, for example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0091] In the embodiments of the present application, prefixes such as "first" and "second" are used only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0092] Figure 1 1 is a functional block diagram of a vehicle 100 provided in an embodiment of the present application. The vehicle 100 may include a perception system 120 and a computing platform 150, wherein the perception system 120 may include one or more sensors for sensing information about the environment around the vehicle 100. For example, the perception system 120 may include a positioning system, and the positioning system may be a global positioning system (GPS), or a Beidou system or other positioning systems. The perception system 120 may also include one or more of an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera device.

[0093] Some or all functions of the vehicle 100 may be controlled by the computing platform 150. The computing platform 150 may include one or more processors, such as processors 151 to 15n (n is a positive integer). The processor is a circuit with signal processing capability. In one implementation, the processor may be a circuit with instruction reading and execution capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by a processor such as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, the processor may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform 150 may also include a memory, the memory is used to store instructions, and some or all of the processors 151 to 15n may call the instructions in the memory to implement corresponding functions.

[0094] The vehicles involved in the embodiments of the present application are vehicles in a broad sense, which may be means of transportation (such as commercial vehicles, passenger cars, motorcycles, flying cars, trains, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as mowers, harvesters, etc.), amusement equipment, toy vehicles, etc. The embodiments of the present application do not specifically limit the type of vehicles.

[0095] As mentioned above, a very important indicator for measuring safety is the end-to-end latency of the data flow link of the autonomous driving system, such as the latency from the acquisition of raw signals by sensors such as lidar and cameras to the issuance of braking and other instructions to the chassis domain controller. The running time of each node in the data flow link will have a direct and significant impact. A lower end-to-end latency (e.g., less than 100ms) can greatly improve the safety factor of vehicles and pedestrians, and is one of the key goals for the later optimization of systems and applications.

[0096] At present, the end-to-end delay optimization methods commonly used by autonomous driving system engineers mostly use some limited scenarios to adjust and optimize the parameters of autonomous driving systems and applications. The methods of parameter adjustment and optimization include manual tuning based on expert experience, direct tuning on the vehicle with online closed-loop feedback, and tuning using abstract models, simulators, and emulators combined with AI tuning algorithms. However, these methods are basically configured for some common working conditions and universal environments, and it is difficult to make adaptive dynamic adjustments for different scenarios during vehicle driving, and the generalization is insufficient; in addition, the adjustment process is generally very time-consuming. Even with the computing resources of cloud servers, the entire process takes several hours.

[0097] The embodiments of the present application provide a data processing method, device and vehicle, wherein the vehicle can send environmental information around the vehicle and the current status information of the vehicle to a cloud server, and the cloud server can send delay optimization configuration information corresponding to the environmental information and status information to the vehicle based on the mapping relationship between the environmental information, status information and delay optimization configuration information. The vehicle can process the data collected by the sensor through the delay optimization configuration information, which helps to reduce the end-to-end delay, thereby helping to ensure the safety of the vehicle and pedestrians.

[0098] Before introducing the technical solutions provided by the embodiments of the present application, several technical terms related to the embodiments of the present application are first introduced.

[0099] Message passing: A general term for a type of data communication method between processes or software components in a computer system. It abstracts and encapsulates the data to be communicated into "messages". The two or more parties involved in the communication can transfer messages between processes or components by calling primitives such as message sending and receiving, thereby completing data communication.

[0100] Autonomous driving intelligent operating system: An operating system based on the portable operating system interface (POSIX) standard, suitable for the high-performance computing and high-bandwidth communications required for autonomous driving, providing functions such as vehicle environment perception, intelligent decision-making and path planning.

[0101] Application node: In the autonomous driving intelligent operating system, an application with specific autonomous driving functions (such as perception, fusion and planning) encapsulated by communication middleware (such as robot operating system (ROS) and communication management (CM) in automotive open system architecture (AUTOSAR), etc.), usually a collection of single processes and multiple threads. Two or more nodes realize the transmission of messages and service calls between processes or components by calling communication middleware.

[0102] Message channel: In the autonomous driving intelligent operating system, the communication channel designated between application nodes, which consists of the sender's sending port, the receiver's receiving port, the channel name, the message format, etc.

[0103] Currently, the latency of many autonomous driving applications will fluctuate due to the influence of external environment or scenario, resulting in changes in the end-to-end latency of autonomous driving, which in turn affects the actual deployment effect of the end-to-end latency optimization configuration calculated offline. Many autonomous driving algorithms are scenario and data driven. When autonomous driving vehicles operate in different environments, different codes or model logics will be executed. For example, when the scenarios of highways, urban areas and underground parking change, or even when the traffic volume on the road changes or the weather in which the vehicle is located changes, the logic actually executed by the application is quite different, so the end-to-end latency will be affected by the scenario and external environment, and its baseline and jitter will be quite different in different scenarios and environments. It is precisely this fluctuation in external scenarios and environments that will greatly affect the effect of a single latency optimization configuration configured offline, causing its effect in some scenarios to be greatly reduced. Even extreme scenarios and environments may have negative effects, which have a significant impact on the safety of autonomous driving systems and applications.

[0104] Figure 2 A schematic diagram of the relationship between application nodes in ROS is shown. ROS includes nodes 1, 2, 3 and 4, wherein node 1 can be used to obtain images collected by sensors, node 2 can be used to perform image segmentation on the images collected by sensors, node 3 can be used to identify the type of obstacles, the distance from the vehicle, the speed of obstacles, etc. obtained after image segmentation, and node 4 can be used to determine vehicle control instructions based on the recognition results of node 3 and send the vehicle control instructions to the chassis domain controller.

[0105] The latency of the above-mentioned autonomous driving application can be understood as the latency of a single node processing data, for example, the latency of node 1 processing data.

[0106] The above-mentioned end-to-end delay of autonomous driving can be understood as the delay of the node set in the ROS communication system processing data. For example, the delay from node 1 obtaining the image collected by the sensor to node 4 sending the vehicle control command to the chassis domain controller.

[0107] Figure 3 FIG. 3 is a schematic flow chart of a data processing method 300 provided in an embodiment of the present application. Figure 3 As shown, the method 300 can be executed by a vehicle and a cloud server, and the method 300 includes steps S310 to S340.

[0108] S310, the vehicle obtains first environmental information around the vehicle and first state information of the vehicle.

[0109] Optionally, the vehicle obtains first environmental information around the vehicle, including: the vehicle can determine the first environmental information based on data collected by a sensor (for example, a camera, a millimeter-wave radar, a lidar, etc.).

[0110] Exemplarily, Table 1 shows the environment information and status information acquired by the vehicle.

[0111] Table 1

[0112]

[0113]

[0114] The above automatic driving modes can be understood as the automatic driving modes currently enabled by the vehicle, such as navigation cruise assistant (NCA), integrated cruise assistant (ICA), automatic valet parking (AVP), etc. Auxiliary functions may include autonomous emergency braking (AEB), lane departure warning (LDW), etc.

[0115] The environmental information and status information shown in Table 1 above are merely illustrative, and the vehicle may obtain more or less information than that shown in Table 1.

[0116] Optionally, the environmental information may include data collected by the vehicle's sensors (e.g., one or more of a camera, a laser radar, a millimeter-wave radar, a GPS, and an IMU). The vehicle may send the data collected by the sensor to a cloud server, which determines the environment in which the vehicle is located (e.g., geographic location, time, road conditions, etc.).

[0117] S320, the vehicle sends the first environment information and the first status information to the cloud server.

[0118] Correspondingly, the cloud server can receive the first environment information and the first status information sent by the vehicle.

[0119] Optionally, after determining the first environmental information based on data collected by the sensor, the vehicle may send the first environmental information and the first status information to a cloud server; alternatively, the vehicle may also send the data collected by the sensor and the first status information to a cloud server, and the cloud server may determine the first environmental information based on the data collected by the sensor.

[0120] Optionally, the vehicle may periodically send environmental information about the vehicle and status information about the vehicle to the cloud server.

[0121] Optionally, before the cloud server receives the first environmental information and the first status information sent by the vehicle, the method 300 also includes: the cloud server receives third status information from the autonomous driving vehicle and data collected by sensors in the autonomous driving vehicle; the cloud server indexes the third status information and the data collected by the sensor according to time and location to obtain the third environmental information and the third status information.

[0122] Optionally, the cloud server indexes the third state information and the data collected by the sensor according to time and location to obtain the third environment information and the third state information, including: the cloud server indexes the third state information and the data collected by the sensor according to time and location through a crowdsourcing algorithm or a group intelligence perception algorithm to obtain the third environment information and the third state information.

[0123] S330, the cloud server sends first delay optimization configuration information to the vehicle based on the first environmental information, the first status information and the mapping relationship, the mapping relationship includes a mapping relationship between environmental information, status information and delay optimization configuration information, and the first delay optimization configuration information includes a first delay optimization parameter when the vehicle's processing unit processes data collected by the sensor.

[0124] Optionally, the mapping relationship may also be referred to as a latency optimization configuration information retrieval library. The cloud server may store a latency optimization configuration information retrieval library, which includes a mapping relationship between environment information, state information, and latency optimization configuration information.

[0125] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0126] Exemplarily, the vehicle can send navigation information (for example, navigation information from the vehicle's current location to the destination) to a cloud server, and the cloud server can determine the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future based on the navigation information. The cloud server can determine the delay optimization configuration information at multiple moments in the future based on the environmental information around the vehicle at multiple moments in the future, the status information of the vehicle at multiple moments in the future, and the mapping relationship, and send it to the vehicle. In this way, the vehicle can save the delay optimization configuration information at multiple moments in the future in advance. When the environmental information around the vehicle changes, and / or the status information of the vehicle changes, the data collected by the sensor can be processed by selecting appropriate delay optimization configuration information from the delay optimization configuration information at multiple moments in the future. In this way, it helps to avoid frequent interactions between the vehicle and the cloud server.

[0127] Exemplarily, the vehicle can also determine the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future based on the navigation information, and then send the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future to the cloud server.

[0128] The first delay optimization configuration information may include Figure 2 The delay optimization parameters when at least some of the nodes 2, 3 and 4 are shown to process data collected by sensors.

[0129] Exemplarily, the latency optimization parameters include one or more of priority information of tasks in the processing unit, core binding information, and scheduling parameters of a scheduler in an operating system (eg, a Linux system).

[0130] Optionally, the method 300 also includes: when the vehicle determines that the delay for processing the data collected by the sensor is greater than or equal to the delay threshold based on the first delay optimization parameter, obtaining second environmental information around the vehicle and second state information of the vehicle; the vehicle sends the second environmental information and the second state information to the cloud server; the cloud server sends second delay optimization configuration information to the vehicle based on the second environmental information, the second state information and the mapping relationship, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the vehicle processes the second data collected by the sensor based on the second delay optimization parameter.

[0131] For example, in T 1 At the moment, the road type where the vehicle is located is a highway, the road speed limit is 120 kilometers per hour (km / h), and the CPU usage of the vehicle is 50%. The vehicle can send this information to the cloud server, and the cloud server determines the delay optimization configuration information 1 according to the mapping relationship and sends it to the vehicle. The vehicle can process the data collected by the sensor according to the delay configuration information 1.

[0132] In T 2 Time (for example, T 1 At a certain time after the time instant, the road type where the vehicle is located is an urban road, the road speed limit is 80 km / h and the CPU occupancy rate of the vehicle is 80%. When the vehicle processes the data collected by the sensor according to the delay optimization configuration information 1, the delay threshold (for example, 100 ms) is greater than or equal to the delay threshold, the vehicle can 2 The corresponding environment information and status information at the time are sent to the cloud server, and the cloud server then 2 The environment information and status information at the moment, as well as the mapping relationship determine the delay optimization configuration information 2 and send it to the vehicle. The vehicle can process the data collected by the sensor according to the delay configuration information 2. When the delay of the processing unit processing the sensor collected data is greater than or equal to the delay threshold due to changes in the environment information and / or status information, the vehicle can request new delay optimization configuration information from the cloud server, so that the vehicle can process the data collected by the sensor through the new delay optimization configuration information, which helps to reduce the end-to-end delay in a timely manner, and also helps to improve the safety of the vehicle and the driving safety of the user.

[0133] Optionally, before the cloud server sends the first delay optimization configuration information to the vehicle, the method 300 also includes: according to the first environment information and the first status information, using the HNSW algorithm to retrieve the mapping relationship to obtain the first delay optimization configuration information.

[0134] Optionally, before the cloud server receives the first environmental information and the first status information sent by the vehicle, the method 300 also includes: the cloud server obtains the third environmental information around the autonomous driving vehicle and the third status information of the autonomous driving vehicle; determines the first delay when processing the data collected by the sensor according to the third delay optimization parameter set by the offline simulation system, the third environmental information and the third status information; updates the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; and the cloud server saves the correspondence between the third environmental information, the third status information and the fourth delay optimization parameter in the mapping relationship.

[0135] The above-mentioned autonomous driving vehicle can be understood as a road test vehicle, and the third environment information and the third state information can be information collected by the road test vehicle.

[0136] The third delay optimization parameter may be a preset delay optimization parameter, or may be a delay optimization parameter optimized last time. For example, the preset delay optimization parameter indicates that the priorities of multiple tasks in the processing unit are the same.

[0137] Exemplarily, the cloud server may store an offline simulation system, and the cloud server may set the priority of the autonomous driving application process and the parameters corresponding to the core binding (i.e., the third delay optimization parameter) on the offline simulation system, and input the third environment information and the third state information into the offline simulation system. The cloud server may monitor the end-to-end delay during actual operation on the offline simulation system. The cloud server may update the third delay optimization parameter according to the end-to-end delay during the actual operation, thereby obtaining a fourth delay optimization parameter. The cloud server may save the correspondence between the third environment information, the third state information, and the fourth delay optimization parameter in the mapping relationship.

[0138] When the cloud server updates the third delay optimization parameter according to the end-to-end delay during the actual operation, the fourth delay optimization parameter can be obtained by repeatedly iterating the heuristic AI tuning algorithm, the reinforcement learning algorithm, and the deep learning algorithm. In this way, the end-to-end delay calculation method is more accurate and no additional simulation verification is required before deployment on the cloud server.

[0139] Optionally, before the cloud server receives the first environmental information and the first state information sent by the vehicle, the method 300 also includes: the cloud server obtains third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; the cloud server extracts delay model data based on the third environmental information and the third state information data, and the delay model data includes one or more of the number of application threads, periodic data of the threads, time probability distribution of thread running, and dependency relationship of thread data; the cloud server inputs the delay model data into a delay simulator to obtain a second delay; the cloud server determines a fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; the cloud server saves the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0140] Exemplarily, the cloud server can extract the delay model data of the autonomous driving system and application under the third environmental information through the third environmental information and the third state information sent by the autonomous driving vehicle. The delay model data includes but is not limited to one or more of the number of application threads, the periodic data of the threads, the probability distribution of the thread running time, and the dependency of the thread data. Then, the end-to-end delay is directly calculated by the delay simulator, and the fourth delay optimization parameter is obtained by repeated iteration using a heuristic AI tuning algorithm, reinforcement learning or deep learning tuning algorithm. The advantage of this method is that the overhead of simulation delay estimation is small, large-scale parallel optimization can be performed, and the convergence speed is fast.

[0141] Optionally, before the cloud server receives the first environmental information and the first status information sent by the vehicle, the method 300 also includes: the cloud server obtains the third environmental information around the autonomous driving vehicle and the third status information of the autonomous driving vehicle; the cloud server inputs the third environmental information and the third status information data into the prediction model to obtain a fourth delay optimization parameter; the cloud server saves the third environmental information, the third status information and the fourth delay optimization parameter in the mapping relationship.

[0142] Exemplarily, the prediction model can be trained using a training data set, which includes the environment information of the vehicle, the state information of the vehicle, and the delay optimization configuration information corresponding to the environment information and the state information (task priority information, core binding information, scheduling parameters). The trained prediction model can be saved in a cloud server. In this way, after the autonomous driving vehicle (or, road test vehicle) uploads the road test data (including the environment information around the road test vehicle and the state information of the road test vehicle) to the cloud server, the cloud server can input the road test data into the prediction model to obtain the corresponding delay optimization configuration information. The cloud server can save the corresponding relationship between the environment information, the state information, and the delay optimization configuration information in the mapping relationship.

[0143] Optionally, the cloud server obtains third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle, including: the cloud server receives the third state information from the autonomous driving vehicle and data collected by sensors in the autonomous driving vehicle; the cloud server indexes the third state information and the data collected by the sensors according to time and location to obtain the third environmental information and the third state information.

[0144] Optionally, the cloud server indexes the third state information and the data collected by the sensor according to time and location to obtain the third environment information and the third state information, including: indexing the third state information and the data collected by the sensor according to time and location through a crowdsourcing algorithm or a group intelligence perception algorithm to obtain the third environment information and the third state information.

[0145] S340: The vehicle processes the first data collected by the sensor according to the first time delay optimization parameter.

[0146] Optionally, the vehicle includes a main processing unit and a backup processing unit, and the vehicle processes the first data collected by the sensor according to the first delay optimization parameter, including: the vehicle deploys the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit in processing the first data, and the vehicle deploys the first delay optimization parameter in the main processing unit.

[0147] Exemplarily, the main processing unit may be a main domain, and the backup processing unit may be a secure backup domain. Alternatively, the main processing unit may be a main processor, and the backup processing unit may be a backup processor.

[0148] The delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit in processing the first data. It can be understood that within a preset time length, the average value of the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the average value of the delay of the main processing unit in processing the first data; and / or, the maximum value of the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the maximum value of the delay of the main processing unit in processing the first data.

[0149] Optionally, the vehicle deploys the first delay optimization parameter before the main processing unit, and the method 300 further includes: within the first time period, determining that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0150] Figure 4 A schematic diagram of the system architecture provided by an embodiment of the present application is shown. The system architecture includes a vehicle 410 and a cloud server 420, wherein the vehicle 410 may include a software platform and a hardware platform, the software platform includes an autonomous driving application 411 and an operating system and middleware 412, and the operating system and middleware 412 include a scene-aware delay monitoring module 4121 and a delay optimization request and configuration module 4122. The hardware platform includes a CPU 413, a GPU 414, a memory 415, a network card 416, a sensor 417, and a controller 418, etc.

[0151] The cloud server 420 may include an offline scene simulation database 421, an offline scene simulation system 422, and a delay feature vector retrieval library 423. The delay feature vector retrieval library may store the above mapping relationship.

[0152] The scene-aware delay monitoring module 4121 can be used to perceive the environmental information around the vehicle 410 and the end-to-end delay changes in the current scene, and upload the environmental information and the status information of the vehicle 410 to the cloud server 420.

[0153] The above scene-aware delay monitoring module 4121 can upload environmental information to the cloud server 420, or can also upload data collected by the sensor 417 to the cloud server 420. The cloud server 420 can determine the environmental information around the vehicle 410 based on the data collected by the sensor 417.

[0154] The delay optimization request and configuration module 4122 can be used to send a request to the cloud server 420 to optimize the end-to-end delay in the current scenario, and to receive the delay optimization configuration information sent by the cloud server 420 and configure the delay optimization configuration information on the vehicle 410.

[0155] The offline scene simulation database 421 can be used to record environmental data and vehicle status information of different vehicles in different scenes such as different roads and weather, including but not limited to point cloud data collected by lidar, image data collected by cameras, point cloud data collected by millimeter wave radar, data collected by GPS and inertial sensors, etc., as well as operation logs of various hardware and software in the vehicle, which can be subsequently used for autonomous driving scene playback, simulation, etc.

[0156] The offline scenario simulation system 422 can be used to read the environmental information and status information in the offline scenario simulation database 421, and provide automatic driving scenario-level simulation and simulation functions for different times, climates, geographical locations and road conditions.

[0157] The delay feature vector retrieval library 423 can be used to record environmental information, status information and delay optimization configuration information calculated by the offline scenario simulation system 422 in different scenarios. The environmental information includes but is not limited to time, geographical location, climate conditions, road conditions and structure, etc., and the status information includes but is not limited to hardware specifications, system status, automatic driving status, etc.

[0158] CPU 413 or GPU 414 can be used to provide necessary computing resources for autonomous driving systems and applications.

[0159] The network card 416 can be used to provide network communication resources for the autonomous driving system and applications.

[0160] Memory 415 can be used to provide storage resources for autonomous driving systems and applications.

[0161] Sensor 417 can be used to provide perception data resources for autonomous driving systems and applications.

[0162] Controller 418 can be used to provide chassis control capability services for the autonomous driving vehicle.

[0163] The autonomous driving application 411 obtains and controls the CPU resources, GPU resources, network card resources, memory resources, and data collected by sensors required for operation through the operating system and middleware 412, and controls the operation of the vehicle 410 through the controller 418.

[0164] The scene-aware delay monitoring module 4121 analyzes the environment information, state information and end-to-end delay information of the vehicle 410 from the sensor data and application processing data during the operation of the autonomous driving application 411, which can be stored in the memory 415 or directly sent to the cloud server 420 through the network card 416. The cloud server 420 can record this information in the offline scene simulation database 421.

[0165] The delay optimization request and configuration module 4122 can obtain the end-to-end delay from the scene-aware delay monitoring module 4121. When the end-to-end delay needs to be optimized, it will send an optimization request to the cloud server 420 through the network card 416, and receive the delay optimization configuration information sent by the delay feature vector retrieval library 423, and then deploy it on the current autonomous driving operating system and application.

[0166] Figure 5 Another schematic diagram of the system architecture provided by the embodiment of the present application is shown. The scene-aware delay monitoring module 4121 in the system architecture may include a collection monitoring module 41211 and a dynamic configuration module 41212, and the delay optimization request and configuration module 4122 may include a network communication module 41221 and a configuration deployment module 41222. The cloud server 420 may also include a scheduling and orchestration automatic optimization system 424 and a delay feature vector retrieval module 425.

[0167] The acquisition monitoring module 41211 can use the scheduling, communication and other log information of the running process of the autonomous driving application and combine with the algorithm to obtain the environmental information around the vehicle 410 and the status information of the vehicle 410. Then, the information is sent to the cloud server 420 through the network communication module 41221. The cloud server 420 can record this information in the offline scene simulation database 421.

[0168] The dynamic configuration module 41212 can dynamically obtain the end-to-end delay of the vehicle 410 from the acquisition monitoring module 41211. When the current actual end-to-end delay is worse than expected, an optimization request can be sent to the cloud server 420 through the network communication module 41221. The optimization request may include environmental information around the vehicle 410 and the current state information of the vehicle 410. After the delay optimization configuration information is obtained from the cloud server 420, it is sent to the configuration deployment module 41222, which deploys it on the autonomous driving system and application, thereby optimizing the end-to-end delay.

[0169] The scheduling and orchestration automatic optimization system 424 can be used to regularly pull data from the offline scenario simulation database 421 through the offline scenario simulation system 422 on the cloud server 420 to simulate different autonomous driving scenarios and observe the end-to-end delay of the key link data flow of autonomous driving, and then automatically optimize the delay optimization configuration information corresponding to different environmental information and status information, and record the correspondence between the environmental information, status information and delay optimization configuration information in the delay feature vector retrieval library 423.

[0170] Figure 6 A schematic flow chart of a data processing method 600 provided in an embodiment of the present application is shown. The method 600 may be executed by a cloud server 420, and the method 600 includes:

[0171] S610 , constructing an offline simulation database 421 based on time and geographic location on the cloud server 420 .

[0172] Exemplarily, step S610 may include two stages:

[0173] The first stage is to automatically collect sensor data and status information during the operation of the autonomous driving vehicle (or road test vehicle) and upload it to the cloud server 420.

[0174] The second stage is to integrate the data uploaded by each autonomous driving vehicle (or road test vehicle) according to the geographical location and time in the cloud server 420 to form an offline scenario simulation database 421.

[0175] Exemplarily, the cloud server 420 integrates the data uploaded by each autonomous driving vehicle according to geographic location and time through crowdsourcing or group intelligence perception technology to form an offline scene simulation database 421.

[0176] The data that the autonomous driving vehicle needs to collect in Phase 1 includes, but is not limited to, point cloud data from lidar, image data from cameras, point cloud data from millimeter-wave radars, data from sensors such as GPS and IMU, as well as information such as weather, temperature, humidity, light intensity, and visibility. Phase 2 will use the time series and geographic location coordinates of the data from different sources in Phase 1 as the main index, and use group intelligence perception algorithms or crowdsourcing algorithms to build a unified offline scene simulation database from the scattered data, which can be replayed and simulated in various autonomous driving scenarios for future testing and optimization.

[0177] S620, through the offline simulation system 422 and the scheduling and scheduling automatic optimization system 424, the correspondence between the environmental information around the vehicle, the vehicle status information and the delay optimization configuration information in a certain time slice is obtained, and the correspondence is saved in the delay feature vector retrieval library 423.

[0178] Step S620 can be based on the data in the offline scenario simulation database 421 of step S610, and simulate and calculate the delay optimization configuration information corresponding to the environmental information and status information under a certain time slice on the cloud server 420, and then save the environmental information, status information and corresponding delay optimization configuration information under the time slice in the delay feature vector retrieval library 423.

[0179] Step S620 can be carried out after the offline scene simulation database 421 is initially constructed, and can also be carried out synchronously when the offline scene simulation database 421 is subsequently updated.

[0180] Exemplarily, the offline scenario simulation system 422 selects environmental information and state information at different time slices from the offline scenario simulation database 421 in a sampling or traversal manner, and simulates and replays the end-to-end delay of the autonomous driving system application in different scenarios through the offline scenario simulation system 422. Then, the delay optimization configuration information is determined through a machine learning algorithm.

[0181] The above steps S610 and S620 may be completed before the vehicle 410 travels.

[0182] S630, receiving an optimization request sent by the vehicle 410 and sending delay optimization configuration information to the vehicle 410, wherein the optimization request includes environmental information around the vehicle 410 and status information of the vehicle 410.

[0183] For example, the vehicle 410 may periodically upload the environment information around the vehicle 410 and the status information of the vehicle 410 to the cloud server 420. The cloud server 420 may retrieve the delay optimization configuration information from the delay feature vector retrieval library 423 according to the environment information and status information sent by the vehicle 410 and send it to the vehicle 410.

[0184] The above step S630 may occur while the vehicle 410 is driving.

[0185] Optionally, the latency optimization configuration information may include a latency range in addition to the task priority, core binding information, scheduling parameters, etc. When the latency of the vehicle 410 processing the data collected by the sensor according to the latency optimization configuration information is not within the latency range, the vehicle 410 may resend the optimization request to the cloud server.

[0186] The hardware in the above hardware platform can be an autonomous driving hardware processing platform, where the AI ​​CPU and AI Core provide CPU and GPU computing resources respectively, the Ethernet card provides Gigabit network card communication capabilities, the universal flash storage (UFS) provides large-capacity storage resources, the sensor provides point cloud data, image data, GPS and other data, and the controller provides chassis control capabilities. The above operating system can be a Linux system.

[0187] The acquisition and monitoring module 41211 of the autonomous driving hardware processing platform collects and records a large amount of raw sensor data during the autonomous driving process of the road test vehicle, including but not limited to the point cloud coordinate data of the lidar, the image and video data of the camera, the spatiotemporal coordinate data of the GPS, the timing data of the IMU, etc. Due to the large amount of data, the data at this stage will be stored in the UFS through the CM communication module during the autonomous driving process. When the road test is completed, the information stored in the UFS is directly uploaded to the cloud server through the Ethernet card or the memory reader / writer external workstation. At the same time, the status information of the vehicle during the autonomous driving process will also be uploaded to the cloud server, including but not limited to the frame information, vehicle status, hardware SoC resource usage information, network status, system software operation status, vehicle self-driving status information, etc.

[0188] After receiving different road test data from different vehicles, the cloud server analyzes the time, starting and ending positions, routes, etc. of each road test data, and then aligns the collected sensor data and status information data according to the time and GPS coordinate position, and records them as a spatiotemporal database of autonomous driving scene data, that is, an offline scene simulation database. At the same time, heuristic algorithms and neural network-based AI algorithms will also be used to automatically analyze the weather conditions, solar terms information, and road congestion conditions of each road test data, and establish indexes other than time and GPS coordinates to facilitate rapid retrieval and use by subsequent offline scene simulation systems.

[0189] The offline scene simulation system 422 of the cloud server 420 can call data from the offline scene simulation database 421 constructed in step S610 to simulate different autonomous driving scenes at different times, geographical locations, weather, road conditions, etc., and connect to the software and hardware platform or the autonomous driving system and application to measure the accuracy of the running results of the autonomous driving application and the performance of the entire system. Optionally, considering that the scene data (including environmental information and status information) collected and uploaded in the real world is limited, the offline scene simulation system uses an enhanced AI algorithm, which can splice the scene data in different road test data into the same simulation scene, and can also add vehicles, obstacles and pedestrians that did not exist in the playback data through generative AI technology, and even change the features in the original scene, such as changing the weather, traffic flow, road surface properties, etc.

[0190] During the offline scene simulation process, the offline scene simulation system extracts the environment information and state information within the time window at a fixed frequency (such as 1 Hz or 0.1 Hz) to form time serialization data. The time serialization data may be as shown in Table 1 above.

[0191] The offline scene simulation system 422 constructs the environmental information and state information extracted in all time slices into a delay feature vector retrieval library 423. These environmental information and state information can be independently used for future autonomous driving vehicle retrieval, and some of the data is scene-independent data, such as hardware specification feature data. Some data fluctuates greatly with time and scene changes, such as system state characteristics. After the delay feature vector retrieval library 423 is established, it can be regularly maintained. For example, when each autonomous driving software or hardware iteration is updated, the above steps S610 and S620 need to be repeated.

[0192] At the same time, the scheduling and scheduling automatic optimization system 424 based on the AI ​​algorithm performs end-to-end delay optimization for each time slice of the simulation, with environmental information and state information as input and delay optimization configuration information for the time slice as output. The scheduling and scheduling automatic optimization system 424 can update the output to the delay feature vector retrieval library 423, thereby establishing a correspondence between environmental information, state information, and delay optimization configuration information. In the embodiment of the present application, the delay optimization configuration information under the time slice can be determined by the two methods described below.

[0193] Method 1

[0194] Directly set the priority information and core binding information of the tasks in the autonomous driving application on the offline simulation system 422, observe the end-to-end delay during actual operation on the offline simulation system 422, and repeatedly iterate and calculate the delay optimization configuration information under the time slice by combining the heuristic AI tuning algorithm, reinforcement learning or deep learning tuning algorithm. This method one is to adjust the priority and core binding of the task. The advantage of this method one is that the end-to-end delay calculation method is more accurate and no additional simulation verification is required before deployment.

[0195] The above-mentioned offline simulation system 422 may include a vehicle dynamics simulation system and an automatic driving simulation system.

[0196] Method 2

[0197] First, the state information extracted within the time slice is combined to extract and generate the delay model data of the autonomous driving system and application corresponding to the environmental information under the time slice. The delay model data includes but is not limited to the number of application threads, the periodic data of the threads, the probability distribution of the thread running time, the dependency of the thread data, etc. The end-to-end delay is directly obtained by inputting the delay model data into the delay simulator, and the delay optimization configuration information under the time slice is repeatedly calculated by combining the heuristic AI tuning algorithm, reinforcement learning or deep learning tuning algorithm. The advantage of this second method is that the overhead of simulation delay estimation is small, large-scale parallel optimization can be performed, and the convergence speed is fast.

[0198] Taking into account the large amount of data in the delay feature vector retrieval library, the embodiment of the present application is based on HNSW technology and combines the idea of ​​a jump table to construct a fast retrieval layer for the delay feature vector for the original linked list. Because the delay feature vectors generated by the same road test data have strong locality, there are a large number of identical redundant data in the same time window and the same geographical location. Therefore, the delay feature vectors can be sorted according to the degree of change and the redundancy can be merged. For example, the hardware specifications, system versions and status are unchanged in the same road test. Sparse stratification is performed according to the density of the delay feature vector, starting with the sparse layer, and then querying the dense layer step by step if it meets the conditions. This can improve the retrieval speed of the delay optimization configuration information and reduce the response delay of the cloud server. Figure 7 An example diagram of a delay feature vector fast retrieval layer data structure provided in an embodiment of the present application is shown.

[0199] Figure 8 A schematic flow chart of a data processing method 800 provided in an embodiment of the present application is shown. The method 800 may be executed by a vehicle, and the method 800 includes:

[0200] S810, the vehicle determines whether the delay of the processing unit in processing the data collected by the sensor within the time window is greater than or equal to the delay threshold.

[0201] When the delay of processing the data collected by the sensor within the time window is greater than or equal to the delay threshold, execute S820; otherwise, return to continue executing S810.

[0202] Exemplarily, the above step S810 can be implemented by the following two methods:

[0203] Method 1

[0204] By using AI-based algorithms, the sensor-perceived data and vehicle status information are dynamically inferred, and pre-trained models are used to infer changes from external environment information, vehicle status information to the scene in which the vehicle is located.

[0205] Method 2

[0206] By monitoring the end-to-end delay of the vehicle. When the end-to-end delay mean or jitter exceeds the delay threshold within the preset time, it indicates that the external scene of the vehicle has changed and the delay optimization configuration information needs to be redeployed. The advantages of method 2 are that it occupies fewer resources of the autonomous driving software and hardware platform, has low computing overhead, and has a fast response speed and high accuracy in redeploying the delay optimization configuration information when the scene changes.

[0207] S820, the vehicle sends an optimization request to the cloud server, where the optimization request includes environmental information around the vehicle and status information of the vehicle.

[0208] Exemplarily, the dynamic configuration module 41212 sends the environmental information around the vehicle and the status information of the vehicle to the cloud server, and waits for the cloud server to re-search the delay optimization configuration information suitable for the current scene in the delay feature vector retrieval library and send it to the vehicle. The cloud server can reduce the search response time, and can use the massive computing resources in the cloud server to accurately find the appropriate delay optimization configuration information without occupying the local computing resources of the vehicle. When the version is updated, only the delay feature vector index library in the cloud server needs to be updated, and the vehicle is not aware of it.

[0209] The above is an example of a vehicle sending an optimization request to a cloud server when the time delay feature vector retrieval library is deployed on the cloud server, but the present application is not limited thereto. The time delay feature vector retrieval library can also be deployed in a vehicle.

[0210] Exemplarily, the cloud server can deploy the delay feature vector retrieval library on the vehicle's autonomous driving hardware processing platform through model lossy compression technology. When the vehicle is in an autonomous driving state, if the end-to-end delay of the vehicle is greater than or equal to the delay threshold, the dynamic configuration module 41212 can re-search the delay optimization configuration information suitable for the current scenario in the compressed delay feature vector retrieval library based on the environmental information around the vehicle and the vehicle's status information. In this way, the vehicle can obtain the updated delay optimization configuration information in a timely manner without being connected to the Internet, and can better comply with functional safety certification.

[0211] S830, the vehicle receives delay optimization configuration information determined by the cloud server based on the environmental information and status information, and the delay optimization configuration information includes delay optimization configuration parameters.

[0212] S840: The vehicle deploys the delay optimization configuration parameters on the safety backup domain.

[0213] S850: Within a preset time period, if the delay of processing the data collected by the sensors in the primary domain is greater than the delay of processing the data collected by the sensors in the safety backup domain, the vehicle deploys the delay optimization configuration information on the primary domain.

[0214] The above main domain and safety backup domain can process the data collected by the same sensor.

[0215] Exemplarily, after the dynamic configuration module obtains the latest delay optimization configuration information from the delay feature vector retrieval library of the cloud server, it sends it to the configuration deployment module 41222. The configuration deployment module 41222 deploys the delay optimization configuration information in the safety backup domain of autonomous driving to observe its end-to-end delay optimization effect. The safety backup domain and the main domain can both be on the autonomous driving hardware processing platform, and both deploy a full range of autonomous driving applications. If the vehicle control instructions output by the main domain and the safety backup domain are the same within a preset time length and the delay of the safety backup domain processing data collected by the sensor is better than the delay of the main domain processing data collected by the sensor, the configuration deployment module 41222 can deploy the delay optimization configuration information in the main domain.

[0216] Fig. 9 FIG. 9 is a schematic block diagram of a data processing device 900 provided in an embodiment of the present application. Fig. 9As shown, the device 900 includes: an acquisition unit 910, used to acquire first environmental information around a vehicle and first status information of the vehicle; a sending unit 920, used to send the first environmental information and the first status information to a cloud server; a receiving unit 930, receiving first delay optimization configuration information sent by the cloud server, the cloud server saves a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; the processing unit 940, used to process the first data collected by the sensor according to the first delay optimization parameter.

[0217] Optionally, the acquisition unit 910 is used to obtain second environmental information around the vehicle and second state information of the vehicle when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the sending unit 920 is used to send the second environmental information and the second state information to the cloud server; the receiving unit 930 is used to receive second delay optimization configuration information sent by the cloud server, the cloud server stores a mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information includes the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit 940 is used to process the second data collected by the sensor according to the second delay optimization parameter.

[0218] Optionally, the processing unit 940 includes a main processing unit and a backup processing unit, and the processing unit 940 is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay when the backup processing unit processes the first data according to the first delay optimization parameter is less than the delay when the main processing unit processes the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0219] Optionally, the device 900 further includes: a determination unit, configured to determine, within the first time period, whether the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0220] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at multiple moments in the future.

[0221] Optionally, the first environmental information includes at least one of time, geographic location, climate and road conditions; and / or, the first status information includes at least one of the vehicle's hardware specifications, system status, and autonomous driving status.

[0222] For example, the acquisition unit 910 may be Figure 1 The computing platform in the computing platform or the processing circuit, processor or controller in the computing platform. Taking the acquisition unit 910 as the processor 121 in the computing platform as an example, the processor 121 can acquire the environment information around the vehicle and the state information of the vehicle.

[0223] For another example, the sending unit 920 and the receiving unit 930 may be communication devices in the vehicle 100 .

[0224] For another example, the processing unit 940 may be Figure 1 The computing platform or the processing circuit, processor or controller in the computing platform. Taking the processing unit 940 as the processor 122 in the computing platform as an example, the processor 122 can process the data collected by the sensor according to the delay optimization configuration information sent by the cloud server.

[0225] The functions implemented by the above acquisition unit 910 and the functions implemented by the processing unit 940 can be implemented by different processors, or can also be implemented by the same processor, which is not limited in the embodiment of the present application.

[0226] Fig.10 FIG. 1 shows a schematic flow chart of a data processing device 1000 provided in an embodiment of the present application. Fig.10 As shown, the device 1000 includes: an acquisition unit 1010, used to acquire first environmental information around a vehicle and first state information of the vehicle; a determination unit 1020, used to determine first delay optimization configuration information according to the first environmental information, the first state information and a mapping relationship, the mapping relationship including a mapping relationship between the first environmental information, the first state information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; the processing unit 1030, used to process the first data collected by the sensor according to the first delay optimization parameter.

[0227] Optionally, the acquisition unit 1010 is also used to obtain second environmental information around the vehicle and second state information of the vehicle when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the determination unit 1020 is also used to determine second delay optimization configuration information based on the second environmental information, the second state information and the mapping relationship, the mapping relationship including the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit 1030 is also used to process the second data collected by the sensor according to the second delay optimization parameter.

[0228] Optionally, the processing unit 1030 includes a main processing unit and a backup processing unit, and the processing unit 1030 is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0229] Optionally, the determination unit 1020 is further used to determine, within the first time period, that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0230] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0231] For example, the acquisition unit 1010 may be Figure 1 The computing platform in the computing platform or the processing circuit, processor or controller in the computing platform. Taking the acquisition unit 1010 as the processor 121 in the computing platform as an example, the processor 121 can acquire the environmental information around the vehicle and the status information of the vehicle.

[0232] For another example, the determining unit 1020 may be Figure 1 The computing platform in the computing platform or the processing circuit, processor or controller in the computing platform. Taking the determination unit 1020 as the processor 122 in the computing platform as an example, the processor 122 may store a mapping relationship between the environment information, the state information and the delay optimization configuration information. The processor 122 may determine the appropriate delay optimization configuration information in the current scenario according to the acquired environment information, the state information and the mapping relationship.

[0233] For another example, the processing unit 1030 may be Figure 1 The computing platform or the processing circuit, processor or controller in the computing platform. Taking the control unit 1030 as the processor 123 in the computing platform as an example, the processor 123 can delay the optimization configuration information and process the data collected by the sensor.

[0234] The functions implemented by the acquisition unit 1010, the functions implemented by the determination unit 1020, and the functions implemented by the processing unit 1030 may be implemented by different processors, or may be implemented by the same processor, or some functions may be implemented by the same processor. This embodiment of the application does not limit this.

[0235] Fig.11 FIG. 1 shows a schematic block diagram of a data processing device 1100 provided in an embodiment of the present application. Fig.11 As shown, the device 1100 includes: an acquisition unit 1110, used to acquire first environmental information around a vehicle and first state information of the vehicle; a determination unit 1120, used to determine first delay optimization configuration information according to the first environmental information, the first state information and a mapping relationship, the mapping relationship including a mapping relationship between environmental information, state information and delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; a sending unit 1130, used to send the first delay optimization configuration information to the vehicle.

[0236] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0237] Optionally, the device 1100 further includes: an information retrieval unit, configured to retrieve the mapping relationship by using a hierarchical navigable small-world HNSW algorithm according to the first environment information and the first state information to obtain the first delay optimization configuration information.

[0238] Optionally, the device 1100 also includes a parameter updating unit and a storage unit. The acquisition unit 1110 is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and first state information sent by the vehicle; the determination unit 1120 is further used to determine the first delay when processing data collected by the sensor according to a third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; the parameter updating unit is used to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; and the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0239] Optionally, the device also includes a delay simulation unit, and the acquisition unit 1110 is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and the first state information sent by the vehicle; the determination unit 1120 is further used to extract delay model data based on the third environmental information and the third state information data, and the delay model data includes one or more of the number of threads of the application, the periodic data of the threads, the time probability distribution of the thread running and the dependency relationship of the thread data; the delay simulation unit is used to input the delay model data into the delay simulator to obtain a second delay; the determination unit 1120 is further used to determine the fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0240] Optionally, the acquisition unit 1110 is used to: receive data collected by the sensor of the autonomous driving vehicle and the third state information; index the data collected by the sensor and the third state information according to time and location to obtain the third environment information and the third state information.

[0241] Optionally, the acquisition unit 1110 is used to: index the data collected by the sensor and the third state information according to time and location through a crowdsourcing algorithm or a group perception algorithm to obtain the third environment information and the third state information.

[0242] It should be understood that the division of the units in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the units in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the device, wherein the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor, and the memory is a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD. Taking FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured through the configuration file, so as to realize the functions of some or all of the above units. All units of the above device may be implemented entirely in the form of a processor calling software, or entirely in the form of a hardware circuit, or partially in the form of a processor calling software and the rest in the form of a hardware circuit.

[0243] Each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0244] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of SoC. The SoC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0245] An embodiment of the present application also provides a device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device executes the method or steps executed by the above embodiment.

[0246] Alternatively, if the device is located in a vehicle, the processing unit may be Figure 1 Processors 121-12n are shown.

[0247] An embodiment of the present application further provides a data processing system, the path planning system comprising one or more sensors and a computing platform, wherein the computing platform comprises the above-mentioned data processing device 900 or data processing device 1000.

[0248] An embodiment of the present application further provides a vehicle, which may include the above-mentioned data processing device 900, or, include the above-mentioned data processing device 1000, or, include the above-mentioned data processing system.

[0249] The embodiment of the present application further provides a server, which includes the above-mentioned data processing device 1100.

[0250] An embodiment of the present application further provides a computer program product, which includes: a computer program code, and when the computer program code is run on a computer, the computer executes the above-mentioned data processing method.

[0251] The embodiment of the present application further provides a computer-readable medium, wherein the computer-readable medium stores a program code. When the computer program code is executed on a computer, the computer executes the above-mentioned data processing method.

[0252] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or a power-on erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0253] It should be understood that in the embodiment of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.

[0254] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0255] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0260] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0261] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered. Within the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: Acquiring first environmental information around a vehicle and first state information of the vehicle; Sending the first environment information and the first status information to a cloud server; Receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environment information, the first state information, and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; The first data collected by the sensor is processed according to the first delay optimization parameter.

2. The method according to claim 1, characterized in that The method further comprises: When the delay of the processing unit for processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to a delay threshold, acquiring second environment information around the vehicle and second state information of the vehicle; Sending the second environment information and the second status information to the cloud server; receiving second delay optimization configuration information sent by the cloud server, wherein the cloud server stores a mapping relationship between the second environment information, the second state information, and the second delay optimization configuration information, and the second delay optimization configuration information includes a second delay optimization parameter when the processing unit processes the data collected by the sensor; The second data collected by the sensor is processed according to the second delay optimization parameter.

3. The method according to claim 1 or 2, characterized in that: The processing unit includes a main processing unit and a backup processing unit, and the processing of the first data collected by the sensor according to the first delay optimization parameter includes: deploying the first delay optimization parameter in the backup processing unit; In a first time period, when the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, the first delay optimization parameter is deployed in the main processing unit.

4. The method according to claim 3, characterized in that The method further comprises: deploying the first delay optimization parameter before the main processing unit: In the first time period, it is determined that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

5. The method according to any one of claims 1 to 4, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay configuration information includes delay optimization parameters when the processing unit processes the data collected by the sensor at multiple moments in the future.

6. The method according to any one of claims 1 to 5, characterized in that The first environmental information includes at least one of time, geographical location, climate and road conditions; and / or, The first status information includes at least one of the hardware specifications, system status, and automatic driving status of the vehicle.

7. A data processing method, characterized in that: include: Acquiring first environmental information around a vehicle and first state information of the vehicle; Based on the first environmental information, the first state information and the mapping relationship, the first delay optimization configuration information is sent to the vehicle, the mapping relationship includes a mapping relationship between environmental information, state information and delay optimization configuration information, and the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes the data collected by the sensor.

8. The method according to claim 7, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay optimization configuration information includes delay optimization parameters for processing the data collected by the sensor at multiple future moments.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: According to the first environment information and the first state information, a hierarchical navigable small-world HNSW algorithm is used to retrieve the mapping relationship to obtain the first delay optimization configuration information.

10. The method according to any one of claims 7 to 9, characterized in that Before acquiring the first environment information around the vehicle and the first state information of the vehicle, the method further includes: Acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; Determine a first delay when processing data collected by the sensor according to a third delay optimization parameter set by the offline simulation system, the third environment information, and the third state information; According to the first delay, the third delay optimization parameter is updated to obtain a fourth delay optimization parameter; The correspondence between the third environment information, the third state information and the fourth delay optimization parameter is stored in the mapping relationship.

11. The method according to any one of claims 7 to 9, characterized in that Before acquiring the first environment information around the vehicle and the first state information of the vehicle, the method further includes: Acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; Extracting latency model data according to the third environment information and the third state information, where the latency model data includes one or more of the number of application threads, periodicity data of threads, time probability distribution of thread running, and dependency relationship of thread data; Inputting the delay model data into a delay simulator to obtain a second delay; determining, according to the second delay, a fourth delay optimization parameter corresponding to the third environment information and the third state information; The correspondence between the third environment information, the third state information and the fourth delay optimization parameter is stored in the mapping relationship.

12. A data processing device, characterized in that: include: An acquisition unit, configured to acquire first environment information around a vehicle and first state information of the vehicle; A sending unit, configured to send the first environment information and the first status information to a cloud server; a receiving unit, configured to receive first delay optimization configuration information sent by the cloud server, wherein the cloud server stores a mapping relationship between the first environment information, the first state information, and the first delay optimization configuration information, wherein the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; The processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

13. The device according to claim 12, characterized in that The acquisition unit is further configured to acquire second environment information around the vehicle and second state information of the vehicle when the delay of the processing unit for processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to a delay threshold; The sending unit is further used to send the second environment information and the second state information to the cloud server; The receiving unit is further used to receive second delay optimization configuration information sent by the cloud server, the cloud server saves a mapping relationship between the second environment information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information includes a second delay optimization parameter when the processing unit processes the data collected by the sensor; The processing unit is further configured to process the second data collected by the sensor according to the second delay optimization parameter.

14. The device according to claim 12 or 13, characterized in that The processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to: deploying the first delay optimization parameter in the backup processing unit; In a first time period, when the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, the first delay optimization parameter is deployed in the main processing unit.

15. The device according to claim 14, characterized in that The device also includes: A determination unit is used to determine that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit within the first time period before the processing unit deploys the first delay optimization parameter in the main processing unit.

16. The device according to any one of claims 12 to 15, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay configuration information includes delay optimization parameters when the processing unit processes the data collected by the sensor at multiple moments in the future.

17. The device according to any one of claims 12 to 16, characterized in that The first environmental information includes at least one of time, geographical location, climate and road conditions; and / or, The first status information includes at least one of the hardware specifications, system status, and automatic driving status of the vehicle.

18. A data processing device, characterized in that: include: An acquisition unit, configured to acquire first environment information around a vehicle and first state information of the vehicle; a determining unit, configured to determine first delay optimization configuration information according to the first environment information, the first state information, and a mapping relationship, wherein the mapping relationship includes a mapping relationship between environment information, state information, and delay optimization configuration information, and the first delay optimization configuration information includes a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; A sending unit is used to send the first delay optimization configuration information to the vehicle.

19. The device according to claim 18, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay optimization configuration information includes delay optimization parameters for processing the data collected by the sensor at multiple future moments.

20. The device according to claim 18 or 19, characterized in that The device also includes: An information retrieval unit is used to retrieve the mapping relationship by using a hierarchical navigable small-world HNSW algorithm according to the first environment information and the first state information to obtain the first delay optimization configuration information.

21. The device according to any one of claims 18 to 20, characterized in that The device also includes a parameter updating unit and a storage unit. The acquisition unit is further used to acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; The determining unit is further used to determine a first delay for processing data collected by the sensor according to a third delay optimization parameter set by the offline simulation system, the third environment information and the third state information; The parameter updating unit is further configured to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; The storage unit is further configured to store a correspondence between the third environment information, the third state information and the fourth delay optimization parameter in the mapping relationship.

22. The device according to any one of claims 18 to 20, characterized in that The device also includes a time delay simulation unit and a storage unit. The acquisition unit is further used to acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; The determining unit is further configured to extract latency model data according to the third environment information and the third state information, wherein the latency model data includes one or more of the number of application threads, periodicity data of threads, time probability distribution of thread running, and dependency relationship of thread data; The delay simulation unit is used to input the delay model data into a delay simulator to obtain a second delay; The determining unit is further configured to determine, according to the second delay, a fourth delay optimization parameter corresponding to the third environment information and the third state information; The storage unit is used to store the corresponding relationship between the third environment information, the third state information and the fourth delay optimization parameter in the mapping relationship.

23. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program stored in the memory, so that the apparatus performs the method according to any one of claims 1 to 6.

24. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program stored in the memory, so that the apparatus performs the method according to any one of claims 7 to 11.

25. A vehicle, characterized in that: Comprising the apparatus as claimed in any one of claims 12 to 17, or comprising the apparatus as claimed in claim 23.

26. A server, characterized in that: Comprising the apparatus as claimed in any one of claims 18 to 22, or comprising the apparatus as claimed in claim 24.

27. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 11 is implemented.

Images