Data processing method and apparatus, and vehicle

Abstract

The present application can be applied to the field of intelligent driving. Provided are a data processing method and apparatus, and a vehicle. The data processing method comprises: acquiring first environmental information of the surroundings of a vehicle and first state information of the vehicle; sending the first environmental information and the first state information to a cloud server; receiving first delay optimization configuration information sent by the cloud server, wherein the cloud server stores mapping relationships between the first environmental information, the first state information and the first delay optimization configuration information, and the first delay optimization configuration information comprises a first delay optimization parameter for when a processing unit in the vehicle processes data collected by a sensor; and according to the first delay optimization parameter, processing first data collected by the sensor. The embodiments of the present application can be applied to an intelligent vehicle or a new-energy vehicle, facilitate reducing an end-to-end delay, and thus facilitate ensuring the driving safety of a user.

Description

Data processing method, device and vehicle

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 29, 2023, with application number 202310792509.3 and invention name “Data processing method, device and vehicle”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of intelligent driving, and more specifically, to a data processing method, device and vehicle. Background Art

[0003] With the rapid development of autonomous driving artificial intelligence (AI) algorithms and technologies in recent years, a large number of traditional automakers and emerging autonomous driving companies have entered the market to develop high-level, fully autonomous driving systems and applications targeting Level 4 (L4). However, there are certain concerns about the implementation of autonomous driving technology, one of the most pressing issues being whether it can ensure the safety of other vehicles and pedestrians. A very important safety metric is the end-to-end latency of the autonomous driving system's data flow link, such as the time from the acquisition of raw signals by sensors such as lidar and cameras to the issuance of braking commands to the chassis domain controller. Low end-to-end latency (e.g., less than 100ms) can significantly improve vehicle and pedestrian safety and is a key goal for subsequent system and application optimization.

[0004] Therefore, how to reduce end-to-end latency has become an urgent problem to be solved.

[0005] Summary of the Invention

[0006] The present application provides a data processing method, device, and vehicle, which help reduce end-to-end latency, thereby helping to ensure user driving safety.

[0007] In the first aspect, the present application provides a data processing method, which includes: obtaining first environmental information around a vehicle and first status information of the vehicle; sending the first environmental information and the first status information to a cloud server; receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; and processing the first data collected by the sensor according to the first delay optimization parameter.

[0008] Based on the above technical solution, the cloud server can send corresponding latency optimization configuration information to the vehicle based on the vehicle's surrounding environment information, vehicle status information, and mapping relationships. In this way, the vehicle can obtain the appropriate latency optimization configuration information for the current scenario from the cloud server, helping to reduce the latency of the vehicle's processing unit when processing data collected by sensors, thereby helping to reduce end-to-end latency and ensuring user driving safety.

[0009] The mapping relationship stored in the above cloud server may be a mapping relationship between environment information, state information, and latency optimization configuration information in different scenarios.

[0010] When the front-end to end delay optimization configuration is basically configured offline based on the experience of engineers for some universal environments, it is difficult to make adaptive dynamic adjustments for different scenarios during vehicle driving, and the current end-to-end delay optimization configuration adjustment process is very time-consuming. In the embodiment of the present application, by deploying delay optimization configuration information for different scenarios in advance on the cloud server, the vehicle can obtain appropriate delay optimization configuration information in different scenarios. At the same time, the cloud server determines the appropriate delay optimization configuration information for the vehicle's current scenario through a mapping relationship, which can also reduce the waiting time for the vehicle to obtain the delay optimization configuration information.

[0011] In some possible implementations, the first latency optimization parameter includes one or more of a priority parameter of a task in a processing unit, a core binding parameter, and a scheduling parameter of a scheduler in an operating system.

[0012] In some possible implementations, the first delay optimization configuration information includes end-to-end delay optimization parameters, where the end-to-end delay is the end-to-end delay of the data flow link of the autonomous driving system in the vehicle, such as the delay from the collection of raw data by the sensor (for example, camera, lidar, etc.) to the issuance of vehicle control instructions to the vehicle controller (for example, chassis controller).

[0013] The above end-to-end delay may include the delay of the above processing unit processing the data collected by the sensor.

[0014] In some possible implementations, the first environmental information includes at least one of time, geographic location, climate, and road conditions.

[0015] In some possible implementations, the first status information includes at least one of the vehicle's hardware specifications, system status, and autonomous driving status.

[0016] In some possible implementations, the vehicle is a vehicle in an autonomous driving state.

[0017] In combination with the first aspect, in a possible implementation of the first aspect, the method also includes: when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold, obtaining second environmental information around the vehicle and second status information of the vehicle; sending the second environmental information and the second status information to the cloud server; receiving second delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the second environmental information, the second status information and the second delay optimization configuration information, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; and processing the second data collected by the sensor according to the second delay optimization parameter.

[0018] Based on the above technical solution, when the delay of the processing unit processing the data collected by the sensor during the vehicle's driving is greater than or equal to the delay threshold, the vehicle can send the second environmental information around it and the second status information of the vehicle to the cloud server. The cloud server can send updated delay optimization configuration information to the vehicle based on the mapping relationship. In this way, the vehicle can adaptively adjust the delay optimization configuration information during driving to achieve the effect of automatic online performance optimization, which helps to reduce end-to-end delay and thus helps to ensure the driving safety of users. At the same time, when the vehicle adjusts the deployed delay optimization configuration information, there is no need for the participation of existing autonomous driving application developers, such as modifying code or recompiling.

[0019] While driving, the vehicle can dynamically perceive performance degradation due to changing scenarios by extracting environmental information and vehicle status information within the current time window. Based on the changed scenario, the cloud server determines the appropriate latency optimization configuration for the changed scenario.

[0020] In some possible implementations, when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the first delay threshold or less than or equal to the second delay threshold, second environmental information around the vehicle and second state information of the vehicle are obtained, wherein the second delay threshold is less than the first delay threshold.

[0021] Alternatively, when the delay of the processing unit in processing the data collected by the sensor according to the first delay optimization parameter is not within the delay range, second environmental information around the vehicle and second state information of the vehicle are obtained.

[0022] In some possible implementations, the first delay optimization configuration information includes the delay threshold or the delay range.

[0023] Based on the above technical solution, the vehicle can obtain the latency range through the first latency optimization configuration information sent by the cloud server, thereby monitoring in real time the relationship between the latency of the processing unit processing sensor data and the latency range. If the latency falls outside the latency range, the vehicle can request new latency optimization configuration information from the cloud server, thereby avoiding safety risks caused by deteriorating end-to-end latency.

[0024] In combination with the first aspect, in a possible implementation of the first aspect, the processing unit includes a main processing unit and a backup processing unit, and processes the first data collected by the sensor according to the first delay optimization parameter, including: deploying the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0025] Based on the above technical solution, by first deploying the first delay optimization parameter to the backup processing unit, the validity of the first delay optimization parameter can be verified in advance without affecting the normal driving of the vehicle.

[0026] In combination with the first aspect, in a possible implementation of the first aspect, the first delay optimization parameter is deployed before the main processing unit, and the method also includes: within the first time period, determining that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit.

[0027] Based on the above technical solution, before deploying the first delay optimization parameters to the main processing unit, it can be determined that the vehicle control instructions output by the backup processing unit within the first time period are the same as the vehicle control instructions output by the main processing unit, which helps to improve the effectiveness of verifying the first delay optimization configuration information.

[0028] In combination with the first aspect, in a possible implementation of the first aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at the multiple moments in the future.

[0029] Based on the above technical solution, the vehicle can obtain the delay configuration parameters at multiple moments in the future in advance. This avoids the frequent interactions between the vehicle and the cloud server in the future, and helps to reduce the waiting time for the vehicle to obtain delay optimization configuration information.

[0030] In some possible implementations, the method includes: the vehicle sends navigation information to a cloud server, and the cloud server determines the vehicle's environmental information and status information at multiple future moments based on the navigation information, thereby sending to the vehicle the delay optimization parameters for the processing unit to process the data collected by the sensor at multiple future moments.

[0031] In a second aspect, a data processing method is provided, which includes: obtaining first environmental information around a vehicle and first status information of the vehicle; determining first delay optimization configuration information based on the first environmental information, the first status information and a mapping relationship, the mapping relationship including a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when a processing unit in the vehicle processes data collected by a sensor; and processing the first data collected by the sensor according to the first delay optimization parameter.

[0032] Based on the above technical solution, the vehicle can determine the appropriate delay optimization configuration information for the current scenario based on the mapping relationship saved in the vehicle, which helps to reduce the delay of the vehicle's processing unit when processing data collected by sensors, thereby helping to reduce end-to-end delay, thereby helping to ensure user driving safety.

[0033] In combination with the second aspect, in a possible implementation of the second aspect, the method also includes: when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold, obtaining second environmental information around the vehicle and second state information of the vehicle; determining second delay optimization configuration information based on the second environmental information, the second state information and the mapping relationship, the mapping relationship including the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; processing the second data collected by the sensor according to the second delay optimization parameter.

[0034] Based on the above technical solution, when the delay in the processing of sensor data by the processing unit during vehicle operation is greater than or equal to the delay threshold, the vehicle can re-determine updated delay optimization configuration information based on the second environmental information surrounding the vehicle, the second state information of the vehicle, and the mapping relationship. In this way, the vehicle can adaptively adjust the delay optimization configuration information during operation, achieving automatic online performance optimization, thereby helping to reduce end-to-end latency and ensuring user driving safety.

[0035] In combination with the second aspect, in a possible implementation of the second aspect, the processing unit includes a main processing unit and a backup processing unit, and processes the first data collected by the sensor according to the first delay optimization parameter, including: deploying the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0036] In combination with the second aspect, in a possible implementation of the second aspect, the first delay optimization parameter is deployed before the main processing unit, and the method also includes: within the first time period, determining that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit.

[0037] In combination with the second aspect, in a possible implementation of the second aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0038] In a third aspect, the present application provides a data processing method, which includes: obtaining first environmental information around a vehicle and first status information of the vehicle; sending first delay optimization configuration information to the vehicle based on the first environmental information, the first status information and the mapping relationship, the mapping relationship including the mapping relationship between environmental information, status information and delay optimization configuration information, and the first delay optimization configuration information including the first delay optimization parameter when the processing unit in the vehicle processes the data collected by the sensor.

[0039] Based on the above technical solution, the cloud server can send corresponding latency optimization configuration information to the vehicle based on the mapping relationship between the vehicle's surrounding environment information and the vehicle's status information. In this way, the vehicle can obtain the appropriate latency optimization configuration information for the current scenario from the cloud server, helping to reduce the latency of the vehicle's processing unit when processing data collected by sensors, thereby helping to reduce end-to-end latency and ensuring user driving safety.

[0040] In combination with the third aspect, in a possible implementation of the third aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0041] In combination with the third aspect, in a possible implementation of the third aspect, the method further includes: based on the first environment information and the first state information, using a hierarchical navigable small world graphs (HNSW) algorithm to retrieve the mapping relationship to obtain the first delay optimization configuration information.

[0042] Based on the above technical solution, the cloud server can retrieve the mapping relationship based on the HNSW algorithm to obtain the first delay optimization configuration information, which helps to reduce the delay when the cloud server searches for the corresponding delay optimization configuration information through the information sent by the vehicle.

[0043] In combination with the third aspect, in a possible implementation of the third aspect, before obtaining the first environmental information around the vehicle and the first state information of the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and the third state information of the autonomous driving vehicle; determining the first delay when processing data collected by the sensor based on the third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; updating the third delay optimization parameter based on the first delay to obtain a fourth delay optimization parameter; and storing the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0044] Based on this technical solution, we set initial latency optimization parameters through an offline simulation system, then observed and optimized the end-to-end latency during actual operation on the offline simulation system. This resulted in a more accurate end-to-end latency calculation method, eliminating the need for additional simulation verification.

[0045] In combination with the third aspect, in a possible implementation of the third aspect, before obtaining the first environmental information around the vehicle and the first state information of the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; extracting delay model data based on the third environmental information and the third state information data, the delay model data including one or more of the number of application threads, thread periodic data, time probability distribution of thread running, and dependency relationship of thread data; inputting the delay model data into a delay simulator to obtain a second delay; determining a fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; and storing the correspondence between the third environmental information, the third state information, and the fourth delay optimization parameter in the mapping relationship.

[0046] Based on the above technical solution, simulating end-to-end delay through a delay simulator can reduce the overhead of simulation delay estimation, and can also perform large-scale parallel optimization with fast convergence speed.

[0047] In some possible implementations, before receiving the first environmental information and the first state information sent by the vehicle, the method also includes: obtaining third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; inputting the third environmental information and the third state information into a prediction model to obtain a fourth delay optimization parameter, the prediction model is trained by a training data set, and the training data set includes sample environmental information, sample state information and the sample delay optimization parameter; the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter is stored in the mapping relationship.

[0048] In some possible implementations, obtaining the third environmental information around the autonomous driving vehicle and the third state information of the autonomous driving vehicle includes: receiving data collected by sensors of the autonomous driving vehicle and the third state information; indexing the data collected by the sensors and the third state information according to time and location to obtain the third environmental information and the third state information.

[0049] In some possible implementations, the data collected by the sensor and the third state information are indexed according to time and location to obtain the third environment information and the third state information, including: through a crowdsourcing algorithm or a group intelligence perception algorithm, the data collected by the sensor and the third state information are indexed according to time and location to obtain the third environment information and the third state information.

[0050] Based on the above technical solution, targeted scheduling strategies can be provided for a wide range of autonomous driving scenarios (i.e., a combination of vehicle environment information and vehicle status information). By utilizing crowdsourcing or group intelligence algorithms, there is no need to devote extensive manpower and resources to optimizing latency for different extreme scenarios (or situations). Each version update only requires updating the mapping relationship in the cloud server, eliminating the need for users to make modifications.

[0051] In a fourth aspect, the present application provides a data processing device, which includes: an acquisition unit for acquiring first environmental information around a vehicle and first status information of the vehicle; a sending unit for sending the first environmental information and the first status information to a cloud server; a receiving unit for receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when the processing unit in the vehicle processes data collected by a sensor; the processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

[0052] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the acquisition unit is used to obtain the second environmental information around the vehicle and the second status information of the vehicle when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the sending unit is used to send the second environmental information and the second status information to the cloud server; the receiving unit is used to receive the second delay optimization configuration information sent by the cloud server, and the cloud server stores the mapping relationship between the second environmental information, the second status information and the second delay optimization configuration information, and the second delay optimization configuration information includes the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit is used to process the second data collected by the sensor according to the second delay optimization parameter.

[0053] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to deploy the first delay optimization parameter on the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed on the main processing unit.

[0054] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the device also includes: a determination unit, used to determine that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit within the first time period.

[0055] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at the multiple moments in the future.

[0056] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the first environmental information includes at least one of time, geographic location, climate and road conditions; and / or, the first status information includes at least one of the vehicle's hardware specifications, system status, and automatic driving status.

[0057] In a fifth aspect, the present application provides a data processing device, which includes: an acquisition unit for acquiring first environmental information around a vehicle and first status information of the vehicle; a determination unit for determining first delay optimization configuration information based on the first environmental information, the first status information and a mapping relationship, the mapping relationship including a mapping relationship between the first environmental information, the first status information and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when the processing unit in the vehicle processes data collected by a sensor; the processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

[0058] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the acquisition unit is further used to obtain second environmental information around the vehicle and second state information of the vehicle when the delay of the processing unit processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the determination unit is further used to determine second delay optimization configuration information based on the second environmental information, the second state information and the mapping relationship, the mapping relationship including the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit is also used to process the second data collected by the sensor according to the second delay optimization parameter.

[0059] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to deploy the first delay optimization parameter on the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed on the main processing unit.

[0060] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the determination unit is further used to determine, within the first time period, that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0061] In combination with the fifth aspect, in a possible implementation of the fifth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0062] In a sixth aspect, the present application provides a data processing device, which includes: an acquisition unit for acquiring first environmental information around a vehicle and first status information of the vehicle; a determination unit for determining first delay optimization configuration information based on the first environmental information, the first status information and a mapping relationship, wherein the mapping relationship includes a mapping relationship between environmental information, status information and delay optimization configuration information, and the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; a sending unit for sending the first delay optimization configuration information to the vehicle.

[0063] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at the multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at the multiple moments in the future.

[0064] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes: an information retrieval unit, used to retrieve the mapping relationship based on the first environment information and the first state information using a hierarchical navigable small-world HNSW algorithm to obtain the first delay optimization configuration information.

[0065] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes a parameter updating unit and a storage unit. The acquisition unit is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and first state information sent by the vehicle; the determination unit is further used to determine the first delay when processing data collected by the sensor based on the third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; the parameter updating unit is used to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0066] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the device also includes a delay simulation unit, and the acquisition unit is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and first state information sent by the vehicle; the determination unit is further used to extract delay model data based on the third environmental information and the third state information data, and the delay model data includes one or more of the number of application threads, thread periodic data, time probability distribution of thread running, and dependency relationship of thread data; the delay simulation unit is used to input the delay model data into a delay simulator to obtain a second delay; the determination unit is further used to determine a fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0067] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the acquisition unit is used to: receive data collected from the sensor of the autonomous driving vehicle and the third state information; index the data collected by the sensor and the third state information according to time and location to obtain the third environment information and the third state information.

[0068] In combination with the sixth aspect, in a possible implementation of the sixth aspect, the acquisition unit is used to: index the data collected by the sensor and the third state information according to time and location through a crowdsourcing algorithm or a group perception algorithm to obtain the third environmental information and the third state information.

[0069] In the seventh aspect, the present application provides a data processing device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device performs any possible data processing method in the first aspect or the second aspect.

[0070] In an eighth aspect, the present application provides a data processing device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device executes any possible data processing method in the third aspect.

[0071] In the ninth aspect, the present application provides a data processing system, which includes a sensor and a computing platform, and the computing platform includes any possible data processing device in the fourth aspect, or includes any possible data processing device in the fifth aspect, or includes any possible data processing device in the seventh aspect.

[0072] In the tenth aspect, the present application provides a vehicle, which includes any possible data processing device in the fourth aspect, or includes any possible data processing device in the fifth aspect, or includes the data processing device described in the seventh aspect, or includes the data processing system described in the ninth aspect.

[0073] In the eleventh aspect, the present application provides a server, which includes any possible data processing device in the sixth aspect, or includes the data processing device described in the eighth aspect.

[0074] In a twelfth aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute any possible data processing method in the first or second aspect.

[0075] In a thirteenth aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed on a computer, enables the computer to execute any one of the possible data processing methods in the third aspect.

[0076] It should be noted that the above-mentioned computer program code can be stored in whole or in part on the first storage medium, wherein the first storage medium can be packaged together with the processor or separately packaged with the processor, and the embodiments of the present application do not specifically limit this.

[0077] In a fourteenth aspect, the present application provides a computer-readable medium storing a program code. When the computer program code runs on a computer, the computer executes any possible data processing method in the first or second aspect.

[0078] In a fifteenth aspect, the present application provides a computer-readable medium storing a program code. When the computer program code runs on a computer, the computer executes any possible data processing method in the third aspect.

[0079] In a sixteenth aspect, the present application provides a chip comprising a circuit for executing any possible data processing method in the first or second aspect above.

[0080] In the seventeenth aspect, the present application provides a chip, which includes a circuit for executing any possible data processing method in the third aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] FIG1 is a functional block diagram of a vehicle provided in an embodiment of the present application.

[0082] Figure 2 is a schematic diagram of the relationship between application nodes in the robot operating system ROS.

[0083] FIG3 is a schematic flow chart of a data processing method provided in an embodiment of the present application.

[0084] FIG4 is a schematic diagram of the system architecture provided in an embodiment of the present application.

[0085] FIG5 is another schematic diagram of the system architecture provided in an embodiment of the present application.

[0086] FIG6 is another schematic flow chart of the data processing method provided in an embodiment of the present application.

[0087] FIG7 is an example diagram of a data structure of a fast retrieval layer of a delay feature vector provided in an embodiment of the present application.

[0088] FIG8 is another schematic flowchart of the data processing method provided in an embodiment of the present application.

[0089] FIG9 is a schematic block diagram of a data processing device provided in an embodiment of the present application.

[0090] FIG10 is another schematic block diagram of a data processing device provided in an embodiment of the present application.

[0091] FIG11 is another schematic block diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. For example, "at least one of A and B" is similar to "A and / or B", describing the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0093] In the embodiments of the present application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present application does not constitute a restriction on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary restriction. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0094] FIG1 is a functional block diagram of a vehicle 100 provided in an embodiment of the present application. The vehicle 100 may include a perception system 120 and a computing platform 150, wherein the perception system 120 may include one or more sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a positioning system, which may be a global positioning system (GPS), a BeiDou system, or other positioning systems. The perception system 120 may also include one or more of an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0095] Some or all functions of vehicle 100 may be controlled by computing platform 150. Computing platform 150 may include one or more processors, such as processors 151 to 15n (n is a positive integer). A processor is a circuit capable of processing signals. In one implementation, the processor may be a circuit capable of reading and executing instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationships of hardware circuits. The logical relationships of the hardware circuits may be fixed or reconfigurable. For example, the processor may be a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration file to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, the processor may also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform 150 may also include a memory for storing instructions, and some or all of the processors 151 to 15n may call the instructions in the memory to implement corresponding functions.

[0096] The vehicles involved in the embodiments of the present application are vehicles in a broad sense, which can be transportation vehicles (such as commercial vehicles, passenger cars, motorcycles, flying cars, trains, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as lawn mowers, harvesters, etc.), amusement equipment, toy vehicles, etc. The embodiments of the present application do not specifically limit the type of vehicles.

[0097] As mentioned earlier, a crucial safety metric is the end-to-end latency of the autonomous driving system's data flow chain. For example, the latency from when sensors like lidar and cameras collect raw signals to when they issue braking commands to the chassis domain controller. The runtime of each node in the data flow chain significantly impacts safety. Low end-to-end latency (e.g., less than 100ms) significantly improves vehicle and pedestrian safety and is a key goal for subsequent system and application optimization.

[0098] Currently, autonomous driving system engineers commonly use end-to-end latency optimization methods to adjust and optimize the parameters of autonomous driving systems and applications within a limited number of scenarios. Parameter adjustment and optimization methods include manual tuning based on expert experience, direct on-vehicle tuning using online closed-loop feedback, and tuning using abstract models, simulators, and emulators combined with AI tuning algorithms. However, these methods are primarily designed for common operating conditions and universal environments, making it difficult to dynamically adjust for different scenarios during vehicle operation and lacking generalizability. Furthermore, the adjustment process is typically very time-consuming, and even with the computing resources of cloud servers, the entire process can take several hours.

[0099] Embodiments of the present application provide a data processing method, apparatus, and vehicle. A vehicle can transmit environmental information surrounding the vehicle and its current status to a cloud server. The cloud server can then, based on a mapping relationship between the environmental information, status information, and latency optimization configuration information, transmit latency optimization configuration information corresponding to the environmental and status information to the vehicle. The vehicle can process sensor data using this latency optimization configuration information, helping to reduce end-to-end latency and thereby ensuring the safety of both the vehicle and pedestrians.

[0100] Before introducing the technical solutions provided by the embodiments of the present application, several technical terms related to the embodiments of the present application are first introduced.

[0101] Message passing: A general term for data communication methods between processes or software components in a computer system. It abstracts and encapsulates the data to be communicated as "messages." The communicating parties use primitives such as message sending and receiving to pass messages between processes or components, thus completing data communication.

[0102] Autonomous driving intelligent operating system: An operating system based on the Portable Operating System Interface (POSIX) standard, suitable for the high-performance computing and high-bandwidth communications required for autonomous driving, providing functions such as vehicle environment perception, intelligent decision-making, and path planning.

[0103] Application node: An application within an autonomous driving intelligent operating system that implements specific autonomous driving functions (such as perception, fusion, and planning) encapsulated using communication middleware (e.g., the Robot Operating System (ROS) or the Communication Management (CM) component of the Automotive Open System Architecture (AUTOSAR)). Application nodes are typically single-process, multi-threaded applications. Two or more nodes invoke the communication middleware to transfer messages and service calls between processes or components.

[0104] Message channel: In an autonomous driving intelligent operating system, a designated communication channel between application nodes, consisting of the sender's sending port, the receiver's receiving port, the channel name, the message format, etc.

[0105] The latency of many current autonomous driving applications fluctuates due to external environmental factors or scenarios. This can lead to variations in end-to-end latency, which in turn impacts the actual deployment of end-to-end latency optimization configurations calculated offline. Many autonomous driving algorithms are scenario- and data-driven. When autonomous vehicles operate in different environments, they execute different code or model logic. For example, the logic actually executed by the application can differ significantly depending on the scenario and external environment, such as highways, urban areas, and underground parking. Even with changes in road traffic volume or weather conditions, the actual logic executed by the application can vary significantly. Therefore, end-to-end latency is affected by the scenario and external environment, resulting in significant variations in baseline and jitter across different scenarios and environments. This fluctuation in external scenarios and environments can significantly impact the effectiveness of a single latency optimization configuration configured offline, significantly reducing its effectiveness in some scenarios. Extreme scenarios and environments can even have negative effects, significantly impacting the safety of autonomous driving systems and applications.

[0106] Figure 2 shows a schematic diagram of the relationship between application nodes in ROS. ROS includes nodes 1, 2, 3, and 4. Node 1 can be used to acquire images collected by sensors, node 2 can be used to segment the images collected by sensors, node 3 can be used to identify the type of obstacle, the distance to the vehicle, the speed of the obstacle, etc., based on the segmented image results, and node 4 can be used to determine vehicle control commands based on the recognition results of node 3 and send the vehicle control commands to the chassis domain controller.

[0107] The latency of the aforementioned autonomous driving application can be understood as the latency of a single node processing data, for example, the latency of node 1 processing data.

[0108] The above-mentioned end-to-end delay of autonomous driving can be understood as the delay of the node set in the ROS communication system processing data. For example, the delay from node 1 obtaining the image collected by the sensor to node 4 sending the vehicle control command to the chassis domain controller.

[0109] Figure 3 shows a schematic flow chart of a data processing method 300 provided in an embodiment of the present application. As shown in Figure 3, the method 300 can be executed by a vehicle and a cloud server, and the method 300 includes steps S310 to S340.

[0110] S310: The vehicle obtains first environmental information around the vehicle and first status information of the vehicle.

[0111] Optionally, the vehicle obtains first environmental information around the vehicle, including: the vehicle can determine the first environmental information based on data collected by a sensor (for example, a camera, a millimeter-wave radar, a lidar, etc.).

[0112] Exemplarily, Table 1 shows the environmental information and status information acquired by the vehicle.

[0113] Table 1

[0114] The above autonomous driving modes can be understood as the autonomous driving modes currently enabled in the vehicle, such as navigation cruise assistant (NCA), integrated cruise assistant (ICA), and auto valet parking (AVP). Auxiliary functions may include autonomous emergency braking (AEB) and lane departure warning (LDW).

[0115] The environmental information and status information shown in Table 1 above are merely illustrative, and the vehicle may obtain more or less information than that shown in Table 1.

[0116] Optionally, the environmental information may include data collected by the vehicle's sensors (e.g., one or more of a camera, lidar, millimeter-wave radar, GPS, and IMU). The vehicle may send the data collected by the sensors to a cloud server, which then determines the vehicle's environment (e.g., geographic location, time, road conditions, etc.).

[0117] S320: The vehicle sends the first environment information and the first status information to the cloud server.

[0118] Accordingly, the cloud server can receive the first environment information and the first status information sent by the vehicle.

[0119] Optionally, the vehicle can determine the first environmental information based on the data collected by the sensor, and then send the first environmental information and the first status information to the cloud server; or, the vehicle can also send the data collected by the sensor and the first status information to the cloud server, and the cloud server determines the first environmental information based on the data collected by the sensor.

[0120] Optionally, the vehicle may periodically send information about the environment surrounding the vehicle and information about the vehicle's status to the cloud server.

[0121] Optionally, before the cloud server receives the first environmental information and first status information sent by the vehicle, the method 300 also includes: the cloud server receives third status information from the autonomous driving vehicle and data collected by sensors in the autonomous driving vehicle; the cloud server indexes the third status information and the data collected by the sensors according to time and location to obtain the third environmental information and the third status information.

[0122] Optionally, the cloud server indexes the third state information and the data collected by the sensor according to time and location to obtain the third environmental information and the third state information, including: the cloud server indexes the third state information and the data collected by the sensor according to time and location through a crowdsourcing algorithm or a group intelligence perception algorithm to obtain the third environmental information and the third state information.

[0123] S330, the cloud server sends first delay optimization configuration information to the vehicle based on the first environmental information, the first status information and the mapping relationship. The mapping relationship includes a mapping relationship between environmental information, status information and delay optimization configuration information. The first delay optimization configuration information includes a first delay optimization parameter when the vehicle's processing unit processes data collected by the sensor.

[0124] Optionally, the mapping relationship may also be referred to as a latency optimization configuration information retrieval library. The cloud server may store a latency optimization configuration information retrieval library, which includes a mapping relationship between environment information, state information, and latency optimization configuration information.

[0125] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0126] Exemplarily, the vehicle can send navigation information (for example, navigation information from the vehicle's current location to the destination) to a cloud server, and the cloud server can determine the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future based on the navigation information. The cloud server can determine the delay optimization configuration information at multiple moments in the future based on the environmental information around the vehicle at multiple moments in the future, the status information of the vehicle at multiple moments in the future, and the mapping relationship, and send it to the vehicle. In this way, the vehicle can save the delay optimization configuration information at multiple moments in the future in advance. When the environmental information around the vehicle changes, and / or the status information of the vehicle changes, the data collected by the sensor can be processed by selecting appropriate delay optimization configuration information from the delay optimization configuration information at multiple moments in the future. In this way, it helps to avoid frequent interactions between the vehicle and the cloud server.

[0127] For example, the vehicle can also determine the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future based on the navigation information, and then send the environmental information around the vehicle at multiple moments in the future and the status information of the vehicle at multiple moments in the future to the cloud server.

[0128] The first delay optimization configuration information may include delay optimization parameters when at least some of the nodes among node 2, node 3 and node 4 shown in FIG2 process data collected by sensors.

[0129] Exemplarily, the latency optimization parameters include one or more of priority information of tasks in the processing unit, core binding information, and scheduling parameters of a scheduler in an operating system (eg, a Linux system).

[0130] Optionally, the method 300 also includes: when the vehicle determines that the delay for processing the data collected by the sensor is greater than or equal to the delay threshold based on the first delay optimization parameter, obtaining second environmental information around the vehicle and second status information of the vehicle; the vehicle sends the second environmental information and the second status information to the cloud server; the cloud server sends second delay optimization configuration information to the vehicle based on the second environmental information, the second status information and the mapping relationship, the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the vehicle processes the second data collected by the sensor based on the second delay optimization parameter.

[0131] For example, at time T1, the vehicle is on a highway with a speed limit of 120 kilometers per hour (km / h) and a CPU usage of 50%. The vehicle can send this information to the cloud server, which then determines latency optimization configuration information 1 based on the mapping relationship and sends it to the vehicle. The vehicle can then process the data collected by the sensors based on this latency configuration information 1.

[0132] At time T2 (for example, a time after time T1), the road type on which the vehicle is located is an urban road, the road speed limit is 80km / h and the vehicle's CPU occupancy rate is 80%. When the vehicle processes the data collected by the sensor according to the delay optimization configuration information 1 for a delay greater than or equal to the delay threshold (for example, 100ms), the vehicle can send the corresponding environmental information and status information at time T2 to the cloud server, so that the cloud server determines the delay optimization configuration information 2 based on the environmental information and status information at time T2 and the mapping relationship and sends it to the vehicle. The vehicle can process the data collected by the sensor according to the delay configuration information 2. When the delay of the processing unit processing the data collected by the sensor is greater than or equal to the delay threshold due to changes in the environmental information and / or status information, the vehicle can request new delay optimization configuration information from the cloud server, so that the vehicle can process the data collected by the sensor through the new delay optimization configuration information, which helps to reduce the end-to-end delay in a timely manner and also helps to improve the safety of the vehicle and the driving safety of the user.

[0133] Optionally, before the cloud server sends the first delay optimization configuration information to the vehicle, the method 300 further includes: searching the mapping relationship using the HNSW algorithm according to the first environment information and the first status information to obtain the first delay optimization configuration information.

[0134] Optionally, before the cloud server receives the first environmental information and first status information sent by the vehicle, the method 300 also includes: the cloud server obtains third environmental information around the autonomous driving vehicle and third status information of the autonomous driving vehicle; determines the first delay when processing data collected by the sensor based on the third delay optimization parameter set by the offline simulation system, the third environmental information and the third status information; updates the third delay optimization parameter based on the first delay to obtain a fourth delay optimization parameter; the cloud server saves the correspondence between the third environmental information, the third status information and the fourth delay optimization parameter in the mapping relationship.

[0135] The above-mentioned autonomous driving vehicle can be understood as a road test vehicle, and the third environment information and the third state information can be information collected by the road test vehicle.

[0136] The third delay optimization parameter may be a preset delay optimization parameter, or may be a delay optimization parameter optimized last time. For example, the preset delay optimization parameter indicates that the priorities of multiple tasks in the processing unit are the same.

[0137] Exemplarily, the cloud server may store an offline simulation system, and the cloud server may set the priority of the autonomous driving application process and the parameters corresponding to the core binding (i.e., the third delay optimization parameter) on the offline simulation system, and input the third environment information and the third state information into the offline simulation system. The cloud server may monitor the end-to-end delay during actual operation on the offline simulation system. The cloud server may update the third delay optimization parameter based on the end-to-end delay during actual operation, thereby obtaining a fourth delay optimization parameter. The cloud server may save the correspondence between the third environment information, the third state information, and the fourth delay optimization parameter in the mapping relationship.

[0138] When the cloud server updates the third latency optimization parameter based on the actual end-to-end latency during runtime, it can iteratively derive the fourth latency optimization parameter by combining heuristic AI tuning algorithms, reinforcement learning algorithms, and deep learning algorithms. This makes the end-to-end latency calculation method more accurate and eliminates the need for additional simulation verification before deployment on the cloud server.

[0139] Optionally, before the cloud server receives the first environmental information and the first status information sent by the vehicle, the method 300 also includes: the cloud server obtains the third environmental information around the autonomous driving vehicle and the third status information of the autonomous driving vehicle; the cloud server extracts delay model data based on the third environmental information and the third status information data, and the delay model data includes one or more of the number of application threads, the periodic data of the threads, the time probability distribution of the thread running and the dependency relationship of the thread data; the cloud server inputs the delay model data into the delay simulator to obtain the second delay; the cloud server determines the fourth delay optimization parameter corresponding to the third environmental information and the third status information based on the second delay; the cloud server saves the correspondence between the third environmental information, the third status information and the fourth delay optimization parameter in the mapping relationship.

[0140] Exemplarily, the cloud server can extract the delay model data of the autonomous driving system and application under the third environmental information and the third state information sent by the autonomous driving vehicle. The delay model data includes but is not limited to one or more of the number of application threads, the periodic data of the threads, the probability distribution of the thread running time, and the dependency relationship of the thread data. The end-to-end delay is then directly calculated using a delay simulator, and the fourth delay optimization parameter is obtained by repeated iteration using a heuristic AI tuning algorithm, reinforcement learning, or deep learning tuning algorithm. The advantage of this method is that the overhead of simulation delay estimation is low, large-scale parallel optimization can be performed, and the convergence speed is fast.

[0141] Optionally, before the cloud server receives the first environmental information and first status information sent by the vehicle, the method 300 also includes: the cloud server obtains the third environmental information around the autonomous driving vehicle and the third status information of the autonomous driving vehicle; the cloud server inputs the third environmental information and the third status information data into the prediction model to obtain a fourth delay optimization parameter; the cloud server saves the third environmental information, the third status information and the fourth delay optimization parameter in the mapping relationship.

[0142] Exemplarily, the prediction model can be trained using a training data set, which includes the vehicle's environmental information, the vehicle's state information, and the delay optimization configuration information corresponding to the environmental information and state information (task priority information, core binding information, scheduling parameters). The trained prediction model can be saved in a cloud server. In this way, after the autonomous driving vehicle (or, road test vehicle) uploads the road test data (including the environmental information around the road test vehicle and the state information of the road test vehicle) to the cloud server, the cloud server can input the road test data into the prediction model to obtain the corresponding delay optimization configuration information. The cloud server can save the corresponding relationship between the environmental information, state information, and delay optimization configuration information in the mapping relationship.

[0143] Optionally, the cloud server obtains third environmental information around the autonomous driving vehicle and third status information of the autonomous driving vehicle, including: the cloud server receives the third status information from the autonomous driving vehicle and data collected by sensors in the autonomous driving vehicle; the cloud server indexes the third status information and the data collected by the sensors according to time and location to obtain the third environmental information and the third status information.

[0144] Optionally, the cloud server indexes the third state information and the data collected by the sensor according to time and location to obtain the third environmental information and the third state information, including: indexing the third state information and the data collected by the sensor according to time and location through a crowdsourcing algorithm or a group intelligence perception algorithm to obtain the third environmental information and the third state information.

[0145] S340: The vehicle processes the first data collected by the sensor according to the first time delay optimization parameter.

[0146] Optionally, the vehicle includes a main processing unit and a backup processing unit, and the vehicle processes the first data collected by the sensor according to the first delay optimization parameter, including: the vehicle deploys the first delay optimization parameter on the backup processing unit; within a first time period, the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit in processing the first data, and the vehicle deploys the first delay optimization parameter on the main processing unit.

[0147] Exemplarily, the main processing unit may be a main domain, and the backup processing unit may be a secure backup domain. Alternatively, the main processing unit may be a main processor, and the backup processing unit may be a backup processor.

[0148] The delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit in processing the first data. It can be understood that within the preset time length, the average value of the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the average value of the delay of the main processing unit in processing the first data; and / or, the maximum value of the delay of the backup processing unit in processing the first data according to the first delay optimization parameter is less than the maximum value of the delay of the main processing unit in processing the first data.

[0149] Optionally, the vehicle deploys the first delay optimization parameter before the main processing unit. The method 300 further includes: within the first time period, determining that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0150] Figure 4 shows a schematic diagram of the system architecture provided by an embodiment of the present application. The system architecture includes a vehicle 410 and a cloud server 420. Vehicle 410 may include a software platform and a hardware platform. The software platform includes an autonomous driving application 411 and an operating system and middleware 412. The operating system and middleware 412 include a scene-aware latency monitoring module 4121 and a latency optimization request and configuration module 4122. The hardware platform includes a CPU 413, a GPU 414, a memory 415, a network card 416, a sensor 417, and a controller 418.

[0151] The cloud server 420 may include an offline scenario simulation database 421, an offline scenario simulation system 422, and a delay feature vector retrieval library 423. The delay feature vector retrieval library may store the above mapping relationship.

[0152] The scene-aware delay monitoring module 4121 can be used to perceive the environmental information around the vehicle 410 and the end-to-end delay changes in the current scene, and at the same time upload the environmental information and the status information of the vehicle 410 to the cloud server 420.

[0153] The above scenario-aware delay monitoring module 4121 can upload environmental information to the cloud server 420, or can also upload data collected by the sensor 417 to the cloud server 420. The cloud server 420 can determine the environmental information around the vehicle 410 based on the data collected by the sensor 417.

[0154] The delay optimization request and configuration module 4122 can be used to send a request to the cloud server 420 to optimize the end-to-end delay in the current scenario, and receive the delay optimization configuration information sent by the cloud server 420 and configure the delay optimization configuration information on the vehicle 410.

[0155] The offline scenario simulation database 421 can be used to record environmental data and vehicle status information of different vehicles in different scenarios such as different roads and weather conditions, including but not limited to point cloud data collected by lidar, image data collected by cameras, point cloud data collected by millimeter-wave radar, data collected by GPS and inertial sensors, etc., as well as operation logs of various hardware and software in the vehicle, which can be subsequently used for autonomous driving scenario playback, simulation, etc.

[0156] The offline scenario simulation system 422 can be used to read the environmental information and status information in the offline scenario simulation database 421, and provide autonomous driving scenario-level simulation and simulation functions for different times, climates, geographical locations and road conditions.

[0157] The delay feature vector retrieval library 423 can be used to record environmental information, status information in different scenarios, and delay optimization configuration information calculated by the offline scenario simulation system 422. Environmental information includes but is not limited to time, geographical location, climate conditions, road conditions and structure, etc. Status information includes but is not limited to hardware specifications, system status, automatic driving status, etc.

[0158] The CPU 413 or the GPU 414 can be used to provide the necessary computing resources for the autonomous driving system and applications.

[0159] The network card 416 can be used to provide network communication resources for the autonomous driving system and applications.

[0160] Memory 415 can be used to provide storage resources for autonomous driving systems and applications.

[0161] Sensor 417 can be used to provide perception data resources for autonomous driving systems and applications.

[0162] The controller 418 can be used to provide chassis control capabilities for the autonomous driving vehicle.

[0163] The autonomous driving application 411 obtains and controls the CPU resources, GPU resources, network card resources, memory resources, and sensor-collected data necessary for operation through the operating system and middleware 412 , and controls the operation of the vehicle 410 through the controller 418 .

[0164] The scenario-aware latency monitoring module 4121 analyzes the vehicle 410's environment, status, and end-to-end latency information from sensor data and application processing data during the operation of the autonomous driving application 411. This information can be stored in the memory 415 or directly sent to the cloud server 420 via the network card 416. The cloud server 420 can record this information in the offline scenario simulation database 421.

[0165] The delay optimization request and configuration module 4122 can obtain the end-to-end delay from the scene-aware delay monitoring module 4121. When end-to-end delay optimization is required, it will send an optimization request to the cloud server 420 through the network card 416, and receive the delay optimization configuration information sent by the delay feature vector retrieval library 423, and then deploy it on the current autonomous driving operating system and application.

[0166] Figure 5 shows another schematic diagram of the system architecture provided by an embodiment of the present application. In this system architecture, the scenario-aware latency monitoring module 4121 may include a collection and monitoring module 41211 and a dynamic configuration module 41212. The latency optimization request and configuration module 4122 may include a network communication module 41221 and a configuration and deployment module 41222. The cloud server 420 may also include an automatic scheduling and optimization system 424 and a latency feature vector retrieval module 425.

[0167] The acquisition and monitoring module 41211 can use the scheduling, communication, and other log information from the autonomous driving application's operation process and combine it with algorithms to obtain information about the environment surrounding the vehicle 410 and the vehicle's status. This information is then sent to the cloud server 420 via the network communication module 41221. The cloud server 420 can record this information in the offline scenario simulation database 421.

[0168] Dynamic configuration module 41212 can dynamically obtain the end-to-end latency of vehicle 410 from acquisition and monitoring module 41211. If the actual end-to-end latency is worse than expected, an optimization request can be sent to cloud server 420 via network communication module 41221. This optimization request can include information about the environment surrounding vehicle 410 and the current state of vehicle 410. Once the latency optimization configuration information is obtained from cloud server 420, it is sent to configuration deployment module 41222, which deploys it in the autonomous driving system and applications, thereby optimizing end-to-end latency.

[0169] The scheduling and orchestration automatic optimization system 424 can be used to regularly pull data from the offline scenario simulation database 421 through the offline scenario simulation system 422 on the cloud server 420 to simulate different autonomous driving scenarios and observe the end-to-end delay of the data flow of the key link of autonomous driving, and then automatically optimize to find the delay optimization configuration information corresponding to different environmental information and status information, and record the correspondence between the environmental information, status information and delay optimization configuration information in the delay feature vector retrieval library 423.

[0170] FIG6 shows a schematic flow chart of a data processing method 600 provided in an embodiment of the present application. The method 600 may be executed by the cloud server 420 and includes:

[0171] S610 , constructing an offline simulation database 421 based on time and geographic location on the cloud server 420 .

[0172] Exemplarily, step S610 may include two stages:

[0173] Phase 1 is to automatically collect sensor data and status information during the operation of the autonomous driving vehicle (or road test vehicle) and upload it to the cloud server 420.

[0174] In the second stage, the cloud server 420 integrates the data uploaded by each autonomous driving vehicle (or road test vehicle) according to geographic location and time to form an offline scenario simulation database 421.

[0175] Exemplarily, the cloud server 420 integrates the data uploaded by each autonomous driving vehicle according to geographic location and time through crowdsourcing or group intelligence perception technology to form an offline scene simulation database 421.

[0176] The data required to be collected by autonomous vehicles in Phase 1 includes, but is not limited to, LiDAR point cloud data, camera image data, millimeter-wave radar point cloud data, data from sensors such as GPS and IMU, as well as information such as weather, temperature, humidity, light intensity, and visibility. Phase 2 will integrate the data from various sources in Phase 1, using time series and geographic coordinates as primary indexes. Using crowd-sensing or crowdsourcing algorithms, this fragmented data will be constructed into a unified offline scenario simulation database, enabling playback and simulation of various autonomous driving scenarios for future testing and optimization.

[0177] S620, through the offline simulation system 422 and the scheduling and scheduling automatic optimization system 424, obtain the correspondence between the environmental information around the vehicle, the vehicle status information and the delay optimization configuration information at a certain time slice, and save the correspondence in the delay feature vector retrieval library 423.

[0178] Step S620 can simulate and calculate the delay optimization configuration information corresponding to the environmental information and status information under a certain time slice on the cloud server 420 based on the data in the offline scenario simulation database 421 of step S610, and then save the environmental information, status information and corresponding delay optimization configuration information under the time slice in the delay feature vector retrieval library 423.

[0179] Step S620 can be carried out after the offline scenario simulation database 421 is initially constructed, and can also be carried out synchronously when the offline scenario simulation database 421 is subsequently updated.

[0180] For example, the offline scenario simulation system 422 selects environmental and state information at different time slices from the offline scenario simulation database 421 in a sampling or traversal manner, and simulates and replays the end-to-end latency of the autonomous driving system application in different scenarios through the offline scenario simulation system 422. The machine learning algorithm is then used to determine the latency optimization configuration information.

[0181] The above steps S610 and S620 may be completed before the vehicle 410 travels.

[0182] S630 , receiving an optimization request sent by the vehicle 410 and sending delay optimization configuration information to the vehicle 410 , where the optimization request includes environmental information around the vehicle 410 and status information of the vehicle 410 .

[0183] For example, vehicle 410 may periodically upload environmental information surrounding vehicle 410 and status information of vehicle 410 to cloud server 420. Cloud server 420 may retrieve delay optimization configuration information from delay feature vector retrieval library 423 based on the environmental information and status information sent by vehicle 410 and send the information to vehicle 410.

[0184] The above step S630 may occur while the vehicle 410 is traveling.

[0185] Optionally, the latency optimization configuration information may include a latency range in addition to the task priority, core binding information, and scheduling parameters. If the latency of vehicle 410 processing sensor data collected according to the latency optimization configuration information falls outside the latency range, vehicle 410 may resend an optimization request to the cloud server.

[0186] The hardware in the aforementioned hardware platform can be an autonomous driving hardware processing platform, where the AI ​​CPU and AI Core provide CPU and GPU computing resources, respectively; the Ethernet card provides gigabit network communication capabilities; the universal flash storage (UFS) provides large-capacity storage resources; the sensors provide point cloud data, image data, GPS data, and other data; and the controller provides chassis control capabilities. The aforementioned operating system can be Linux.

[0187] The autonomous driving hardware processing platform's acquisition and monitoring module 41211 collects and records a large amount of raw sensor data during the autonomous driving process, including but not limited to LiDAR point cloud coordinate data, camera image and video data, GPS spatiotemporal coordinate data, and IMU time series data. Due to the large amount of data, this data is stored in the Unified File System (UFS) during the autonomous driving process via the CM communication module. At the conclusion of the road test, the information stored in the UFS is directly uploaded to the cloud server via an Ethernet card or an external workstation with a memory reader / writer. Simultaneously, vehicle status information during the autonomous driving process is also uploaded to the cloud server, including but not limited to chassis information, vehicle status, hardware SoC resource usage, network status, system software operating status, and vehicle autonomous driving status.

[0188] After receiving the different drive test data from different vehicles, the cloud server analyzes the time, start and end locations, and route of each drive test data. It then aligns the collected sensor data and status information according to time and GPS coordinates, recording them as a spatiotemporal database of autonomous driving scenario data, known as the offline scenario simulation database. Heuristic algorithms and neural network-based AI algorithms are also used to automatically analyze the weather conditions, solar terms, and road congestion status of each drive test data, creating an index beyond time and GPS coordinates to facilitate rapid retrieval and use by the subsequent offline scenario simulation system.

[0189] Offline scenario simulation system 422 of cloud server 420 can access data from offline scenario simulation database 421 constructed in step S610 to simulate different autonomous driving scenarios at different times, geographic locations, weather conditions, road conditions, and so on. By integrating this data into software and hardware platforms or autonomous driving systems and applications, the accuracy of autonomous driving application operating results and overall system performance can be measured. Optionally, given the limited amount of scene data (including environmental and status information) collected and uploaded from the real world, the offline scenario simulation system utilizes an enhanced AI algorithm to combine scene data from different drive test data into a single simulation scene. Generative AI techniques can also be used to add previously non-existent vehicles, obstacles, pedestrians, and so on to the playback data, or even to modify features of the original scene, such as weather, traffic volume, or road surface properties.

[0190] During the offline scenario simulation process, the offline scenario simulation system extracts environmental information and state information within a time window at a fixed frequency (such as 1 Hz or 0.1 Hz) to form time-series data. The time-series data may be as shown in Table 1 above.

[0191] The offline scenario simulation system 422 constructs a time delay feature vector retrieval library 423 from the environmental and state information extracted within all time slices. This environmental and state information can be independently retrieved by the autonomous vehicle later. Some of this data is scenario-independent, such as hardware specification characteristics. Some data, such as system state characteristics, fluctuates significantly over time and scenario. Once established, the time delay feature vector retrieval library 423 can be regularly maintained. For example, steps S610 and S620 need to be repeated for each iteration of the autonomous driving software or hardware.

[0192] At the same time, the AI-based scheduling and orchestration automatic optimization system 424 performs end-to-end delay optimization for each time slice of the simulation, with environmental information and state information as input and delay optimization configuration information for that time slice as output. The scheduling and orchestration automatic optimization system 424 can update the output to the delay feature vector retrieval library 423, thereby establishing a correspondence between environmental information, state information, and delay optimization configuration information. In the embodiment of the present application, the delay optimization configuration information for the time slice can be determined by the following two methods.

[0193] Method 1

[0194] Directly set the task priority and core binding information for the autonomous driving application on offline simulation system 422, observe the end-to-end latency during actual operation on offline simulation system 422, and iteratively calculate the latency optimization configuration information for that time slice using a heuristic AI tuning algorithm, reinforcement learning, or deep learning tuning algorithm. This method adjusts task priority and core binding. The advantage of this method is that the end-to-end latency calculation method is more accurate and does not require additional simulation verification before deployment.

[0195] The above-mentioned offline simulation system 422 may include a vehicle dynamics simulation system and an automatic driving simulation system.

[0196] Method 2

[0197] First, the state information extracted within the time slice is combined to extract and generate the delay model data of the autonomous driving system and application corresponding to the environmental information under the time slice. The delay model data includes but is not limited to the number of application threads, thread periodicity data, the probability distribution of thread running time, thread data dependencies, etc. The end-to-end delay is directly obtained by inputting the delay model data into the delay simulator. The delay optimization configuration information under the time slice is repeatedly calculated by combining heuristic AI tuning algorithms, reinforcement learning, or deep learning tuning algorithms. The advantage of this second method is that the overhead of simulation delay estimation is low, large-scale parallel optimization can be performed, and the convergence speed is fast.

[0198] Taking into account the very large amount of data in the delay feature vector retrieval library, the embodiment of the present application is based on HNSW technology and combines the idea of ​​a jump table to construct a delay feature vector fast retrieval layer for the original linked list. Because the delay feature vectors generated by the same road test data have strong locality, there are a large number of identical redundant data in the same time window and the same geographical location. Therefore, the delay feature vectors can be sorted according to the degree of change and the redundancy can be merged. For example, the hardware specifications, system versions and status are unchanged in the same road test. Sparse stratification is performed according to the density of the delay feature vector, starting with the sparse layer, and then querying the dense layer step by step if the conditions are met. This can improve the retrieval speed of the delay optimization configuration information and reduce the response delay of the cloud server. Figure 7 shows an example diagram of the delay feature vector fast retrieval layer data structure provided by the embodiment of the present application.

[0199] FIG8 shows a schematic flow chart of a data processing method 800 provided in an embodiment of the present application. The method 800 may be executed by a vehicle and includes:

[0200] S810: The vehicle determines whether the delay of the processing unit in processing the data collected by the sensor within the time window is greater than or equal to the delay threshold.

[0201] When the delay of processing the data collected by the sensor within the time window is greater than or equal to the delay threshold, execute S820; otherwise, return to execute S810.

[0202] Exemplarily, the above step S810 can be implemented by the following two methods:

[0203] Method 1

[0204] By using AI-based algorithms, the sensor-perceived data and vehicle status information are dynamically inferred, and pre-trained models are used to infer changes from external environment information, vehicle status information to the scene in which the vehicle is located.

[0205] Method 2

[0206] By monitoring the vehicle's end-to-end latency, if the end-to-end latency mean or jitter exceeds a threshold within a preset timeframe, it indicates a change in the vehicle's external scene and requires redeployment of latency optimization configuration information. Method 2 has the advantages of requiring fewer resources on the autonomous driving software and hardware platform, lowering computational overhead, and providing a fast and accurate response to scenario changes when redeploying latency optimization configuration information.

[0207] S820, the vehicle sends an optimization request to the cloud server, where the optimization request includes the environment information around the vehicle and the status information of the vehicle.

[0208] Exemplarily, dynamic configuration module 41212 transmits environmental information surrounding the vehicle and vehicle status information to a cloud server, waiting for the cloud server to re-search the latency feature vector index library for latency optimization configuration information suitable for the current scenario and transmit it to the vehicle. Using the cloud server reduces search response time and leverages the cloud server's vast computing resources to accurately locate appropriate latency optimization configuration information without consuming local vehicle computing resources. Version updates only require updating the latency feature vector index library in the cloud server, without the vehicle being aware of the update.

[0209] The above description is based on an example in which a vehicle sends an optimization request to a cloud server when the time delay feature vector retrieval library is deployed on the cloud server. The embodiments of the present application are not limited thereto. The time delay feature vector retrieval library can also be deployed in the vehicle.

[0210] For example, a cloud server can deploy a delay feature vector retrieval library on the vehicle's autonomous driving hardware processing platform using lossy model compression technology. When the vehicle is in autonomous driving mode and its end-to-end latency is greater than or equal to a latency threshold, the dynamic configuration module 41212 can re-search the compressed delay feature vector retrieval library for latency optimization configuration information appropriate for the current scenario based on the vehicle's surrounding environment and status information. This allows the vehicle to promptly obtain updated latency optimization configuration information even when offline, enabling better compliance with functional safety certification.

[0211] S830, the vehicle receives delay optimization configuration information determined by the cloud server based on the environmental information and status information, where the delay optimization configuration information includes delay optimization configuration parameters.

[0212] S840: The vehicle deploys the delay optimization configuration parameters on the safety backup domain.

[0213] S850: Within a preset time period, if the delay of processing the data collected by the sensors in the primary domain is greater than the delay of processing the data collected by the sensors in the safety backup domain, the vehicle deploys the delay optimization configuration information on the primary domain.

[0214] The above primary domain and safety backup domain can process data collected by the same sensor.

[0215] Exemplarily, the dynamic configuration module obtains the latest delay optimization configuration information from the delay feature vector retrieval library of the cloud server and sends it to the configuration deployment module 41222. The configuration deployment module 41222 deploys the delay optimization configuration information in the safety backup domain of autonomous driving to observe its end-to-end delay optimization effect. The safety backup domain and the main domain can both be on the autonomous driving hardware processing platform, and both deploy a full range of autonomous driving applications. If the vehicle control commands output by the main domain and the safety backup domain are the same within a preset time period and the delay of the safety backup domain processing the data collected by the sensors is better than the delay of the main domain processing the data collected by the sensors, the configuration deployment module 41222 can deploy the delay optimization configuration information in the main domain.

[0216] Figure 9 shows a schematic block diagram of a data processing device 900 provided in an embodiment of the present application. As shown in Figure 9, the device 900 includes: an acquisition unit 910 for acquiring first environmental information surrounding a vehicle and first status information of the vehicle; a sending unit 920 for sending the first environmental information and the first status information to a cloud server; a receiving unit 930 for receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environmental information, the first status information, and the first delay optimization configuration information, the first delay optimization configuration information including first delay optimization parameters for a processing unit in the vehicle to process data collected by a sensor; and a processing unit 940 for processing the first data collected by the sensor according to the first delay optimization parameters.

[0217] Optionally, the acquisition unit 910 is used to obtain second environmental information around the vehicle and second status information of the vehicle when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the sending unit 920 is used to send the second environmental information and the second status information to the cloud server; the receiving unit 930 is used to receive second delay optimization configuration information sent by the cloud server, and the cloud server stores a mapping relationship between the second environmental information, the second status information and the second delay optimization configuration information, and the second delay optimization configuration information includes the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit 940 is used to process the second data collected by the sensor according to the second delay optimization parameter.

[0218] Optionally, the processing unit 940 includes a main processing unit and a backup processing unit, and the processing unit 940 is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay when the backup processing unit processes the first data according to the first delay optimization parameter is less than the delay when the main processing unit processes the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0219] Optionally, the device 900 further includes: a determining unit, configured to determine, within the first time period, whether the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0220] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when the processing unit processes data collected by the sensor at multiple moments in the future.

[0221] Optionally, the first environmental information includes at least one of time, geographic location, climate and road conditions; and / or, the first status information includes at least one of the vehicle's hardware specifications, system status, and automatic driving status.

[0222] For example, the acquisition unit 910 may be the computing platform in Figure 1 or a processing circuit, processor, or controller in the computing platform. For example, if the acquisition unit 910 is the processor 121 in the computing platform, the processor 121 may acquire environmental information around the vehicle and vehicle status information.

[0223] For another example, the sending unit 920 and the receiving unit 930 may be communication devices in the vehicle 100 .

[0224] For another example, processing unit 940 may be the computing platform in Figure 1 or a processing circuit, processor, or controller in the computing platform. For example, if processing unit 940 is processor 122 in the computing platform, processor 122 may process the data collected by the sensor based on the latency optimization configuration information sent by the cloud server.

[0225] The functions implemented by the above-mentioned acquisition unit 910 and the functions implemented by the processing unit 940 can be implemented by different processors, or can also be implemented by the same processor, which is not limited in the embodiment of the present application.

[0226] Figure 10 shows a schematic flow chart of a data processing device 1000 provided in an embodiment of the present application. As shown in Figure 10, the device 1000 includes: an acquisition unit 1010 for acquiring first environmental information surrounding a vehicle and first state information of the vehicle; a determination unit 1020 for determining first latency optimization configuration information based on the first environmental information, the first state information, and a mapping relationship, wherein the mapping relationship includes a mapping relationship between the first environmental information, the first state information, and the first latency optimization configuration information, and the first latency optimization configuration information includes a first latency optimization parameter for a processing unit in the vehicle to process data collected by a sensor; and a processing unit 1030 for processing the first data collected by the sensor based on the first latency optimization parameter.

[0227] Optionally, the acquisition unit 1010 is also used to obtain second environmental information around the vehicle and second state information of the vehicle when the delay when the processing unit processes the data collected by the sensor according to the first delay optimization parameter is greater than or equal to the delay threshold; the determination unit 1020 is also used to determine second delay optimization configuration information based on the second environmental information, the second state information and the mapping relationship, the mapping relationship including the mapping relationship between the second environmental information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information including the second delay optimization parameter when the processing unit processes the data collected by the sensor; the processing unit 1030 is also used to process the second data collected by the sensor according to the second delay optimization parameter.

[0228] Optionally, the processing unit 1030 includes a main processing unit and a backup processing unit, and the processing unit 1030 is used to deploy the first delay optimization parameter in the backup processing unit; within a first time period, the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, and the first delay optimization parameter is deployed in the main processing unit.

[0229] Optionally, the determining unit 1020 is further configured to determine, within the first time period, whether the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

[0230] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0231] For example, the acquisition unit 1010 may be the computing platform in Figure 1 or a processing circuit, processor, or controller in the computing platform. For example, if the acquisition unit 1010 is the processor 121 in the computing platform, the processor 121 may acquire environmental information around the vehicle and status information of the vehicle.

[0232] For another example, determination unit 1020 may be the computing platform in FIG1 , or a processing circuit, processor, or controller in the computing platform. For example, if determination unit 1020 is processor 122 in the computing platform, processor 122 may store a mapping relationship between environment information, state information, and latency optimization configuration information. Processor 122 may determine appropriate latency optimization configuration information for the current scenario based on the acquired environment information, state information, and the mapping relationship.

[0233] For another example, processing unit 1030 may be the computing platform in Figure 1 or a processing circuit, processor, or controller in the computing platform. For example, if control unit 1030 is processor 123 in the computing platform, processor 123 may delay optimizing configuration information and process data collected by sensors.

[0234] The functions implemented by the acquisition unit 1010, the functions implemented by the determination unit 1020, and the functions implemented by the processing unit 1030 may be implemented by different processors, or may be implemented by the same processor, or some functions may be implemented by the same processor. This embodiment of the application does not limit this.

[0235] Figure 11 shows a schematic block diagram of a data processing device 1100 provided in an embodiment of the present application. As shown in Figure 11, device 1100 includes: an acquisition unit 1110 for acquiring first environmental information surrounding a vehicle and first state information of the vehicle; a determination unit 1120 for determining first latency optimization configuration information based on the first environmental information, the first state information, and a mapping relationship, wherein the mapping relationship includes a mapping relationship between environmental information, state information, and latency optimization configuration information, and the first latency optimization configuration information includes first latency optimization parameters for a processing unit in the vehicle when processing data collected by sensors; and a transmission unit 1130 for transmitting the first latency optimization configuration information to the vehicle.

[0236] Optionally, the first environmental information includes environmental information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; wherein, the first delay optimization configuration information includes delay optimization parameters when processing data collected by the sensor at multiple moments in the future.

[0237] Optionally, the apparatus 1100 further includes: an information retrieval unit, configured to retrieve the mapping relationship using a hierarchical navigable small-world (HNSW) algorithm according to the first environment information and the first state information to obtain the first delay optimization configuration information.

[0238] Optionally, the device 1100 also includes a parameter updating unit and a storage unit. The acquisition unit 1110 is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and first state information sent by the vehicle; the determination unit 1120 is further used to determine the first delay when processing data collected by the sensor based on the third delay optimization parameter set by the offline simulation system, the third environmental information and the third state information; the parameter updating unit is used to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0239] Optionally, the device also includes a delay simulation unit, and the acquisition unit 1110 is further used to obtain third environmental information around the autonomous driving vehicle and third state information of the autonomous driving vehicle before obtaining the first environmental information and first state information sent by the vehicle; the determination unit 1120 is further used to extract delay model data based on the third environmental information and the third state information data, and the delay model data includes one or more of the number of application threads, thread periodic data, time probability distribution of thread running and dependency relationship of thread data; the delay simulation unit is used to input the delay model data into the delay simulator to obtain a second delay; the determination unit 1120 is further used to determine the fourth delay optimization parameter corresponding to the third environmental information and the third state information based on the second delay; the storage unit is used to store the correspondence between the third environmental information, the third state information and the fourth delay optimization parameter in the mapping relationship.

[0240] Optionally, the acquisition unit 1110 is used to: receive data collected from the sensor of the autonomous driving vehicle and the third state information; index the data collected by the sensor and the third state information according to time and location to obtain the third environment information and the third state information.

[0241] Optionally, the acquisition unit 1110 is configured to index the data collected by the sensor and the third state information according to time and location through a crowdsourcing algorithm or a group intelligence perception algorithm to obtain the third environment information and the third state information.

[0242] It should be understood that the division of the various units in the above device is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity, or they may be physically separated. Furthermore, the units in the device may be implemented in the form of a processor calling software; for example, the device includes a processor connected to a memory storing instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or the functions of the various units in the device, where the processor is, for example, a general-purpose processor such as a CPU or a microprocessor, and the memory is a memory within the device or a memory external to the device. Alternatively, the units in the device may be implemented in the form of hardware circuits, and the functions of some or all of the units may be implemented through the design of the hardware circuits. The hardware circuits may be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units may be implemented through the design of the logical relationships between the components within the circuits. In another implementation, the hardware circuit may be implemented using a PLD, such as an FPGA, which may include a large number of logic gate circuits, and the connections between the logic gate circuits may be configured using a configuration file to implement the functions of some or all of the above units. All units of the above apparatus may be implemented entirely in the form of software called by a processor, or entirely in the form of hardware circuits, or partially in the form of software called by a processor and the rest in the form of hardware circuits.

[0243] Each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0244] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0245] An embodiment of the present application also provides a device, which includes a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the device executes the method or steps performed by the above embodiment.

[0246] Optionally, if the device is located in a vehicle, the processing unit may be the processors 121 - 12n shown in FIG. 1 .

[0247] An embodiment of the present application further provides a data processing system, which includes one or more sensors and a computing platform, wherein the computing platform includes the above-mentioned data processing device 900 or data processing device 1000.

[0248] An embodiment of the present application further provides a vehicle, which may include the above-mentioned data processing device 900, or the above-mentioned data processing device 1000, or the above-mentioned data processing system.

[0249] An embodiment of the present application further provides a server, which includes the above-mentioned data processing device 1100.

[0250] An embodiment of the present application further provides a computer program product, which includes: computer program code, which enables the computer to execute the above-mentioned data processing method when the computer program code is run on a computer.

[0251] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable medium stores a program code. When the computer program code is run on a computer, the computer executes the above-mentioned data processing method.

[0252] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or a power-on erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0253] It should be understood that in the embodiment of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.

[0254] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0255] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0256] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0259] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0260] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0261] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be covered and fall within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: include: Acquiring first environmental information around a vehicle and first state information of the vehicle; Sending the first environment information and the first status information to a cloud server; Receiving first delay optimization configuration information sent by the cloud server, the cloud server storing a mapping relationship between the first environment information, the first state information, and the first delay optimization configuration information, the first delay optimization configuration information including a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; The first data collected by the sensor is processed according to the first delay optimization parameter.

2. The method according to claim 1, characterized in that: The method further comprises: When the delay of the processing unit for processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to a delay threshold, acquiring second environment information around the vehicle and second state information of the vehicle; Sending the second environment information and the second status information to the cloud server; receiving second delay optimization configuration information sent by the cloud server, wherein the cloud server stores a mapping relationship between the second environment information, the second state information, and the second delay optimization configuration information, and the second delay optimization configuration information includes a second delay optimization parameter when the processing unit processes the data collected by the sensor; The second data collected by the sensor is processed according to the second delay optimization parameter.

3. The method according to claim 1 or 2, characterized in that: The processing unit includes a main processing unit and a backup processing unit, and the processing of the first data collected by the sensor according to the first delay optimization parameter includes: deploying the first delay optimization parameter in the backup processing unit; In a first time period, when the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, the first delay optimization parameter is deployed in the main processing unit.

4. The method according to claim 3, characterized in that The method further comprises: deploying the first delay optimization parameter before the main processing unit: In the first time period, it is determined that the vehicle control instruction output by the backup processing unit is the same as the vehicle control instruction output by the main processing unit.

5. The method according to any one of claims 1 to 4, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay configuration information includes delay optimization parameters when the processing unit processes the data collected by the sensor at multiple moments in the future.

6. The method according to any one of claims 1 to 5, characterized in that The first environmental information includes at least one of time, geographical location, climate and road conditions; and / or, The first status information includes at least one of the hardware specifications, system status, and automatic driving status of the vehicle.

7. A data processing method, characterized in that: include: Acquiring first environmental information around a vehicle and first state information of the vehicle; Based on the first environmental information, the first state information and the mapping relationship, the first delay optimization configuration information is sent to the vehicle, the mapping relationship includes a mapping relationship between environmental information, state information and delay optimization configuration information, and the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes the data collected by the sensor.

8. The method according to claim 7, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay optimization configuration information includes delay optimization parameters for processing the data collected by the sensor at multiple future moments.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: According to the first environment information and the first state information, a hierarchical navigable small-world HNSW algorithm is used to retrieve the mapping relationship to obtain the first delay optimization configuration information.

10. The method according to any one of claims 7 to 9, characterized in that Before acquiring the first environment information around the vehicle and the first state information of the vehicle, the method further includes: Acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; Determine a first delay when processing data collected by the sensor according to a third delay optimization parameter set by the offline simulation system, the third environment information, and the third state information; According to the first delay, the third delay optimization parameter is updated to obtain a fourth delay optimization parameter; The correspondence between the third environment information, the third state information and the fourth delay optimization parameter is stored in the mapping relationship.

11. The method according to any one of claims 7 to 9, characterized in that Before acquiring the first environment information around the vehicle and the first state information of the vehicle, the method further includes: Acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; Extracting latency model data according to the third environment information and the third state information, where the latency model data includes one or more of the number of application threads, periodicity data of threads, time probability distribution of thread running, and dependency relationship of thread data; Inputting the delay model data into a delay simulator to obtain a second delay; determining, according to the second delay, a fourth delay optimization parameter corresponding to the third environment information and the third state information; The correspondence between the third environment information, the third state information and the fourth delay optimization parameter is stored in the mapping relationship.

12. A data processing device, characterized in that: include: An acquisition unit, configured to acquire first environment information around a vehicle and first state information of the vehicle; A sending unit, configured to send the first environment information and the first status information to a cloud server; a receiving unit, configured to receive first delay optimization configuration information sent by the cloud server, wherein the cloud server stores a mapping relationship between the first environment information, the first state information, and the first delay optimization configuration information, wherein the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; The processing unit is used to process the first data collected by the sensor according to the first delay optimization parameter.

13. The device according to claim 12, characterized in that The acquisition unit is further configured to acquire second environment information around the vehicle and second state information of the vehicle when the delay of the processing unit for processing the data collected by the sensor according to the first delay optimization parameter is greater than or equal to a delay threshold; The sending unit is further used to send the second environment information and the second state information to the cloud server; The receiving unit is further used to receive second delay optimization configuration information sent by the cloud server, the cloud server saves a mapping relationship between the second environment information, the second state information and the second delay optimization configuration information, and the second delay optimization configuration information includes a second delay optimization parameter when the processing unit processes the data collected by the sensor; The processing unit is further configured to process the second data collected by the sensor according to the second delay optimization parameter.

14. The device according to claim 12 or 13, characterized in that The processing unit includes a main processing unit and a backup processing unit, and the processing unit is used to: deploying the first delay optimization parameter in the backup processing unit; In a first time period, when the delay of the backup processing unit processing the first data according to the first delay optimization parameter is less than the delay of the main processing unit processing the first data, the first delay optimization parameter is deployed in the main processing unit.

15. The device according to claim 14, characterized in that The device also includes: A determination unit is used to determine that the vehicle control instructions output by the backup processing unit are the same as the vehicle control instructions output by the main processing unit within the first time period before the processing unit deploys the first delay optimization parameter in the main processing unit.

16. The device according to any one of claims 12 to 15, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay configuration information includes delay optimization parameters when the processing unit processes the data collected by the sensor at multiple moments in the future.

17. The device according to any one of claims 12 to 16, characterized in that The first environmental information includes at least one of time, geographical location, climate and road conditions; and / or, The first status information includes at least one of the hardware specifications, system status, and automatic driving status of the vehicle.

18. A data processing device, characterized in that: include: An acquisition unit, configured to acquire first environment information around a vehicle and first state information of the vehicle; a determining unit, configured to determine first latency optimization configuration information according to the first environment information, the first state information and the mapping relationship, The mapping relationship includes a mapping relationship between environment information, state information and delay optimization configuration information, wherein the first delay optimization configuration information includes a first delay optimization parameter when the processing unit in the vehicle processes data collected by the sensor; A sending unit is used to send the first delay optimization configuration information to the vehicle.

19. The device according to claim 18, characterized in that The first environment information includes environment information of the vehicle at multiple moments in the future, and the first state information includes state information of the vehicle at multiple moments in the future; The first delay optimization configuration information includes delay optimization parameters for processing the data collected by the sensor at multiple future moments.

20. The device according to claim 18 or 19, characterized in that The device also includes: An information retrieval unit is used to retrieve the mapping relationship by using a hierarchical navigable small-world HNSW algorithm according to the first environment information and the first state information to obtain the first delay optimization configuration information.

21. The device according to any one of claims 18 to 20, characterized in that The device also includes a parameter updating unit and a storage unit. The acquisition unit is further used to acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; The determining unit is further used to determine a first delay for processing data collected by the sensor according to a third delay optimization parameter set by the offline simulation system, the third environment information and the third state information; The parameter updating unit is further configured to update the third delay optimization parameter according to the first delay to obtain a fourth delay optimization parameter; The storage unit is further configured to store a correspondence between the third environment information, the third state information and the fourth delay optimization parameter in the mapping relationship.

22. The device according to any one of claims 18 to 20, characterized in that The device also includes a time delay simulation unit and a storage unit. The acquisition unit is further used to acquire third environment information around the autonomous driving vehicle and third state information of the autonomous driving vehicle; The determining unit is further configured to extract latency model data according to the third environment information and the third state information, wherein the latency model data includes one or more of the number of application threads, periodicity data of threads, time probability distribution of thread running, and dependency relationship of thread data; The delay simulation unit is used to input the delay model data into a delay simulator to obtain a second delay; The determining unit is further configured to determine, according to the second delay, a fourth delay optimization parameter corresponding to the third environment information and the third state information; The storage unit is used to store the corresponding relationship between the third environment information, the third state information and the fourth delay optimization parameter in the mapping relationship.

23. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program stored in the memory, so that the apparatus performs the method according to any one of claims 1 to 6.

24. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program stored in the memory, so that the apparatus performs the method according to any one of claims 7 to 11.

25. A vehicle, characterized in that: Comprising the apparatus as claimed in any one of claims 12 to 17, or comprising the apparatus as claimed in claim 23.

26. A server, characterized in that: Comprising the apparatus as claimed in any one of claims 18 to 22, or comprising the apparatus as claimed in claim 24.

27. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented, or the method according to any one of claims 7 to 11 is implemented.

Images