Sensor data optimization method, autonomous vehicle, related apparatus and medium

Abstract

This disclosure provides a sensor data optimization method, an autonomous vehicle, related devices, and a medium, relating to the fields of unmanned vehicles and autonomous driving. The method includes: acquiring quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through a transmission channel; when the quality parameters of the sensor data do not meet preset requirements, calculating an optimal time interval for the autonomous driving system to read the autonomous driving data based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel; and updating the time interval for the autonomous driving system to read the autonomous driving data to the optimal time interval. This disclosure reduces the latency and frame drop rate of sensor data being received by appropriate modules, improving the reliability of the autonomous driving performance of autonomous vehicles.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, specifically to a sensor data optimization method, an autonomous vehicle, related devices, and media. Background Technology

[0002] A vehicle operating in autonomous driving mode (e.g., driverless) refers to a vehicle configurable to operate in autonomous driving mode, navigating its driving environment with minimal or no input from the driver. Autonomous vehicles comprise subsystems such as a driving system, a sensor system, and a control system. The sensor system includes one or more sensors (e.g., cameras, Global Positioning System (GPS), Inertial Measurement Unit (IMU), LiDAR, etc.) for collecting sensor data. Based on sensor data, the autonomous driving system perceives the environment surrounding the vehicle and plans its trajectory, which serves as input to the control system to determine the vehicle's final path. The autonomous driving system, along with the related software for these subsystems such as the driving system, sensor system, and control system, executes on computing devices (e.g., high-performance computing units such as CPUs, GPUs, NPUs, etc.). The computing resources of these devices are typically consumed by multiple threads of the autonomous driving system and the driving system, sensor system, and control system, resulting in high utilization of the computing resources. The computing resources of computing devices are often limited, which can easily lead to sensor data not having the necessary computing resources during autonomous driving. The computing device cannot process the sensor data in a timely manner, resulting in delays and frame drops in the reception of sensor data by the appropriate modules, thereby reducing the reliability of the autonomous driving performance of autonomous vehicles. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a sensor data optimization method, an autonomous vehicle, related devices, and a medium, which can reduce the latency and frame drop rate of sensor data being received by appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle.

[0004] According to a first aspect of this disclosure, a method for optimizing sensor data for autonomous vehicles is provided, comprising:

[0005] To acquire quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through the transmission channel;

[0006] When the quality parameters of the sensor data do not meet the preset requirements, the optimal time interval for the autonomous driving system to read the autonomous driving data is calculated based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel. The autonomous driving data is the interaction data between multiple subsystems of the autonomous vehicle and the autonomous driving system.

[0007] The time interval for the autonomous driving system to read the autonomous driving data is updated to the optimal time interval, wherein the autonomous driving system realizes the autonomous driving of the autonomous vehicle based on the read sensor data.

[0008] Optionally, the quality parameters are determined based on the frame drop rate and / or latency rate.

[0009] Optionally, the autonomous driving data includes multiple autonomous driving data sets. When the quality parameters of the sensor data do not meet preset requirements, calculating the optimal time interval for the autonomous driving system to read the autonomous driving data, based on the number of frames per second transmitted by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel, includes:

[0010] Based on the number of frames per second transmitted from the autonomous driving data and the maximum number of frames per second transmitted from the multiple autonomous driving data, the optimal number of bytes that the autonomous driving system reads at one time is obtained;

[0011] The quotient of the optimal number of bytes of autonomous driving data that the autonomous driving system reads at one time and the data transmission rate of the transmission channel is taken as the optimal time interval for the autonomous driving system to read the autonomous driving data.

[0012] Optionally, obtaining the optimal number of bytes that the autonomous driving system reads at one time based on the number of frames transmitted per second of the autonomous driving data and the maximum number of frames transmitted per second of the plurality of autonomous driving data includes:

[0013] Based on the number of frames transmitted per second and the number of bytes per frame of the autonomous driving data, the total number of bytes transmitted per second of the multiple autonomous driving data is obtained.

[0014] The quotient of the total number of bytes transmitted per second of the multiple autonomous driving data and the maximum number of frames transmitted per second of the multiple autonomous driving data is taken as the optimal number of bytes that the autonomous driving system reads at one time.

[0015] Optionally, the computing device of the autonomous vehicle includes one or more processors, the processors including one or more processor cores, and the sensor data optimization method further includes:

[0016] The first thread that processes the sensor data on the computing device is bound to the first processor core among the one or more processor cores, so that the first thread is processed only by the first processor core.

[0017] Optionally, the multiple subsystems of the autonomous vehicle and the multiple threads of the autonomous driving system are processed by a computing device, and the sensor data optimization method further includes:

[0018] Priorities are set for the multiple threads such that the first thread of the computing device that processes the sensor data has a higher priority than the other real-time threads;

[0019] Based on a real-time preemptive scheduling algorithm, the computing device schedules the multiple threads according to their priorities.

[0020] Optionally, after updating the time interval for the autonomous driving system to read the autonomous driving data to the optimal time interval, the real-time preemptive scheduling algorithm exits when the total number of bytes transmitted per second of the autonomous driving data is lower than the threshold.

[0021] According to a second aspect of this disclosure, a sensor data optimization apparatus for an autonomous vehicle is provided, comprising:

[0022] The quality parameter acquisition module is configured to acquire quality parameters from sensor data read by the autonomous driving system of an autonomous vehicle through a transmission channel.

[0023] The calculation module is configured to, when the quality parameters of the sensor data do not meet the preset requirements, calculate the optimal time interval for the autonomous driving system to read the autonomous driving data based on the number of frames per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel. The autonomous driving data is the interaction data between multiple subsystems of the autonomous vehicle and the autonomous driving system.

[0024] An optimization module is configured to update the time interval at which the autonomous driving system reads the autonomous driving data to the optimal time interval, wherein the autonomous driving system enables the autonomous vehicle to drive automatically based on the read sensor data.

[0025] According to a third aspect of this disclosure, an autonomous vehicle is provided, comprising:

[0026] The sensor system is configured to collect sensor data;

[0027] A sensor data optimization device is configured to perform the method described in any of the above-mentioned embodiments;

[0028] An autonomous driving system is configured to enable the autonomous vehicle to drive automatically based on the sensor data.

[0029] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described above.

[0030] According to a fifth aspect of this disclosure, a storage medium is provided that stores a computer program or instructions, which, when executed by a processor, implement the steps of the method described above.

[0031] According to embodiments of this disclosure, the quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through a transmission channel are obtained. When the quality parameters of the sensor data do not meet preset requirements, the optimal time interval for the autonomous driving system to read the autonomous driving data is calculated based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel. This optimal time interval is then updated. The autonomous driving data refers to the interactive data between multiple subsystems of the autonomous vehicle and the autonomous driving system. This optimal time interval ensures that sensor data is not delayed due to waiting for the autonomous driving system to read data, nor is data loss caused by the autonomous driving system reading data too frequently. This reduces the latency and frame loss rate of sensor data being received by appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle. Furthermore, the optimal time interval for the autonomous driving system to read the autonomous driving data prevents the sensor data from increasing the computational resource burden on the computing device due to the autonomous driving system reading data too frequently, increasing the likelihood that the sensor data will have the necessary computational resources. This further reduces the latency and frame loss rate of sensor data being received by appropriate modules, improving the reliability of the autonomous driving performance of the autonomous vehicle.

[0032] It should be noted that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this disclosure. Attached Figure Description

[0033] Figure 1 This diagram illustrates the structure of an autonomous vehicle according to an embodiment of the present disclosure.

[0034] Figure 2 A schematic diagram of the structure of a computing device provided according to an embodiment of the present disclosure is shown;

[0035] Figure 3This diagram illustrates the structure of an autonomous driving system provided according to an embodiment of the present disclosure.

[0036] Figure 4 This diagram illustrates the structure of a sensor data optimization apparatus provided according to an embodiment of the present disclosure.

[0037] Figure 5 A schematic flowchart of a sensor data optimization method provided according to an embodiment of the present disclosure is shown.

[0038] Figure 6 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0039] To facilitate understanding of this disclosure, a more complete description will now be given with reference to the accompanying drawings, in which preferred embodiments of the present disclosure are shown. However, this disclosure may be implemented in various forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure.

[0040] The following concepts are used in this article:

[0041] An autonomous driving system is a driving system that uses onboard sensor systems to perceive the surrounding environment, automatically plan a driving route, and control an unmanned vehicle to travel along that route. The autonomous driving system in this disclosure can be an advanced driver assistance system (ADAS), an advanced driving system, or any other driving system that includes a planning and control module. Based on behavioral decisions, the planning and control module plans and generates a trajectory that satisfies specific constraints (such as vehicle dynamics constraints, collision avoidance, and passenger comfort). This trajectory serves as the input to the control system, determining the vehicle's final driving path.

[0042] Sensor data: This refers to data acquired through vehicle-mounted sensors, such as cameras, LiDAR, and millimeter-wave radar. The sensor data in this disclosure can be raw video streams, radar point cloud data, or analyzed, structured information or data regarding the position, speed, steering angle, and size of people, vehicles, and objects. For raw video stream data and radar point cloud data, the perception module can process this sensor data into identifiable, structured information or data regarding the position, speed, steering angle, and size of people, vehicles, and objects.

[0043] Figure 1A schematic diagram of the structure of an autonomous vehicle provided according to an embodiment of the present disclosure is shown. The autonomous vehicle 100 can operate in manual mode, fully autonomous driving mode, or partially fully autonomous driving mode. In some embodiments, such as Figure 1 As shown, the autonomous vehicle 100 includes subsystems such as a driving system 110, a sensor system 120, and a control system 130. Each subsystem in the autonomous vehicle 100 includes multiple components, and each subsystem is interconnected via wired or wireless means.

[0044] In some embodiments, the sensor system 120 includes a positioning system 121, an inertial measurement unit (IMU) 122, a radar 123, and a camera 124. The positioning system 121 may be a Global Positioning System (GPS), a BeiDou system, or another positioning system. In some embodiments, the positioning system 121 is used to provide information about the location of the autonomous vehicle 100. The IMU unit 122 is used to sense changes in the position and orientation of the autonomous vehicle 100 based on inertial acceleration. In one example, the IMU unit 122 is a combination of an accelerometer and a gyroscope. The radar 123 is used to sense objects in the surrounding environment of the autonomous vehicle 100 using laser light. In one example, in addition to sensing objects, the radar 123 is also used to sense the speed and / or direction of travel of the objects. In one example, the radar 123 also includes one or more laser sources, a laser scanner, one or more detectors, and other system components. The camera 124 is used to acquire multiple images of the environment surrounding the autonomous vehicle 100. The camera 124 may be a still camera and / or a video camera. In one example, camera 124 is mechanically movable, for example, by mounting camera 124 on a rotating and / or tilting platform. In some embodiments, sensor system 120 may also include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). The audio sensor may be configured to collect sound from the environment surrounding the autonomous vehicle 100. The steering sensor may be configured to sense the steering angle of the steering wheel, the vehicle's wheels, or a combination thereof. The throttle sensor and brake sensor sense the throttle position and braking position of the vehicle, respectively. In one example, the throttle sensor and brake sensor may be integrated into an integrated throttle / brake sensor.

[0045] It should be understood that the sensor data acquired by the sensor system 120 is crucial for the safe operation of the autonomous vehicle 100. In some embodiments, the sensor data includes vehicle data containing information indicating issued driving commands (e.g., accelerator, brake, steering commands) and the vehicle's responses (e.g., speed, acceleration, deceleration, direction) captured by the vehicle's sensors at different points in time (or time frames). In some embodiments, the sensor data also includes obstacle (or object) information, such as obstacle location and obstacle reflectivity. In some embodiments, at the moment the sensors in the sensor system 120 acquire the sensor data, a positioning system timestamp (e.g., a GPS timestamp) is added to each sensor data point based on the current time of the positioning system 121.

[0046] In some embodiments, the propulsion system 110 includes an engine, transmission, energy source, and wheels (tires). In some embodiments, the control system 130 is used to control the autonomous vehicle 100 and its components. The control system 130 includes a steering system 131, a throttle 132, a braking unit 133, a computer vision system 134, a route control system 135, and an obstacle avoidance system 136. In some embodiments, the steering system 131, such as a steering wheel system, is used to adjust the forward direction of the autonomous vehicle 100. The throttle 132 is used to control the operating speed of the engine and thus the speed of the autonomous vehicle 100. The braking unit 133 is used to control the deceleration of the autonomous vehicle 100. In one example, the braking unit 133 may use friction to slow the wheels. In some embodiments, the computer vision system 134 is used to process and analyze images captured by the camera 124 to identify objects and / or features in the environment surrounding the autonomous vehicle 100. These objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 134 uses object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 134 is further configured to map the environment, track objects, and estimate the speed of objects. In some embodiments, the route control system 135 is configured to determine the driving route of the autonomous vehicle 100. In one example, the route control system 135 combines data from sensors, GPS, and one or more predetermined maps to determine the driving route for the autonomous vehicle 100. In some embodiments, the obstacle avoidance system 136 is configured to identify, assess, and avoid or otherwise traverse potential obstacles in the environment of the autonomous vehicle 100.

[0047] In some embodiments, the autonomous vehicle 100 also includes a computing device 140. Figure 2 A schematic diagram of the structure of a computing device provided according to an embodiment of the present disclosure is shown. In some embodiments, such as Figure 2 As shown, the computing device 140 may include one or more processors 22 and a memory 29. The processor 22 includes one or more processor cores (not shown in the figure). The memory 29 in the computing device 140 may be main memory (or simply main memory or RAM) for storing instruction information and / or data information represented by data signals, and may also be used to realize data exchange between the processor 22 and the external storage device 26 (or auxiliary memory or external memory).

[0048] In some situations, the processor 22 may need to access memory 29 to retrieve or modify data in memory 29. Because access to memory 29 is relatively slow, to alleviate the speed difference between the processor 22 and memory 29, the computing device 140 also includes a cache memory 28 coupled to bus 21. The cache memory 28 is used to cache program data or message data that may be repeatedly accessed in memory 29. The cache memory 28 is implemented, for example, by a storage device of the type Static Random Access Memory (SRAM).

[0049] Based on this, the processor 22 may include an instruction execution unit 221, a memory management unit 222, and other components. When executing instructions that require memory modification, the instruction execution unit 221 initiates a write access request, which specifies the data to be written into memory and the corresponding physical address. The memory management unit 222 is used to translate the virtual address specified by these instructions into the physical address mapped by that virtual address. The physical address specified by the write access request can be the same as the physical address specified by the corresponding instruction.

[0050] like Figure 2 As shown, processor 22, cache memory 28, and memory 29 are packaged in a system-on-a-chip (SoC) 201. Designers can configure the SoC architecture to ensure secure communication between the various components in computing device 140.

[0051] In this example, computing device 140 may also include various software components, such as an embedded operating system 202 and application programs 203 shown in the figure. This software may be embedded in memory 29 or stored in external memory 26. In some embodiments, operating system 202 includes a shell 204 and a kernel 205. The shell 204 is an interface between the user and the operating system kernel. The shell 204 is the outermost layer of the operating system, managing the interaction between the user and the operating system, such as waiting for user input, interpreting user input for the operating system, and processing various operating system outputs. The kernel 205 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. The operating system kernel interacts directly with the hardware, typically running threads and providing inter-thread communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0052] In some embodiments, application 203 includes programs related to controlling vehicle movement, such as programs managing the interaction between the vehicle and road obstacles, autonomous driving programs controlling the vehicle's route or speed, programs controlling the interaction between the vehicle and other vehicles on the road, programs planning the vehicle's trajectory based on data collected by sensors, and programs controlling the vehicle's movement according to the planned trajectory, to achieve the autonomous driving function of autonomous vehicle 100. In some embodiments, application 203 includes related software for autonomous driving system 150 and subsystems such as driving system 110, sensor system 120, and control system 130. That is, the related software for autonomous driving system 150 and subsystems such as driving system 110, sensor system 120, and control system 130 is executed on computing device 140. Figure 3 A schematic diagram of the structure of an autonomous driving system provided according to an embodiment of the present disclosure is shown. In some embodiments, such as Figure 3As shown, the autonomous driving system 150 includes a perception module 151, a fusion module 152, and a traffic control module 153. In some embodiments, the perception module 151 executes a specific perception algorithm to perceive the environment around the autonomous vehicle 100 based on sensor data collected by the sensor system 120. Environmental perception typically includes perceiving road surfaces, static objects, and dynamic objects. For dynamic objects, their trajectories are tracked, and the next trajectory (position) of the object is predicted based on the tracking results. In one example, the perception information includes lane configurations (e.g., straight lanes or curved lanes), traffic light signals, relative positions to nearby vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, yield signs). In some embodiments, the perception module 151 uses deep learning algorithms to identify objects in the vehicle's surrounding environment. In some embodiments, the fusion module 152 controls the sensors, combines and processes the data received from each sensor, helps predict the state of traffic participants, and outputs traffic environment information. In some embodiments, the traffic control module 153 calculates the vehicle's safe space based on the information output by the fusion module 152, i.e., performs trajectory planning for the vehicle. For example, the planning and control module 153 combines the information output by the fusion module 152 to plan a safe driving path for the vehicle (also known as the vehicle's predicted trajectory). In some embodiments, the computing device 140 calls the control system 130 based on the vehicle predicted trajectory information output by the planning and control module 153 to control the autonomous vehicle 100 to drive along the trajectory indicated by the predicted trajectory information.

[0053] In some embodiments, application 203 may also be a program for controlling or responding to external devices (e.g., biometric sensors, microphones, speakers, or other I / O components, actuators, or devices), a program for various I / O tasks, a security program, an authentication program, various computing modules, a communication program, a communication support protocol, or other programs, or combinations thereof.

[0054] It should be understood that the computing resources of computing device 140 are occupied by subsystems such as driving system 110, sensor system 120, and control system 130, as well as multiple threads of autonomous driving system 150. The computing resource utilization of computing device 140 is usually very high, and the computing resources of computing device 140 are often limited. A large number of threads such as network or CAN bus interruptions, data reception, copying, transmission, and processing will compete for the computing resources of sensor data processing threads, and may even cause sensor data to be interrupted for more than ten seconds. This can easily lead to sensor data not being able to enjoy the necessary computing resources during autonomous driving, and computing device 140 not being able to process sensor data in a timely manner. This causes delays and frame drops in the reception of sensor data by appropriate modules (e.g., modules in control system 130 and autonomous driving system 150), thereby reducing the reliability of autonomous driving performance of autonomous vehicle 100.

[0055] In some embodiments, the computing device 150 may further include a sensor data optimization device 300. In some embodiments, the sensor data optimization device 300 is used to optimize the latency and frame drop rate of sensor data collected by the sensor system 120 of the autonomous vehicle 100 being received by appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle 100. Since the process of optimizing the latency and frame drop rate of sensor data collected by the sensor system 120 of the autonomous vehicle 100 being received by appropriate modules will be described below, it will not be repeated here.

[0056] In addition, the computing device 140 may also include a storage device 26, a display device 23, an audio device 24, and an input / output device 25. The storage device 26 may be a hard disk, optical disk, or flash memory, etc., coupled to the bus 21 via a corresponding interface, for information storage and retrieval. The display device 23 may be coupled to the bus 21 via a corresponding graphics card, for displaying information according to display signals provided by the bus 21.

[0057] The computing device 150 typically also includes a communication device 27, which allows it to communicate with networks or other devices in various ways. The communication device 27 may include, for example, one or more communication modules; as an example, the communication device 27 may include a wireless communication module adapted to a specific wireless communication protocol.

[0058] Of course, the structure of different computer systems may vary depending on the motherboard, operating system, and instruction set architecture. For example, many current computer systems have an input / output control center connected between the bus 21 and various input / output devices, and this input / output control center may be integrated into the processor 22 or independent of the processor 22.

[0059] Figure 4A schematic diagram of the structure of a sensor data optimization apparatus provided according to an embodiment of the present disclosure is shown. Figure 4 As shown, the sensor data optimization device 300 includes a quality parameter acquisition module 310, a calculation module 320, and an optimization module 330. The quality parameter acquisition module 310, the calculation module 320, and the optimization module 330 can be implemented by hardware, software, or a combination thereof.

[0060] In some embodiments, the quality parameter acquisition module 310 is used to acquire quality parameters of sensor data read by the autonomous driving system 150 of the autonomous vehicle 100 through a transmission channel. In some embodiments, the quality parameter acquisition module 310 includes an autonomous driving system timestamp annotation unit 311 and a log recording and visualization unit 312. It should be understood that the driving system 110, sensor system 120, and control system 130 of the autonomous vehicle 100, as well as the system time of the autonomous driving system 150, are synchronized. The sensor system 120 collects sensor data and, at the moment the sensor data is collected, adds a positioning system timestamp (e.g., GPS timestamp) to each sensor data according to the current time of the positioning system 121. In some embodiments, the autonomous driving system 150 uses a function call to read sensor data (e.g., vehicle data, driving commands, and obstacle information, etc.) and parses it, transmitting the sensor data to the autonomous driving system 150 (e.g., perception module 151) via a communication channel including a serial port, CAN bus, etc. The perception module 151 performs environmental perception around the autonomous vehicle 100 based on the parsed sensor data. In some embodiments, when the autonomous driving system 150 transmits the parsed sensor data to its perception module 151 using a function call, the autonomous driving system timestamp annotation unit 311 adds an autonomous driving system timestamp to each sensor data. In some embodiments, the log recording and visualization unit 312 records the sensor data and the corresponding positioning system timestamp and autonomous driving system timestamp, calculates and visualizes the quality parameters of the sensor data read by the autonomous driving system 150 of the autonomous vehicle 200 through the transmission channel based on the sensor data and the corresponding positioning system timestamp and autonomous driving system timestamp, wherein the quality parameters are determined based on the frame drop rate and / or latency rate. It should be understood that when the time interval between the positioning system timestamp of the current frame data and the positioning system timestamp of the previous frame data is greater than or significantly greater than a preset inter-frame time interval, the current frame data is considered to have frame drop. Based on this, the quotient of the number of frames with frame drop in the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period of time and the total number of frames read can be used as the frame drop rate of the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period of time. If the time interval between the positioning system timestamp and the autonomous driving system timestamp of the current frame data is greater than a preset inter-frame time interval, the current frame data is considered to have a delay. Based on this, the quotient of the number of delayed frames in the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period of time, and the total number of frames read, can be used as the delay rate of the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period of time.

[0061] In some embodiments, the calculation module 320 is used to calculate the optimal time interval for the autonomous driving system 150 to read autonomous driving data based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system 150 through the transmission channel and the data transmission rate of the transmission channel when the quality parameters of the sensor data do not meet the preset requirements. It should be noted that the autonomous driving data is the interaction data between the autonomous driving vehicle 100 and its multiple subsystems (e.g., the driving system 110, sensor system 120, and control system 130), including sensor data, call instructions and control instructions from the autonomous driving system 150 to the control system 130 and other subsystems, and predicted trajectory information, etc. In some embodiments, the calculation module 320 includes a comparison module 321 and an optimal time interval calculation unit 322. In some embodiments, the comparison module 321 compares the quality parameters of the sensor data with preset thresholds, that is, compares the frame loss rate and latency rate of the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period with the corresponding preset thresholds. In one example, if the frame loss rate and latency rate of the sensor data are higher than the corresponding preset thresholds, then the quality parameters of the sensor data are considered to be lower than the preset thresholds. In another example, if either the frame drop rate or the latency rate of the sensor data exceeds a corresponding preset threshold, the quality parameter of the sensor data is considered to be below the preset threshold. In some embodiments, when the quality parameter of the sensor data is below the preset threshold, the optimal time interval calculation unit 322 calculates the optimal time interval for the autonomous driving system to read the autonomous driving data based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system 150 through the transmission channel and the data transmission rate of the transmission channel. In some embodiments, the optimal time interval calculation unit 322 obtains the optimal number of bytes that the autonomous driving system 150 reads at one time based on the number of frames transmitted per second of the autonomous driving data and the maximum number of frames transmitted per second of multiple autonomous driving data. In one example, the optimal time interval calculation unit 322 obtains the total number of bytes transmitted per second of multiple autonomous driving data based on the number of frames transmitted per second of the autonomous driving data and the number of bytes per frame; the quotient of the total number of bytes transmitted per second of multiple autonomous driving data and the maximum number of frames transmitted per second of multiple autonomous driving data is taken as the optimal number of bytes that the autonomous driving system 150 reads at one time. In one example, the formula for calculating the optimal number of bytes that the autonomous driving system 150 reads at one time is as follows:

[0062]

[0063] Where S is the optimal number of bytes that the autonomous driving system can read at one time (FPS). iBPF is the number of frames per second (fps) of the i-th autonomous driving data. i It is the number of bytes per frame (Bytes Per Frame) for the i-th autonomous driving data, 0≤i≤N-1, where N is the number of autonomous driving data read by the autonomous driving system 150 through the transmission channel, FPS. max It is the maximum number of frames per second that the autonomous driving system 150 can transmit through the transmission channel for N autonomous driving data.

[0064] In some embodiments, the optimal time interval calculation unit 322 uses the quotient of the optimal number of bytes of autonomous driving data read by the autonomous driving system 150 at one time and the data transmission rate of the transmission channel as the optimal time interval for the autonomous driving system 150 to read autonomous driving data. In one example, the formula for calculating the optimal time interval for the autonomous driving system 150 to read autonomous driving data is as follows:

[0065]

[0066] Among them, t interval S is the optimal time interval for the autonomous driving system 150 to read autonomous driving data, and V is the optimal number of bytes of autonomous driving data that the autonomous driving system 150 reads at one time. baudrate It is the data transmission rate of the transmission channel.

[0067] In some embodiments, the optimization module 330 updates the time interval for the autonomous driving system 150 to read autonomous driving data to the optimal time interval. The autonomous driving system 150 enables the autonomous vehicle 100 to drive automatically based on the read sensor data. It is understood that this optimal time interval for the autonomous driving system 150 to read autonomous driving data ensures that sensor data is not delayed due to waiting for the autonomous driving system 150 to read data, nor is data loss caused by the autonomous driving system 150 reading data too frequently. This reduces the latency and frame drop rate of sensor data being received by the appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle 100.

[0068] In some embodiments, the quality parameters of the sensor data read by the autonomous driving system 150 through the transmission channel can also be determined based on the frame error rate. In some embodiments, when an error occurs in the current frame data (e.g., the current frame data is incomplete, the data content is incorrect, or it is composed of partial data from multiple frames), the current frame data is considered to have a frame error phenomenon. Based on this, the quotient of the number of frames with frame errors and the total number of frames read by the autonomous driving system 150 through the transmission channel within a certain period of time can be used as the frame error rate of the sensor data read by the autonomous driving system 150 through the transmission channel within a certain period of time. In some embodiments, refer again Figure 1 The sensor system 120 includes a radar 123, which works by emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, information about the target, such as distance, azimuth, altitude, speed, attitude, and even shape, can be obtained, enabling target detection, tracking, and identification. The measurement data from the radar 123 is generally called a point cloud (also known as point cloud data). Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scan data is recorded in the form of points, and each point cloud data contains the position information (i.e., three-dimensional coordinates) of the point cloud points. Some may contain color information (RGB) or reflection intensity information. In some embodiments, the radar 123's structural design requires multiple prisms (also called prism faces) to reflect the laser signal. Assuming the radar 123 has n prism faces, when measuring the same object, each prism face corresponds to a set of point cloud data. That is, the point cloud data includes the point cloud data corresponding to n prisms. Each point cloud data set may also include the azimuth angle of the corresponding prism. A single frame of point cloud data often transmits multiple packets. If network congestion or insufficient computing resources occur, packet loss may occur within a single frame, leading to multiple frames of point cloud data being pieced together to form a single frame, resulting in virtual obstacles. In some embodiments, the optimization module 330 is further configured to perform packet fragmentation processing on the point cloud data if the azimuth angle of the point cloud data is not within the 360-degree range and / or the number of data packets in a single frame of point cloud data differs from a preset threshold. This ensures that the point cloud data frames are not corrupted, thereby reducing the frame error rate of the point cloud data.

[0069] Refer again Figure 2The computing device 140 includes one or more processors 22, each processor 22 including one or more processor cores (not shown). In some embodiments, the optimization module 330 is further configured to perform a core-binding operation on a first thread of the computing device 150 that processes sensor data and a first processor core among the one or more processor cores, so that the first thread is processed only by the first processor core. In this way, after the core-binding operation, the context switching of the kernel on the sensor data processing thread can be reduced, and the necessary computing resources can be allocated to the sensor data processing thread. This allows the computing device to process sensor data in a timely manner, reducing the latency and frame drop rate of sensor data being received by the appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle. In one example, the core-binding operation function is as follows:

[0070] int pthread_setaffinity_np(pthread_t thread,size_t cpusetsize,constcpu_set_t*cpuset);

[0071] The `pthread_setaffinity_np` function performs a core-binding operation. The parameter `thread` is the thread handle, `cpusetsize` is the size of the core-binding operation variable, and `cpuset` is the handle to the core-binding variable. This function can perform the corresponding core-binding operation before the thread starts. It is understood that this core-binding function is existing technology, and therefore any existing method can be used to perform a core-binding operation between the first thread of the computing device 150 that processes sensor data and the first processor core of one or more processor cores.

[0072] In some embodiments, multiple subsystems of the autonomous vehicle 100 (e.g., the driving system 110, sensor system 120, and control system 130) and multiple threads of the autonomous driving system 150 are processed by a computing device 140. The computing resources of the computing device 140 are occupied by the driving system 110, sensor system 120, control system 130, and multiple threads of the autonomous driving system 150. In some embodiments, the optimization module 330 is further configured to set priorities for the multiple subsystems of the autonomous vehicle 100 (e.g., the driving system 110, sensor system 120, and control system 130) and multiple threads of the autonomous driving system 150, such that the priority of the first thread of the computing device 140 processing sensor data is higher than the priority of other real-time threads; and based on a real-time preemptive scheduling algorithm (SCHED_RR), the computing device 140 schedules multiple threads according to their priorities. In some embodiments, after the optimization module 330 updates the time interval for the autonomous driving system 150 to read autonomous driving data to the optimal time interval, the real-time preemptive scheduling algorithm exits when the total number of bytes transmitted per second of autonomous driving data falls below a threshold. This improves the real-time performance and uniformity of scheduling the first thread of the computing device 140 that processes sensor data. The computing device can process sensor data promptly, reducing the latency and frame drop rate of sensor data being received by the appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle. In one example, the function for scheduling multiple threads based on their priorities is as follows:

[0073] int pthread_attr_setschedpolicy(pthread_attr_t*attr,int policy);

[0074] Here, `pthread_attr` is the scheduling function, the `attr` variable is the scheduler configuration variable, and `policy` is the scheduling policy, which is set to `SCHED_RR` here. It's understandable that this scheduling function is existing technology, so any existing method can be used to use this function to schedule multiple threads based on their priorities.

[0075] In one embodiment, multiple subsystems of the autonomous vehicle 100 (e.g., the driving system 110, sensor system 120, and control system 130) communicate with the autonomous driving system 150 via a communication middleware (e.g., ROS, Cyber, or DDS). Sensor data is transmitted as message sequences or message streams between the multiple subsystems of the autonomous vehicle 100 and the autonomous driving system 150, and the message sequences or message streams of sensor data are typically cached in a buffer queue. When there is pulse load in the communication middleware, sensor data may not receive a response from the communication middleware callback function, and the length of the buffer queue used to cache sensor data may not match the real-time processing capability of the current communication middleware, causing latency and frame drops. In one embodiment, the optimization module 330 is further configured to adjust the length of the buffer queue used to cache sensor data to balance the frame drop rate and latency rate when the autonomous driving system reads sensor data.

[0076] Figure 5 A schematic flowchart of a sensor data optimization method according to an embodiment of this disclosure is shown. (See reference...) Figure 5 The sound source localization method provided in this embodiment includes steps S510 to S530.

[0077] In step S510, the quality parameters of the sensor data read by the autonomous driving system of the autonomous vehicle through the transmission channel are obtained.

[0078] In step S520, when the quality parameters of the sensor data do not meet the preset requirements, the optimal time interval for the autonomous driving system to read the autonomous driving data is calculated based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel. The autonomous driving data is the interaction data between multiple subsystems of the autonomous driving vehicle and the autonomous driving system.

[0079] In step S530, the time interval for the autonomous driving system to read the autonomous driving data is updated to the optimal time interval, wherein the autonomous driving system realizes the autonomous driving of the autonomous vehicle based on the read sensor data.

[0080] Since the specific implementation of the sensor data optimization method has been detailed in the above device embodiments, the specific implementation of the sensor data optimization method described above can be found in the aforementioned sensor data optimization device embodiments, and will not be repeated here.

[0081] This disclosure also provides an electronic device, such as... Figure 6As shown, it includes a memory 620, a processor 610, and a program stored in the memory 620 and executable on the processor 610. When the program is executed by the processor 610, it can implement the various processes of the above-described sensor data optimization methods and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0082] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, this disclosure also provides a storage medium storing a computer program or instructions, which, when executed by a processor, can implement the various processes in the embodiments of the above-described sensor data optimization methods.

[0083] Since the instructions stored in the storage medium can execute the steps in the sensor data optimization method provided in the embodiments of this disclosure, the beneficial effects achievable by the sensor data optimization method provided in the embodiments of this disclosure can be realized, as detailed in the preceding embodiments, and will not be repeated here. The specific implementation of each of the above operations can be found in the preceding embodiments, and will not be repeated here.

[0084] In summary, according to the embodiments of this disclosure, the quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through a transmission channel are obtained. When the quality parameters of the sensor data do not meet preset requirements, the optimal time interval for the autonomous driving system to read the autonomous driving data is calculated based on the number of frames transmitted per second of the autonomous driving data read by the autonomous driving system through the transmission channel and the data transmission rate of the transmission channel. Then, the time interval for the autonomous driving system to read the autonomous driving data is updated to the optimal time interval. Here, the autonomous driving data is the interactive data between multiple subsystems of the autonomous vehicle and the autonomous driving system. Thus, the optimal time interval for the autonomous driving system to read the autonomous driving data ensures that sensor data is not delayed due to waiting for the autonomous driving system to read data, nor is data loss caused by the autonomous driving system reading data too frequently. This reduces the latency and frame loss rate of sensor data being received by appropriate modules, thereby improving the reliability of the autonomous driving performance of the autonomous vehicle. Furthermore, the optimal time interval for the autonomous driving system to read the autonomous driving data prevents the sensor data from increasing the computational resource burden on the computing device due to the autonomous driving system reading data too frequently, increasing the likelihood that the sensor data will have the necessary computational resources, thereby reducing the latency and frame loss rate of sensor data being received by appropriate modules, and improving the reliability of the autonomous driving performance of the autonomous vehicle.

[0085] Finally, it should be noted that the above embodiments are merely examples for clearly illustrating this disclosure and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of this disclosure.

Claims

1. A method for optimizing sensor data for autonomous vehicles, characterized in that, include: The quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through the transmission channel are obtained, and the quality parameters are determined based on the frame drop rate and / or latency rate. When the quality parameters of the sensor data do not meet the preset requirements, the sum of the products of the number of frames transmitted per second and the number of bytes per frame of the multiple autonomous driving data read by the autonomous driving system through the transmission channel is taken as the total number of bytes transmitted per second of the multiple autonomous driving data. The quotient of the total number of bytes transmitted per second of the multiple autonomous driving data and the maximum number of frames transmitted per second of the multiple autonomous driving data is taken as the optimal number of bytes that the autonomous driving system reads at one time. The quotient of the optimal number of bytes that the autonomous driving system reads at one time and the data transmission rate of the transmission channel is taken as the optimal time interval for the autonomous driving system to read the multiple autonomous driving data. The autonomous driving data is the interaction data between multiple subsystems of the autonomous vehicle and the autonomous driving system. The interaction data includes sensor data, call instructions and control instructions, and predicted trajectory information. The time interval for the autonomous driving system to read the autonomous driving data is updated to the optimal time interval, wherein the autonomous driving system realizes the autonomous driving of the autonomous vehicle based on the read sensor data.

2. The sensor data optimization method according to claim 1, characterized in that, The computing device of the autonomous vehicle includes one or more processors, the processors including one or more processor cores, and the sensor data optimization method further includes: The first thread that processes the sensor data on the computing device is bound to the first processor core among the one or more processor cores, so that the first thread is processed only by the first processor core.

3. The sensor data optimization method according to any one of claims 1 to 2, characterized in that, The multiple subsystems of the autonomous vehicle and the multiple threads of the autonomous driving system are processed by a computing device, and the sensor data optimization method further includes: Priorities are set for the multiple threads such that the first thread of the computing device that processes the sensor data has a higher priority than the other real-time threads; Based on a real-time preemptive scheduling algorithm, the computing device schedules the multiple threads according to their priorities.

4. The sensor data optimization method according to claim 3, characterized in that, After updating the time interval for the autonomous driving system to read the autonomous driving data to the optimal time interval, the real-time preemptive scheduling algorithm exits when the total number of bytes transmitted per second of the autonomous driving data is lower than the threshold.

5. A sensor data optimization device for autonomous vehicles, characterized in that, include: The quality parameter acquisition module is configured to acquire quality parameters of sensor data read by the autonomous driving system of an autonomous vehicle through a transmission channel, the quality parameters being determined based on frame drop rate and / or latency rate. The calculation module is configured to, when the quality parameters of the sensor data do not meet preset requirements, take the sum of the products of the number of frames transmitted per second and the number of bytes per frame of the multiple autonomous driving data read by the autonomous driving system through the transmission channel as the total number of bytes transmitted per second of the multiple autonomous driving data; take the quotient of the total number of bytes transmitted per second of the multiple autonomous driving data and the maximum number of frames transmitted per second of the multiple autonomous driving data as the optimal number of bytes the autonomous driving system reads at one time; and take the quotient of the optimal number of bytes read at one time of the autonomous driving system and the data transmission rate of the transmission channel as the optimal time interval for the autonomous driving system to read the multiple autonomous driving data. The autonomous driving data is the interaction data between multiple subsystems of the autonomous vehicle and the autonomous driving system, and the interaction data includes sensor data, call instructions and control instructions, and predicted trajectory information. An optimization module is configured to update the time interval at which the autonomous driving system reads the autonomous driving data to the optimal time interval, wherein the autonomous driving system enables the autonomous vehicle to drive automatically based on the read sensor data.

6. An autonomous vehicle, characterized in that, include: The sensor system is configured to collect sensor data; A sensor data optimization apparatus is configured to perform the method according to any one of claims 1 to 4; An autonomous driving system is configured to enable the autonomous vehicle to drive automatically based on the sensor data.

7. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 4.

8. A storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 4.

Images