Method, electronic device, medium and vehicle for reducing autonomous driving perception delay

Abstract

The present invention relates to the field of perception testing technology, and specifically provides a method, electronic device, medium, and vehicle for reducing the perception delay of autonomous driving, aiming to solve the problem of increased perception result delay caused by the overlap of data from different types of sensors in the perception reasoning module time slice. To this end, the method of reducing the perception delay of the present invention includes: acquiring the collected data of the sensor; determining whether the perception reasoning module is in an idle state; if so, controlling the perception reasoning module to process the collected data of the sensor; if not, waiting for the perception reasoning module to be idle until the perception reasoning module processes the collected data of the sensor, and adjusting the sampling time of the sensor to avoid the overlap of the time when the collected data of the sensor and the collected data of other sensors arrive at the perception reasoning module. In this way, the delay of the perception result is reduced, the perception efficiency is improved, and the stability and reliability of autonomous driving are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of perception testing technology, and specifically provides a method, electronic equipment, medium and vehicle for reducing autonomous driving perception delay. Background Art

[0002] Autonomous driving systems use a variety of sensors to perceive the surrounding environment, such as lidar, high-definition cameras, and millimeter-wave radar. The advantage of using multiple sensors is that they can fuse their results, overcoming the limitations of a single sensor's perception capabilities. However, increasing the number and variety of sensors also brings data processing challenges. Different sensor types have different sampling frequencies and independent sampling times.

[0003] Because data from different sensor types is generated independently and processed at different times, the time slices used by the perception and inference modules may overlap, increasing the latency of perception results. This can reduce the efficiency of the autonomous vehicle's perception of the target object.

[0004] Accordingly, this field requires a new solution to reduce the autonomous driving perception delay to solve the above problems. Summary of the Invention

[0005] To overcome these shortcomings, the present invention addresses, or at least partially addresses, the technical problem of increased perception latency caused by overlapping data from different sensor types within the perception and reasoning module's time slices. This invention provides a method, electronic device, medium, and vehicle for reducing autonomous driving perception latency.

[0006] In a first aspect, the present invention provides a method for reducing the perception delay of autonomous driving, comprising the following steps: obtaining collected data from a sensor; determining whether a perception reasoning module is in an idle state; if so, controlling the perception reasoning module to process the collected data from the sensor; if not, waiting for the perception reasoning module to be idle until the perception reasoning module processes the collected data from the sensor, and adjusting the sampling time of the sensor to avoid overlap in the time when the collected data from the sensor and the collected data from other sensors arrive at the perception reasoning module.

[0007] In one embodiment, adjusting the sampling time of the sensor includes: calculating the overlap time between the collected data of the sensor and the collected data of other sensors in the perception and reasoning module; and adjusting the sampling time of the sensor based on the overlap time.

[0008] In one embodiment, adjusting the sampling time of the sensor based on the overlap duration includes: setting an adjustment coefficient corresponding to the sampling time; and adjusting the sampling time of the sensor based on the adjustment coefficient.

[0009] In one embodiment, adjusting the sampling time of the sensor based on the adjustment coefficient includes: delaying the sampling time of the sensor based on the adjustment coefficient.

[0010] In one embodiment, the adjustment coefficient is set to 1 ms to 2 ms.

[0011] In one embodiment, the perception reasoning module is implemented based on a deep learning network.

[0012] In one embodiment, the data collected by the sensor is preprocessed.

[0013] In a second aspect, an electronic device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute any of the aforementioned methods for reducing the autonomous driving perception delay.

[0014] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute any of the aforementioned methods for reducing the autonomous driving perception delay.

[0015] In a fourth aspect, a vehicle is provided, comprising the aforementioned electronic device.

[0016] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0017] The present invention provides a method for reducing the perception delay of autonomous driving. First, the sensor's collected data is obtained, and then the perception reasoning module is determined to be in an idle state. If so, the perception reasoning module is controlled to process the sensor's collected data. If not, the perception reasoning module is idle until it can process the sensor's collected data, and the sampling time of the sensor is adjusted. In this way, the overlap of the time when the sensor's collected data and the collected data of other sensors arrive at the perception reasoning module is avoided. The collected data of each sensor can be directly processed after reaching the perception reasoning module, thereby avoiding the queuing time or competition time for the collected data of each sensor to reach the perception reasoning module, thereby reducing the delay of the perception result. Furthermore, compared with autonomous driving vehicles, the perception efficiency is further improved, and the stability and reliability of autonomous driving are guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The disclosure of the present invention will be more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Furthermore, similar numbers in the drawings represent similar components, wherein:

[0019] Figure 1a 1 is a schematic diagram of time slices in which data collected by a lidar and a camera respectively arrive at a perception and reasoning module according to an embodiment of the present invention;

[0020] Figure 1b 1 is a schematic diagram of time slices in which data collected by a lidar and a camera respectively arrive at a perception and reasoning module according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of the main steps of a method for reducing autonomous driving perception latency according to one embodiment of the present invention;

[0022] Figure 3 This is a complete flowchart of a method for reducing autonomous driving perception latency according to an embodiment of the present invention;

[0023] Figure 4 FIG. 1 is a schematic diagram of the main structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0025] In the description of the present invention, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, and the like. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "one" and "the" may also include the plural forms.

[0026] Currently, autonomous driving systems utilize a variety of sensors to perceive the surrounding environment, such as lidar, high-definition cameras, and millimeter-wave radar. The advantage of using multiple sensors is that they can fuse their results, overcoming the limitations of a single sensor's perception capabilities. However, increasing the number and variety of sensors also presents data processing challenges. Different sensor types have different sampling frequencies, but their sampling times are independent of each other, so the processing time for each frame of data from different sensor types is also independent.

[0027] The perception and inference module (a high-performance computing unit such as a GPU or NPU) in an autonomous driving system is the primary computing power module. If the time slices in which different types of sensors collect data and request the perception and inference module are staggered, contention can be avoided and the latency of perception results can be minimized. However, in reality, as described in the previous paragraph, these are independent. When the time slices in which different types of sensors request the inference module overlap, contention arises.

[0028] For example, it is assumed that two types of sensors are installed on the vehicle, such as a laser radar and an onboard high-definition camera. Figure 1a and Figure 1b As shown in the figure, the large squares represent the time segments occupied by the vehicle-mounted high-definition camera in the perception and reasoning module, and the small squares represent the time segments occupied by the lidar in the perception and reasoning module.

[0029] When the two blocks don't overlap on the timeline, the perception and reasoning module processes the data collected by the two sensors separately. When overlap occurs, whether the perception and reasoning module uses a competition mechanism or a queuing mechanism for the data collected by the two sensors, this will increase the overall processing time of the perception and reasoning module, thereby increasing the latency of the perception results.

[0030] Generally speaking, the sampling frequency of the sensor will fluctuate within a small range. For example, the sampling frequency of the car HD camera is 30hz, but in reality it is distributed in the range of 28hz-30hz. The sampling frequency of the LiDAR marker also fluctuates. Because of this frequency fluctuation, such as Figure 1b As shown, if we zoom in a bit, to the 10-second level, we see a noticeable change in the overlap between the two blocks on the time axis, perhaps from no overlap to overlap. Further zooming in to the minute level reveals large fluctuations in latency, which can be periodic or acyclic, depending on the frequency of the sensor during the observation period.

[0031] Because data from different sensor types is generated independently, data preprocessing time also varies. This can lead to overlapping time slices in the perception and inference modules, increasing latency in perception results. This can reduce the efficiency of the autonomous vehicle's perception of the target object.

[0032] To this end, the present application proposes a method, electronic device, medium, and vehicle for reducing the perception latency of autonomous driving. The method first obtains the sensor's collected data, then determines whether the perception reasoning module is in an idle state. If so, the perception reasoning module is controlled to process the sensor's collected data. If not, the method waits until the perception reasoning module is idle until it can process the sensor's collected data, and adjusts the sensor's sampling time. In this way, the overlap of the sensor's collected data and the collected data of other sensors arriving at the perception reasoning module is avoided. The collected data of each sensor can be directly processed after reaching the perception reasoning module, thereby avoiding the queuing time for the collected data of each sensor to arrive at the perception reasoning module, thereby reducing the delay of the perception result. Furthermore, compared with autonomous driving vehicles, the perception efficiency is further improved, ensuring the stability and reliability of autonomous driving.

[0033] See attached Figure 2 , Figure 2 This is a flowchart of the main steps of a method for reducing autonomous driving perception latency according to an embodiment of the present invention.

[0034] like Figure 2 As shown, the method for reducing the autonomous driving perception delay in the embodiment of the present invention mainly includes the following steps S101-S102.

[0035] Step S101: Acquire data collected by the sensor.

[0036] Specifically, the sensor can be a lidar, a high-definition camera, a millimeter-wave radar, a vehicle-mounted camera, etc., but is not limited to these.

[0037] Step S102: Determine whether the perception and reasoning module is in an idle state; if so, control the perception and reasoning module to process the collected data of the sensor; if not, wait for the perception and reasoning module to be idle until the perception and reasoning module can process the collected data of the sensor, and adjust the sampling time of the sensor to avoid the time overlap between the collected data of the sensor and the collected data of other sensors arriving at the perception and reasoning module.

[0038] The perception and reasoning module may be a module that performs feature extraction on data collected by multiple sensors, but is not limited thereto.

[0039] For example, the perception and reasoning module is illustrated by taking laser radar, photoelectric radar and vehicle-mounted high-definition camera as examples.

[0040] First, obtain the point cloud data collected by the lidar and the millimeter wave radar, as well as the images captured by the on-board high-definition camera.

[0041] The perception and reasoning module processes point cloud data collected by lidar to derive point cloud features of objects. It also processes point cloud data collected by millimeter-wave radar to derive point cloud features. Furthermore, the perception and reasoning module recognizes images captured by onboard high-definition cameras to derive corresponding object features. Therefore, the perception and reasoning module processes or recognizes data collected by multiple sensors to derive corresponding object features.

[0042] In a specific embodiment, the perception reasoning module is implemented based on a deep learning network.

[0043] Exemplarily, the perception and reasoning module may include a deep learning network for feature extraction. Based on the deep learning network, the perception and reasoning module may perform feature extraction on point cloud data collected by a lidar or a millimeter-wave radar, thereby obtaining corresponding features.

[0044] Alternatively, the perception and reasoning module can be a deep learning network that includes object recognition. In this case, the perception and reasoning module can recognize the image captured by the vehicle's high-definition camera to obtain a recognition result.

[0045] After acquiring the sensor data in step S101, the data needs to be processed by the perception and reasoning module. Therefore, it is necessary to determine whether the perception and reasoning module is currently idle. If it is idle, the perception and reasoning module directly processes the sensor data. If it is not idle, meaning the perception and reasoning module is currently processing data from other sensors, the sensor data must wait until the perception and reasoning module is idle and can process the sensor data.

[0046] Meanwhile, when the perception and reasoning module is processing data from other sensors, it adjusts the sensor's sampling time to prevent overlap between the sensor's data and other sensors' arrival times. This adjustment eliminates overlap and allows the module to process incoming sensor data sequentially, reducing latency in perception results.

[0047] In a specific embodiment, adjusting the sampling time of the sensor includes: calculating the overlap time between the collected data of the sensor and the collected data of other sensors in the perception and reasoning module; and adjusting the sampling time of the sensor based on the overlap time.

[0048] Specifically, because the perception and reasoning module is currently processing data from other sensors, the data collected by the sensor needs to wait for a period of time before it can be processed by the perception and reasoning module. This waiting period is the overlap duration. This application mainly adjusts the sensor sampling time based on the calculated overlap duration.

[0049] In this way, it can be ensured that within a relatively short period of time, the time when the collected data of the sensor reaches the perception and reasoning module is staggered with the time when the collected data of other sensors reaches the perception and reasoning module, so that the data collected by the sensor can be quickly processed when it reaches the perception and reasoning module, thereby improving the perception efficiency.

[0050] In a specific embodiment, adjusting the sampling time of the sensor based on the overlap duration includes: setting an adjustment coefficient corresponding to the sampling time; and adjusting the sampling time of the sensor based on the adjustment coefficient.

[0051] In one case, the time it takes for the collected data from a sensor to reach the perception and reasoning module overlaps with the time it takes for the collected data from other sensors to reach the perception and reasoning module for a long time. If the sampling time of the sensor is adjusted in one step, it may cause large fluctuations and affect the stability performance of the vehicle.

[0052] To this end, the present application adopts a fine-tuning method to adjust the sensor sampling time. Although the amplitude of the fine-tuning is small, the sampling frequency of the sensor is very high, so it can be adjusted quickly.

[0053] Specifically, the sensor sampling time is fine-tuned by setting the fine-tuning coefficient.

[0054] In a specific implementation, the adjustment coefficient is set to 1 ms to 2 ms.

[0055] Preferably, the adjustment coefficient in this embodiment can be set to 1 ms, that is, the sampling time of the sensor is adjusted by 1 ms.

[0056] In a specific embodiment, adjusting the sampling time of the sensor based on the adjustment coefficient includes: delaying the sampling time of the sensor based on the adjustment coefficient.

[0057] Specifically, the sensor sampling time is adjusted by delaying it. By setting the adjustment coefficient to 1ms, the sensor sampling time can be delayed by 1ms at a time. This delay, over multiple cycles, prevents the sensor's collected data from overlapping with that of other sensors before reaching the perception and inference module, thereby improving perception efficiency.

[0058] For example, the time segment of sensor A and sensor B in the perception and reasoning module has an overlap of 10ms, and the perception and reasoning module is processing the collected data of sensor B.

[0059] Since the adjustment coefficient is 1ms, the sampling time of sensor A needs to be delayed by 1ms each time. After a delay of 10 cycles, the data collected by sensor A and sensor B can be prevented from overlapping when reaching the perception and inference module.

[0060] It is clear to those skilled in the art that when the overlapping time of the collected data of a sensor and the collected data of other sensors in the perception and reasoning module is less than the adjustment coefficient, after one or one cycle of adjustment, the time overlap of the collected data of the two sensors arriving at the perception and reasoning module can be avoided.

[0061] Based on the above steps S101 and S102, the sensor's collected data is first acquired, and then it is determined whether the perception and reasoning module is in an idle state. If so, the perception and reasoning module is controlled to process the sensor's collected data. If not, the perception and reasoning module waits until it is idle until it can process the sensor's collected data, and the sensor's sampling time is adjusted. In this way, the overlap of the time when the sensor's collected data and the collected data of other sensors arrive at the perception and reasoning module is avoided. The collected data of each sensor can be directly processed after reaching the perception and reasoning module, thereby avoiding the queuing time for the collected data of each sensor to arrive at the perception and reasoning module, and thus avoiding the perception result delay caused by competition between different types of sensors. Furthermore, compared with autonomous vehicles, the perception efficiency is further improved, ensuring the stability and reliability of autonomous driving.

[0062] In a specific embodiment, the method further includes: pre-processing the data collected by the sensor.

[0063] After acquiring sensor data, it can be preprocessed. For example, data collected by sensors like lidar and millimeter-wave radar can be subjected to noise removal or filtering. Images or videos captured by onboard high-definition cameras can be subjected to image alignment and other operations. Preemptively removing noise from the data or performing image alignment can further improve perception accuracy.

[0064] like Figure 3 As shown, taking two sensors as an example, the method of the present application is described in detail.

[0065] First, the collected data from sensor A is acquired, and then preprocessed to remove noise from the data or align the image. Next, it is determined whether the perception and reasoning module is occupied by the collected data from sensor B. If not, the perception and reasoning module directly processes the preprocessed collected data from sensor A. If the perception and reasoning module is occupied at this time, the collected data from sensor A needs to wait until the perception and reasoning module is idle before it can process the collected data from sensor A. When the perception and reasoning module is occupied, the sampling time of sensor A is adjusted to avoid the overlap of the time when the collected data from sensor A and sensor B arrive at the perception and reasoning module, thereby improving perception efficiency and ensuring the stability and reliability of the autonomous driving vehicle.

[0066] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0067] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above-mentioned embodiment of the present invention may also be accomplished by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, it may implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal, and software distribution medium capable of carrying the computer program code. It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0068] Furthermore, the present invention also provides an electronic device. Figure 4As shown, in an electronic device embodiment according to the present invention, the electronic device includes a processor 40 and a storage device 41. The storage device can be configured to store a program for executing the method for reducing the autonomous driving perception latency of the above-mentioned method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the method for reducing the autonomous driving perception latency of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present invention.

[0069] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the method for reducing the autonomous driving perception delay of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned method for reducing the autonomous driving perception delay. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-transitory computer-readable storage medium.

[0070] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for reducing autonomous driving perception latency, characterized in that: The steps include: Get the collected data from the sensor; Determine whether the perception and reasoning module is in an idle state; If so, controlling the perception and reasoning module to process the collected data of the sensor; If not, wait for the perception and reasoning module to be idle until the perception and reasoning module processes the collected data of the sensor, and adjust the sampling time of the sensor to avoid the time when the collected data of the sensor and the collected data of other sensors arrive at the perception and reasoning module overlapping; Adjusting the sampling time of the sensor includes: Calculating the overlapping time of the collected data of the sensor and the collected data of the other sensors in the perception and reasoning module; adjusting a sampling time of the sensor based on the overlap duration; Adjusting the sampling time of the sensor based on the overlap duration includes: Set the adjustment coefficient corresponding to the sampling time; A sampling time of the sensor is adjusted based on the adjustment factor.

2. The method for reducing autonomous driving perception delay according to claim 1, characterized in that: Adjusting the sampling time of the sensor based on the adjustment coefficient includes: delaying the sampling time of the sensor based on the adjustment coefficient.

3. The method for reducing autonomous driving perception delay according to claim 1 or 2, characterized in that: The adjustment coefficient is set to 1ms-2ms.

4. The method for reducing autonomous driving perception latency according to claim 1, characterized in that: The perception and reasoning module is implemented based on a deep learning network.

5. The method for reducing autonomous driving perception delay according to claim 1, characterized in that: Also includes: The collected data of the sensor is preprocessed.

6. An electronic device comprising a processor and a storage device, wherein the storage device is adapted to store a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the method for reducing autonomous driving perception latency according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the method for reducing autonomous driving perception latency according to any one of claims 1 to 5.

8. A vehicle, characterized in that: The electronic device comprising claim 6.

Images