Method and system for dynamically optimizing sensing model reasoning speed based on road risk

Abstract

The invention discloses a method for reasoning speed based on a road risk dynamic optimization perception model, and the method comprises the steps: carrying out the lightweight processing and pre-construction of environment perception models corresponding to different operation speeds according to different risk levels of a road under automatic driving; road environment information is collected, a road environment risk value is calculated in real time based on the road environment information, and the current road environment safety level is determined according to the range of the road risk value; and based on the environmental safety level, selecting environmental perception models corresponding to different running speeds, outputting perception results, and dynamically adjusting the upper limit of the running speed to realize intelligent regulation and control of the running speed of the automatic driving perception system. According to the invention, sensing models of different operation speeds can be dynamically selected according to the road risk level and the upper limit of the planned speed is adjusted, thereby realizing intelligent regulation and control of the operation speed of the sensing system and ensuring automatic driving safety.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method and system thereof for dynamically optimizing the reasoning speed of a perception model based on road risks. Background Art

[0002] Autonomous driving has extremely high requirements for the real-time and accuracy of the perception system. The perception model needs to complete data processing and analysis in a very short time. Existing autonomous driving systems mostly use perception models based on deep learning to cope with complex and changing road environments. However, such models have high computational complexity and slow reasoning speed, which greatly limits the optimization space of autonomous driving systems.

[0003] At present, the mainstream autonomous driving environment perception model is mainly based on BEV (Bird's Eye View) and Transformer to build an environment perception system. However, the use of such large models also leads to problems such as high model calculation complexity and excessive memory usage. These problems will reduce the operating efficiency of the perception system and the real-time performance of the system, thereby increasing the safety risks of autonomous driving vehicles.

[0004] To improve model operation efficiency and enhance system real-time performance, the current mainstream methods use solutions such as model lightweighting and cloud-edge collaboration:

[0005] (1) Model lightweighting methods, such as pruning, quantization, or knowledge distillation. However, methods based on model lightweighting have disadvantages such as accuracy loss, limited generalization ability, and dependence on specific hardware.

[0006] (2) Cloud-edge collaboration is an architectural model that combines cloud computing and edge computing. However, cloud-edge collaboration also has problems such as limited usage environment, resource constraints, management and maintenance complexity, network latency, and insufficient stability.

[0007] In summary, the use of perception systems based on large models can provide accurate road information, but is limited by slow inference speed. Existing model speed-up solutions have problems such as reduced accuracy, poor generalization performance, and environmental restrictions.

[0008] In order to improve the real-time response capability of the autonomous driving system, it is urgent to propose a method to improve the reasoning speed of autonomous driving to address the technical problems existing in the above methods. The method can dynamically select the original large perception model or the lightweight perception model according to the current road environment risk level, and can dynamically adjust the speed limit of the autonomous driving vehicle according to the changes in road risks. This not only improves the safety of the autonomous driving vehicle, but also reserves sufficient adjustment time for the perception model. Summary of the invention

[0009] In order to solve the technical problems of high model calculation complexity and excessive memory usage in the above-mentioned prior art, a method and system for dynamically optimizing the reasoning speed of the perception model based on road risk is realized.

[0010] In a first aspect, an embodiment of the present application provides a method for dynamically optimizing the reasoning speed of a perception model based on road risk, the method comprising:

[0011] Environmental perception model pre-construction step: According to the different risk levels of roads under autonomous driving, lightweight processing pre-constructs environmental perception models corresponding to different computing speeds;

[0012] Environmental early warning step: collect road environment information, calculate the road environment risk value in real time based on the road environment information, and determine the current road environment safety level according to the range of the road risk value;

[0013] System risk control steps: Based on the environmental safety level, select the environmental perception model corresponding to different operating speeds, output the perception results, and dynamically adjust the upper limit of driving speed to achieve intelligent regulation of the operating speed of the autonomous driving perception system.

[0014] Optionally, the above-mentioned environmental perception model in the embodiment of the present invention includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of the input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

[0015] Optionally, the above-mentioned environment perception model pre-construction step in the embodiment of the present invention further includes:

[0016] Preliminary feature extraction step: the collected image is adjusted to preset pixels, image dimension reduction is performed through two-dimensional convolution, and the number of channels is increased to extract preliminary feature maps;

[0017] Deep semantic feature extraction steps: According to the preliminary feature map, position encoding is added to retain the spatial position information of the elements in the image, the preliminary feature map is sent to a sequence composed of multiple Transformer modules, the dimension of the preliminary feature map is adjusted to a suitable scale, and the road environment risk information is extracted.

[0018] Optionally, the above-mentioned environmental early warning step in the embodiment of the present invention includes:

[0019] Obstacle determination step: According to the road environment risk information, the obstacles on the current road are identified through the detection algorithm, and the image coordinates of the obstacles are obtained. The relative distance of the obstacles is obtained according to the external parameters between the image coordinates and the radar sensor, and the obstacle closest to the vehicle is determined;

[0020] Collision time calculation steps: obtain the relative speed between the obstacle target and the vehicle through inertial navigation and radar, and calculate the inverse value of the collision time of the target closest to the vehicle based on the relative distance and relative speed.

[0021] Optionally, the system risk control step in the embodiment of the present invention includes:

[0022] High-speed perception model decision-making steps: When the calculated inverse value of the collision time is less than or equal to the first risk threshold, the current road environment is judged to be relatively safe and is regarded as a non-hazardous road environment, and the high-speed perception model is selected accordingly to output the perception result;

[0023] Medium-speed perception model decision-making steps: When the calculated inverse value of the collision time is greater than the first risk threshold and less than or equal to the second risk threshold, it is judged that the current road environment risk is increased, and the medium-speed perception model is selected accordingly to output the perception result;

[0024] Low-speed perception model decision steps: When the calculated inverse value of the collision time is greater than the second risk threshold and less than or equal to the third risk threshold, it indicates that the current environment requires the algorithm to treat it with caution, and the low-speed perception model is selected accordingly to output the perception result.

[0025] Optionally, the system risk control step in the embodiment of the present invention further includes:

[0026] Speed ​​adjustment steps: When the risk level increases, based on the speed limit factor, the upper limit of driving speed is limited to ensure user safety; the speed limit factor k 限速 It is calculated by the following function:

[0027]

[0028] Among them, TTC -1 It is the inverse value of the collision time. When the risk increases, the speed limit factor increases dynamically.

[0029] Optionally, the upper limit of the driving speed in the embodiment of the present invention is:

[0030]

[0031] in, is the maximum speed limit of the current driving speed, V max Plan the speed limit for the vehicle's autonomous driving state, k 限速 is the speed limiting factor.

[0032] In a second aspect, an embodiment of the present application provides a system for dynamically optimizing the reasoning speed of a perception model based on road risk, using the method for dynamically optimizing the reasoning speed of a perception model based on road risk as described above, the system includes:

[0033] Environmental perception model pre-construction module: used to pre-construct environmental perception models corresponding to different computing speeds based on different risk levels of roads under autonomous driving.

[0034] Environmental warning module: used to collect road environment information, calculate the road environment risk value in real time based on the road environment information, and determine the current road environment safety level according to the range of road risk value;

[0035] System risk control module: used to select the environmental perception model corresponding to different operating speeds based on the environmental safety level, output the perception results, and dynamically adjust the upper limit of driving speed to achieve intelligent control of the operating speed of the autonomous driving perception system;

[0036] The environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of the input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

[0037] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method steps for dynamically optimizing the reasoning speed of the perception model based on road risk.

[0038] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method steps for dynamically optimizing the reasoning speed of the perception model based on road risk as described above are implemented.

[0039] In a fifth aspect, an embodiment of the present application provides an autonomous driving vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risks as described above are implemented.

[0040] Compared with the related prior art, it has the following outstanding beneficial effects:

[0041] 1. The method of the present invention can effectively improve the driving speed of autonomous vehicles by dynamically balancing road environment risks and reasoning speed while meeting system requirements. At the same time, the safety of the system is enhanced through real-time monitoring and response of the early warning module;

[0042] 2. The method of the present invention adopts model lightweight technology as an alternative to the original perception model, avoiding the problem of large differences in generalization performance of perception models with different structures;

[0043] 3. The introduction of the warning module of the method of the present invention enables the system to intelligently adjust the perception speed according to different road conditions, improving the flexibility and adaptability of the system. When facing complex and changing traffic environments, the system can respond quickly to ensure driving safety;

[0044] 4. The proposed method of the present invention provides a new idea and method for the field of accelerating the algorithm of autonomous driving, and promotes the innovation and development of related technologies. In the future, with the continuous advancement and improvement of technology, the application scope and effect of the present invention will be further expanded and improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A schematic diagram of a method for dynamically optimizing the reasoning speed of a perception model based on road risk according to an embodiment of the present invention;

[0047] Figure 2 This is a diagram showing the structure of the model after the SAM model of the present invention is lightened to different degrees;

[0048] Figure 3 This is a flow chart of a method for optimizing the perception and reasoning speed of autonomous driving according to another embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of a speed inference system based on a road risk dynamic optimization perception model according to an implementation example of the present invention;

[0050] Figure 5 A schematic diagram of the system execution flow of an embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of an electronic device shown in an embodiment of the present invention;

[0052] Figure 7 A top view of a vehicle body sensor device for dynamically optimizing perception speed based on road environment risk, provided as an implementation example of the present invention.

[0053] In the above picture:

[0054] 301 Environmental Perception Model Pre-construction Module 302 Environmental Warning Module

[0055] 303 system risk control module. DETAILED DESCRIPTION

[0056] It should be noted that the processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0057] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0058] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0059] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0060] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0061] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0062] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0063] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0064] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0066] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0068] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0069] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0070] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0071] The present invention aims to propose a method and system for dynamically optimizing the reasoning speed of the perception model based on the road environment risk level, introduce an environmental warning module to determine the risk level, and combine the model lightweight technology to lightweight the original large model. The system can dynamically select perception models with different operating speeds according to the road risk level and adjust the planned speed limit, thereby realizing intelligent control of the operating speed of the perception system, ensuring the safety of vehicles and improving the operating efficiency of the vehicle perception system.

[0072] The method of the present invention dynamically adjusts the reasoning speed of the perception algorithm according to the road risk level, thereby improving the real-time performance of the algorithm. At the same time, the method dynamically adjusts the upper limit of the planned speed according to the road risk level, thereby not only improving driving safety, but also further reserving time for algorithm optimization.

[0073] Embodiment 1

[0074] like Figure 1 As shown, the embodiment of the present application provides a method for dynamically optimizing the reasoning speed of the perception model based on road risk, and the method includes:

[0075] Environmental perception model pre-construction step 101: according to different risk levels of roads under autonomous driving, lightweight processing pre-constructs environmental perception models corresponding to different computing speeds; optionally, in the embodiment of the present invention, the environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features;

[0076] Environmental warning step 102: collecting road environment information, and calculating the road environment risk value in real time based on the road environment information, and determining the current road environment safety level according to the range of the road risk value;

[0077] System risk control step 103: Based on the environmental safety level, select the environmental perception model corresponding to different operating speeds, output the perception results, and dynamically adjust the upper limit of the driving speed to achieve intelligent reasoning and regulation of the operating speed of the autonomous driving perception system.

[0078] In a specific embodiment of the present invention, in order to improve the real-time response capability of the autonomous driving system, the present invention combines the advantages of high precision of the original large model and fast speed of the lightweight model, and dynamically selects the original perception large model or the lightweight perception model according to the current road environment risk level. For example, when the road environment risk level is low, a small and fast (lightweighted) perception model is used, and when the road environment risk level is high, a slow and accurate (non-lightweighted) perception model is used. In addition, the present invention can also dynamically adjust the upper speed limit of autonomous driving according to changes in road risks, which not only improves the safety of autonomous driving, but also reserves sufficient adjustment time for the perception model.

[0079] As an optional implementation, Figure 2 As shown, the above-mentioned environmental perception model adopts the SAM large model to construct the environmental perception model, and the environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of the input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

[0080] Among them, in order to further describe the perception system, in the specific embodiment of the present invention, the image encoder in SAM (Segment Anything Model) is used as an example to construct a perception model, but the present invention is not limited to this. The SAM large model has powerful zero-sample and few-sample generalization capabilities, and can segment any object in the image, including objects and scenes that have not been encountered during the training process. This ability is particularly important in autonomous driving, because autonomous driving needs to deal with various complex and changeable road environments and traffic conditions. SAM's precise segmentation and powerful generalization capabilities help autonomous driving to more accurately identify potential dangerous elements in the surrounding environment, thereby improving the safety and reliability of autonomous driving.

[0081] The SAM large model image encoder uses ViT (Vision Transformer) as the basic architecture, which performs well in processing high-resolution images. However, the present invention is not limited to this, and other types of basic architectures can also be used.

[0082] In a specific embodiment of the present invention, in order to further optimize computing resources, the SAM model is split into a coarse feature extraction network and a deep semantic feature extraction network. The data from the sensor is first sent to the coarse feature extraction network to obtain preliminary features, and then the preliminary features are sent to the deep semantic feature extraction network to obtain the results of environmental perception. Since the above-mentioned deep semantic feature network has a high computational complexity, the deep semantic feature network is subjected to model lightweight processing, that is, the environmental perception model includes: a coarse feature extraction network model and a plurality of deep semantic feature network models with different lightweight levels. The shared parameters of the coarse feature extraction network model are fixed to ensure that the input features used by different lightweight deep semantic feature network models are consistent.

[0083] In a specific embodiment of the present invention, in order to reduce the occupation of hardware resources, the present invention divides the overall perception model into two parts according to the structural characteristics of SAM. The perception model structure splitting diagram is shown in FIG. Figure 2 As shown, the convolution model is used as the first part of the perception model to extract rough features of the data. In the specific embodiment of the present invention, the perception operation speed is improved mainly by lightweighting the more time-consuming second part of the model. The second part networks of different lightweight levels share the first part of the model to extract rough features of the data. In the specific implementation example of the present invention, the SAM perception large model can be lightweighted into a low-speed high-precision perception model, a medium-speed medium-precision perception model, and a high-speed low-precision perception model, but the present invention is not limited to this, and the lightweight SAM perception model of the second part can also be divided into other types of models.

[0084] Embodiment 2

[0085] like Figure 3 As shown, another embodiment of the present application provides a method for dynamically optimizing the reasoning speed of the perception model based on road risk, the method comprising:

[0086] Environment perception model pre-construction step 201: according to different risk levels of roads under autonomous driving, lightweight processing pre-constructs environment perception models corresponding to different computing speeds;

[0087] The above-mentioned environment perception step 201 also includes:

[0088] Preliminary feature extraction step 2011: the collected image is adjusted to preset pixels, image dimension reduction is performed through two-dimensional convolution, and the number of channels is increased to extract a preliminary feature map;

[0089] Deep semantic extraction step 2012: According to the preliminary feature map, position encoding is added to retain the spatial position information of the elements in the image, and the preliminary feature map is sent to a deep network composed of multiple Transformer modules to adjust the dimension of the preliminary feature map to a suitable scale.

[0090] In a specific embodiment of the present invention, the image size collected by the camera is first adjusted to 1024x1024 pixels, and then downsampled through a two-dimensional convolution with a convolution kernel of 16 and a step size of 16 to reduce the image size to 64x64, and at the same time increase the number of channels to extract preliminary feature representation, that is, the feature map after dimensionality reduction.

[0091] After obtaining the 64x64 feature map, the model will add position encoding to retain the spatial position information of the elements in the image. Then, these feature maps are sent to a sequence of multiple Transformer modules. Each Transformer module contains a multi-head attention mechanism and residual connection. Finally, the feature map is adjusted to its dimension through multiple Transformer modules to meet the needs of subsequent tasks.

[0092] In the above feature extraction process, step 2012 is usually time-consuming. This is because the self-attention mechanism in the Transformer model needs to perform complex matrix operations when processing a large amount of data, especially when the image is divided into multiple small blocks, each small block needs to interact with all other small blocks to calculate the attention weights between them. Usually the computational complexity of the self-attention mechanism is proportional to the square of the input sequence length (i.e., the number of small blocks). Therefore, with the increase of image resolution and the increase of the number of divided small blocks, the computational complexity of the self-attention mechanism will increase sharply. The self-attention mechanism in the Transformer module is intensive calculation, and it is difficult to significantly reduce its computational complexity through simple optimization means. In order to reduce the occupation of hardware resources, the present invention divides the overall perception model into two parts according to the structural characteristics of SAM, and the preliminary feature extraction model in step 2011 is used as the first part of the perception model to extract rough features of the data. The deep semantic extraction module described in step 2012 is the second part, and the embodiment of the present invention mainly improves the perception operation speed by lightweighting the more time-consuming second part deep semantic feature extraction network. The second part networks of different lightweight levels share the rough features extracted by the preliminary feature extraction network. For example, the original SAM perception model can be lightly and highly lightweighted to obtain medium-speed and high-speed perception models. The perception model structure decomposition diagram is as follows: Figure 2 shown.

[0093] For the lightweight solution of the second part of the model, the specific embodiment of the present invention refers to the lightweight ideas of MobileSam and MobileSamV2. The ViT-H image encoder in the original SAM is replaced with a lightweight Tiny-ViT encoder through the decoupling distillation method, thereby achieving a 60-fold reduction in model volume and a 40-fold increase in running speed, while maintaining good segmentation performance. MobileSAMv2 further optimizes the mask decoder based on MobileSAM, and reduces a large number of redundant grid searches and subsequent filtering steps by directly generating effective masks. MobileSAMv2 reduces the processing time of the mask decoder by at least 16 times compared to MobileSam.

[0094] Environmental warning step 202: collecting road environment information, and calculating the road environment risk value in real time based on the road environment information, and determining the current road environment safety level according to the range of the road risk value;

[0095] Environmental warning is mainly used to analyze the safety of the current driving environment. At present, the methods used to evaluate road safety are mainly divided into three categories, namely, target-oriented evaluation methods, reachable set-based evaluation methods, and potential field theory-based evaluation methods. The target-oriented evaluation method refers to the evaluation of risk based on the degree of proximity to the future trajectory of the target in time and space. Common evaluation indicators mainly include TTC (collision time), THW (headway time), PET (post-occupancy time), TLC (lane crossing time), PSD (braking distance ratio), etc. The evaluation method based on potential field theory draws on the potential field theory in physics and uses the repulsive field to describe the collision risk. The evaluation method based on reachable sets calculates the set of states that the vehicle can reach under certain constraints at each time step, and uses the size of the reachable set to characterize the risk.

[0096] The present invention uses TTC to measure the degree of hidden dangers in the current environment. TTC refers to the time required for a vehicle to reach the nearest obstacle (such as another vehicle, pedestrian, bicycle, etc.) at the current speed to cause a collision. TTC is widely used in automobile safety assessment. The smaller its value, the greater the possibility of a collision in the current state. The expression is as follows:

[0097]

[0098] Among them, Δdis is the relative distance between the vehicle and the obstacle, and Δv is the speed of the obstacle relative to the vehicle. If Δv is a positive number, it means that the distance between the vehicle and the obstacle is increasing, so it is not considered; if Δv is a negative number, it means that the vehicle and the obstacle are approaching each other. After using the absolute value, TTC is still a positive number, indicating the remaining time before the collision.

[0099] The above-mentioned environmental early warning step 202 includes:

[0100] Obstacle determination step 221: identifying obstacles on the current road through a detection algorithm, and obtaining image coordinates of the obstacles, obtaining the relative distance of the obstacles based on the image coordinates and the external parameters between the radar sensor, and determining the obstacle closest to the vehicle;

[0101] Collision time calculation step 2022: The relative speed between the obstacle target and the vehicle is obtained through inertial navigation and radar, and the inverse value of the collision time of the target closest to the vehicle is calculated based on the relative distance and relative speed.

[0102] To enhance its sensitivity to obstacle distance, many researchers use TTC -1 (TTC inverse) is used as the basis for judgment. Existing literature generally regards TTC less than 1.5s as a dangerous situation of vehicle near collision. Therefore, in the specific embodiment of the present invention, TTC is -1 Equal to 0.7 as the boundary between the near collision and the safety scene, that is, the target TTC -1 When ≥0.7, the current driving environment is considered to be extremely risky. -1 <0.7 is considered a safe driving scenario and TTC is calculated -1 The main steps are as follows:

[0103] First, the obstacles on the current road (including but not limited to vehicles, pedestrians, fallen rocks, etc.) are identified through a fast detection algorithm, and the image coordinates of the obstacles are obtained. The relative distance Δdis of the target object is obtained based on the external parameters between the image coordinates and the radar sensor, and the obstacle closest to the vehicle is determined.

[0104] The relative speed Δv between the target and the vehicle is obtained through inertial navigation and radar, and the TTC of the target closest to the vehicle is calculated. -1 value.

[0105] In order to quickly detect road targets, the present invention uses YOLOv8 as a detection algorithm. The YOLOv8 target detection algorithm performs well in terms of reasoning speed, accuracy, training convenience, and model design. It is one of the most popular and successful target detection algorithms in the industry. In practical applications, the detection algorithm can be selected in combination with actual conditions, and the present invention is not limited to this. However, it is necessary to ensure that the computing power burden caused by the detection algorithm should be less than the computing power resources saved by the lightweight model. Considering that the feature extraction steps of YOLOv8 and SAM models are similar, YOLOv8 and the deep semantic feature extraction network share the image input layer to extract preliminary features.

[0106] System risk control step 203: Based on the environmental safety level, select the environmental perception model corresponding to different operating speeds, output the perception results, and dynamically adjust the upper limit of the driving speed to achieve intelligent regulation of the operating speed of the autonomous driving perception system.

[0107] The above-mentioned system risk control step 203 includes:

[0108] Low-speed perception model decision step 2031: When the calculated inverse value of the collision time is greater than the second risk threshold and less than or equal to the third risk threshold, it indicates that the current environment requires the algorithm to be cautious, and the low-speed perception model is selected accordingly, and the perception result is output;

[0109] Medium-speed perception model decision step 2032: When the calculated inverse value of the collision time is greater than the first risk threshold and less than or equal to the second risk threshold, it is determined that the current road environment risk is lower than 2031, and the medium-speed perception model is selected accordingly, and the perception result is output;

[0110] High-speed perception model decision step 2033: When the calculated inverse value of the collision time is less than or equal to the first risk threshold, the current road environment is judged to be relatively safe and is regarded as a non-hazardous road environment. The high-speed perception model is selected accordingly and the perception result is output.

[0111] Receive the calculated output value TTC -1 , according to TTC -1 Road risks are divided into different levels. In order to further improve the safety of the autonomous driving algorithm, the TTC -1 Equal to 0.6 as the safety boundary, and further subdivided into 3 road risk levels. When the output of the early warning module is less than or equal to the first risk threshold of 0.2, it indicates that the current road environment is relatively safe and is considered a non-hazardous road environment. When the output of the early warning module is greater than the first risk threshold of 0.2 but less than or equal to the second risk threshold of 0.4, it indicates that the current road environment risk has increased, and the system needs to sacrifice the algorithm running speed to obtain a more accurate perception result. When the output of the early warning module is between the second risk threshold of 0.4 and the third risk threshold of 0.6, it indicates that the current environment requires the algorithm to be treated with caution, and the perception system requires the highest accuracy perception capability to ensure system safety.

[0112] The above three risk levels correspond to perception models with different computing speeds, so the SAM large model is lightweighted into high-speed perception, medium-speed perception, and low-speed perception models. When the road risk level rises from low to medium, the perception system switches to the medium-speed perception model. When the risk level drops from high to low, the perception system switches to the perception model with the fastest computing speed but sacrifices some accuracy to output the environmental perception results.

[0113] The above-mentioned system risk control step 203 also includes:

[0114] Speed ​​adjustment step 2034: When the risk level increases, based on the speed limit factor, the upper limit of the vehicle speed is limited to ensure user safety; the speed limit factor k 限速 It is calculated by the following function:

[0115]

[0116] Among them, TTC -1 It is the inverse value of the collision time. When the risk increases, the speed limit factor increases dynamically.

[0117] The maximum speed of the above vehicles is:

[0118]

[0119] in, is the upper limit of the vehicle's current speed, V max Plan the speed limit for the vehicle's autonomous driving state, k 限速 is the speed limiting factor.

[0120] Embodiment 3

[0121] like Figure 4 As shown, the embodiment of the present application provides a method for dynamically optimizing the reasoning speed of the perception model based on road risk, and the system includes:

[0122] Environment perception model pre-construction module 301: used for lightweight processing and pre-constructing environment perception models corresponding to different computing speeds according to different risk levels of roads under autonomous driving;

[0123] Environmental warning module 302: used to collect road environment information, calculate the road environment risk value in real time based on the road environment information, and determine the current road environment safety level according to the range of the road risk value;

[0124] System risk control module 303: used to select the environment perception model corresponding to different operating speeds based on the environmental safety level, output the perception result, and dynamically adjust the upper limit of the driving speed to achieve intelligent reasoning and regulation of the operating speed of the autonomous driving perception system;

[0125] The environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of the input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

[0126] The present invention is mainly used to optimize the reasoning time of the autonomous driving environment perception algorithm. The system mainly includes an environment perception model building module, an environment warning module and a system control unit. Among them, the environment perception module is responsible for identifying road environment information; the warning module evaluates the current road conditions and gives the risk level of the current road; the system control unit switches the perception model of different running speeds according to the current road risk level, and dynamically adjusts the maximum speed of the vehicle according to the road risk level.

[0127] The system workflow of the present invention is as follows: Figure 5 As shown:

[0128] 1. Start and mode selection: The user chooses whether to turn on the automatic driving acceleration mode according to the road conditions. If not turned on, the system uses the original slow and high-precision perception model; if turned on, the automatic driving vehicle enters the acceleration mode.

[0129] 2. Early warning module evaluation: In the acceleration mode, the early warning module analyzes the road environment in real time and calculates the risk value based on risk measurement indicators and surrounding environment information.

[0130] 3. System risk control: In the acceleration mode, the system control module receives the risk data indicators output by the early warning module. If the evaluation result is within the safety threshold, the lightweight perception model will continue to be used; if it exceeds the current safety range threshold, a higher-precision perception model will be switched.

[0131] 4. System speed adjustment: When the system switches to a high-precision model, it should limit the corresponding maximum drivable speed and issue an audible and visual warning. The speed limit factor should be dynamically adjusted according to the risk level. The greater the risk level jump, the stronger the system speed limit, and vice versa. Using the speed limit factor can not only improve the safety performance of the autonomous driving vehicle, but also reserve more time for the slow high-precision model to make the best decision, thereby providing greater optimization space for the regulatory control algorithm.

[0132] Embodiment 4

[0133] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for dynamically optimizing the reasoning speed of the perception model based on road risk.

[0134] Embodiment 5

[0135] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risk as described above are implemented.

[0136] In addition, combined Figure 1The method for dynamically optimizing the reasoning speed of the perception model based on road risk described in the embodiment of the present application can be implemented by an electronic device, such as a computer device. Figure 6 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0137] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 6 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0138] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0139] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0140] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the methods for dynamically optimizing the perception model reasoning speed based on road risk in the above embodiments.

[0141] Embodiment 6

[0142] An embodiment of the present application provides an autonomous driving vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risk as described above are implemented.

[0143] Combination Figure 1 The method of dynamically optimizing the reasoning speed of the perception model based on road risk described in the embodiment of the present application can be implemented by an autonomous driving vehicle, such as a vehicle. Figure 7 Schematic diagram of the hardware structure of an autonomous driving vehicle device according to an embodiment of the present application.

[0144] The method and system of the present invention for dynamically optimizing the reasoning speed of the perception model based on road risks can be applied to various application scenarios of autonomous driving vehicles, such as: cars, online taxis, robots, low-altitude aircraft, etc., but the present invention is not limited thereto.

[0145] In summary, compared with the prior art, the method of the present invention has the following beneficial effects:

[0146] 1. The present invention considers adjusting the computing speed of the perception system in combination with the road environment risk, thereby further improving the real-time performance of the perception system, which is innovative;

[0147] 2. The present invention uses model lightweighting as a speed-up replacement solution for the original large model, avoiding the risk of excessive generalization deviation when switching between high-speed perception models and low-speed perception models due to large differences in model structure and training data of different perception models, thereby ensuring the consistency of perception algorithm optimization;

[0148] 3. The present invention adjusts the upper limit of the planned speed of the autonomous driving vehicle based on the road environment risk level, which not only improves the vehicle safety, but also reserves more computing time for the high-precision perception algorithm.

[0149] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for dynamically optimizing the reasoning speed of a perception model based on road risk, characterized in that: The method comprises: Environmental perception model pre-construction step: According to the different risk levels of roads under autonomous driving, lightweight processing pre-constructs environmental perception models corresponding to different computing speeds; Environmental warning step: collecting road environment information, and calculating the road environment risk value in real time based on the road environment information, and determining the current road environment safety level according to the range of the road risk value; System risk control step: Based on the environmental safety level, select the environmental perception model corresponding to different operating speeds, output the perception result, and dynamically adjust the upper limit of the driving speed to achieve intelligent regulation of the operating speed of the autonomous driving perception system.

2. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 1 is characterized in that: The environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

3. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 1 or 2, characterized in that: The environmental perception model pre-construction step also includes: Preliminary feature extraction step: the collected image is adjusted to preset pixels, image dimension reduction is performed through two-dimensional convolution, and the number of channels is increased to extract preliminary feature maps; Deep semantic feature extraction step: according to the preliminary feature map, position coding is added to retain the spatial position information of the elements in the image, the preliminary feature map is sent to a sequence composed of multiple Transformer modules, the dimension of the preliminary feature map is adjusted to a suitable scale, and the road environment risk information is extracted.

4. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 1 or 2, characterized in that: The environmental early warning step comprises: Obstacle determination step: according to the road environment risk information, obstacles on the current road are identified through a detection algorithm, and image coordinates of the obstacles are obtained, and the relative distance of the obstacles is obtained according to the external parameters between the image coordinates and the radar sensor, and the obstacle closest to the vehicle is determined; The collision time calculation step is as follows: the relative speed between the obstacle target and the vehicle is obtained through inertial navigation and radar, and the inverse value of the collision time of the target closest to the vehicle is calculated according to the relative distance and the relative speed.

5. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 4 is characterized in that: The system risk control steps include: High-speed perception model decision step: when the calculated inverse value of the collision time is less than or equal to the first risk threshold, the current road environment is judged to be relatively safe and is regarded as a non-hazardous road environment, and the high-speed perception model is correspondingly selected to output the perception result; Medium-speed perception model decision step: when the calculated inverse value of the collision time is greater than the first risk threshold and less than or equal to the second risk threshold, it is judged that the current road environment risk is increased, and the medium-speed perception model is selected accordingly, and the perception result is output; Low-speed perception model decision steps: When the calculated inverse value of the collision time is greater than the second risk threshold and less than or equal to the third risk threshold, it indicates that the current environment requires the algorithm to treat it with caution, and the low-speed perception model is selected accordingly to output the perception result.

6. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 5 is characterized in that: The system risk control step also includes: Speed ​​adjustment step: When the risk level increases, based on the speed limit factor, the upper limit of the driving speed is limited to ensure user safety; the speed limit factor k 限速 It is calculated by the following function: Among them, TTC -1 It is the inverse value of the collision time. When the risk increases, the speed limit factor increases dynamically.

7. The method for dynamically optimizing the reasoning speed of the perception model based on road risk according to claim 6 is characterized in that: The upper limit of the driving speed is: in, is the current maximum speed limit, V max Plan the speed limit for the vehicle's autonomous driving state, k 限速 is the speed limiting factor.

8. A system for dynamically optimizing the reasoning speed of a perception model based on road risk, using the method for dynamically optimizing the reasoning speed of a perception model based on road risk as claimed in any one of claims 1 to 7, characterized in that: The system comprises: Environmental perception model pre-construction module: used to pre-construct environmental perception models corresponding to different computing speeds based on different risk levels of roads under autonomous driving. Environmental warning module: used to collect road environment information, calculate the road environment risk value in real time based on the road environment information, and determine the current road environment safety level according to the range of the road risk value; System risk control module: used to select the environmental perception model corresponding to different operating speeds based on the environmental safety level, output the perception result, and dynamically adjust the upper limit of the driving speed to achieve intelligent control of the operating speed of the autonomous driving perception system; The environmental perception model includes: a coarse feature extraction network and a deep semantic feature extraction network; the coarse feature extraction network is used to extract preliminary features of input data; the deep semantic feature extraction network is used to further extract deep semantic information based on the preliminary features.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risk described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risk are implemented as described in any one of claims 1 to 7.

11. An autonomous driving vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for dynamically optimizing the reasoning speed of the perception model based on road risk are implemented as described in any one of claims 1 to 7.

Images