Automatic driving behavior planning method and device based on pure vision instance detection

Abstract

The invention discloses an automatic driving behavior planning method and device based on pure vision instance detection, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining a vision input image of a current frame of a vehicle, and determining the current vehicle speed and the current vehicle steering angle in real time; outputting object semantic categories and 3D positions of the plurality of object targets based on the visual input image through a detection model; projecting the object target to an obstacle avoidance area of the vehicle according to each 3D position, and determining a relative distance between a target obstacle and a vehicle object to which the target obstacle belongs; the vehicle-object relative speed of each target obstacle is determined, and the corresponding obstacle risk level is determined in combination with the object semantic category and the vehicle-object relative distance; and based on each obstacle risk level and the associated vehicle-object relative distance, vehicle-object relative speed and object semantic category, the current vehicle speed and the current vehicle steering angle of the vehicle are adjusted through a preset semantic-based obstacle avoidance behavior rule base decision. On the basis of the scheme, the reliability of automatic driving obstacle avoidance planning is integrally improved in an assisting manner.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving behavior planning method and apparatus based on pure vision instance detection. Background Technology

[0002] In autonomous driving systems, environmental perception and real-time obstacle avoidance are key aspects of ensuring safe vehicle operation. By identifying and locating surrounding obstacles, the planning module can be provided with an executable and safe trajectory.

[0003] Traditional obstacle avoidance planning methods for autonomous driving, such as point cloud obstacle avoidance planning methods based on LiDAR, can provide high-precision 3D environment modeling and accurate obstacle detection, but they have high hardware costs, high computing power requirements and high system complexity. At the same time, point cloud data lacks semantic information, which limits the intelligence and naturalness of obstacle avoidance decisions, resulting in low reliability of autonomous driving obstacle avoidance planning. Summary of the Invention

[0004] This invention provides an autonomous driving behavior planning method and device based on pure visual instance detection, which solves the technical problem that traditional autonomous driving obstacle avoidance planning methods mainly adopt point cloud obstacle avoidance mechanisms based on LiDAR, which have the limitations of slow computation efficiency and lack of semantic information in point cloud data, resulting in low reliability of autonomous driving obstacle avoidance planning.

[0005] The first aspect of this invention provides an autonomous driving behavior planning method based on pure vision instance detection, comprising:

[0006] Acquire the visual input image of the vehicle in the current frame and determine the current vehicle speed and steering angle in real time;

[0007] The system performs target detection on the visual input image of the current frame using a preset detection model, and outputs the semantic category and 3D position of multiple object targets.

[0008] Project the corresponding object target onto the obstacle avoidance area of ​​the vehicle according to each of the 3D positions, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle;

[0009] The relative speed between the vehicle and the target obstacle is determined based on the current vehicle speed, and the risk is assessed by combining the semantic category of the object with the relative distance between the vehicle and the object, thus determining the obstacle risk level of each target obstacle.

[0010] Based on the obstacle risk level and the associated vehicle-object relative distance, vehicle-object relative speed and object semantic category, the current vehicle speed and current vehicle steering angle are adjusted by a preset semantic-based obstacle avoidance behavior rule base to determine the target vehicle speed and target vehicle steering angle.

[0011] Optionally, it also includes:

[0012] Perform semantic segmentation on the visual input image of the current frame and output the road mask;

[0013] Based on the road mask, a set of road edge points is extracted, and the set of road edge points is used for curve fitting to determine the road edge line;

[0014] Based on the road edge line, and with the vehicle as a reference, construct the linear equation of the road edge line;

[0015] Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

[0016] Optionally, the step of adjusting the vehicle's current speed and steering angle based on the obstacle risk level and the associated vehicle-object relative distance, vehicle-object relative speed, and object semantic category according to a preset semantic-based obstacle avoidance behavior rule base, and determining the target vehicle speed and target vehicle steering angle, includes:

[0017] The system uses a finite state machine to determine the target obstacle avoidance trigger conditions that match the obstacle risk level based on a preset semantic-based obstacle avoidance behavior rule library. Based on the target obstacle avoidance trigger conditions, the system determines the target obstacle avoidance behavior strategy by combining the vehicle-object relative distance, vehicle-object relative speed and object semantic category. The system then adjusts the vehicle's current speed and steering angle according to the priority order associated with each obstacle risk level, and determines the target vehicle speed and steering angle.

[0018] Optionally, determining the relative vehicle-to-object speed of each of the target obstacles based on the current vehicle speed includes:

[0019] Based on the visual input image of the current frame, perform temporal filtering prediction to determine the instantaneous velocity of each of the object targets;

[0020] The relative speed between the vehicle and the object is calculated using the instantaneous speed of each object and the current speed of the vehicle.

[0021] Optionally, the detection model is a depth-guided Transformer model.

[0022] A second aspect of the present invention provides an autonomous driving behavior planning device based on pure vision instance detection, comprising:

[0023] The data acquisition module is used to acquire the visual input image of the vehicle in the current frame and determine the current vehicle speed and steering angle in real time.

[0024] The object detection module is used to perform object detection on the visual input image of the current frame using a preset detection model, and output the semantic category and 3D position of multiple object targets.

[0025] An obstacle determination module is used to project the corresponding object target onto the obstacle avoidance area of ​​the vehicle according to each of the 3D positions, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle;

[0026] The risk assessment module is used to determine the relative speed between the vehicle and the target obstacle based on the current vehicle speed, and to assess the risk by combining the semantic category of the object with the relative distance between the vehicle and the object, thereby determining the obstacle risk level of each target obstacle.

[0027] The driving planning module is used to adjust the current vehicle speed and current vehicle steering angle of the vehicle based on the risk level of each obstacle and the associated relative distance, relative speed and semantic category of the vehicle and the object, and to determine the target vehicle speed and target vehicle steering angle by using a preset semantic obstacle avoidance behavior rule base.

[0028] Optionally, it also includes a line-following control module for:

[0029] Perform semantic segmentation on the visual input image of the current frame and output the road mask;

[0030] Based on the road mask, a set of road edge points is extracted, and the set of road edge points is used for curve fitting to determine the road edge line;

[0031] Based on the road edge line, and with the vehicle as a reference, construct the linear equation of the road edge line;

[0032] Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

[0033] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in any of the preceding claims.

[0034] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the autonomous driving behavior planning method based on pure vision instance detection as described in any of the preceding claims.

[0035] The fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the autonomous driving behavior planning method based on pure vision instance detection as described in any of the preceding claims.

[0036] As can be seen from the above technical solutions, the present invention has the following advantages: The above solution of the present invention provides an autonomous driving behavior planning based on pure visual instance detection, including: acquiring the visual input image of the current frame of the vehicle and determining the current vehicle speed and the current vehicle steering angle in real time; performing target detection on the visual input image of the current frame through a preset detection model, and outputting the object semantic category and 3D position of multiple object targets; projecting the corresponding object target to the obstacle avoidance area of ​​the vehicle according to each 3D position, determining the target obstacle in the obstacle avoidance area, and determining the vehicle-object relative distance of the target obstacle; determining the vehicle-object relative speed of each target obstacle according to the current vehicle speed, and assessing the risk by combining the object semantic category and the vehicle-object relative distance, and determining the obstacle risk level of each target obstacle; based on each obstacle risk level and the associated vehicle-object relative distance, vehicle-object relative speed and object semantic category, adjusting the current vehicle speed and the current vehicle steering angle through a preset semantic-based obstacle avoidance behavior rule base, and determining the target vehicle speed and the target vehicle steering angle. Based on the above scheme, we combine neural network-driven pure visual 3D object detection (which provides instance-level semantic information) with a semantic rule-based driving obstacle avoidance behavior planning generation method. In the planning process, on the one hand, we avoid point cloud processing and directly perform obstacle avoidance based on instances, so that the obstacle avoidance strategy can obtain richer semantic information. Compared with point cloud-based planning, it exhibits more intelligent and natural behavior, making it more suitable for real-time embedded system application scenarios. On the other hand, compared with end-to-end planning that relies on complex neural networks, it has natural advantages of high computational efficiency, small memory footprint, and strong interpretability, which helps to improve the reliability of autonomous driving obstacle avoidance planning overall. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 The flowchart illustrates the steps of an autonomous driving behavior planning method based on pure visual instance detection, as provided in Embodiment 1 of the present invention.

[0039] Figure 2 This is an architecture diagram of the deep-guided Transformer model provided in Embodiment 1 of the present invention;

[0040] Figure 3 This is a schematic diagram of the depth predictor provided in Embodiment 1 of the present invention;

[0041] Figure 4 This is a structural block diagram of an autonomous driving behavior planning device based on pure vision instance detection, provided in Embodiment 2 of the present invention. Detailed Implementation

[0042] This invention provides an autonomous driving behavior planning method and apparatus based on pure visual instance detection, which addresses the technical problem that traditional autonomous driving obstacle avoidance planning methods mainly adopt point cloud obstacle avoidance mechanisms based on LiDAR, which have limitations such as slow computation efficiency and lack of semantic information in point cloud data, resulting in low reliability of autonomous driving obstacle avoidance planning.

[0043] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0044] Please see Figure 1 The present invention provides an autonomous driving behavior planning method based on pure vision instance detection, comprising:

[0045] Step 101: Obtain the visual input image of the vehicle in the current frame and determine the current vehicle speed and steering angle in real time.

[0046] Visual input images refer to image data acquired through onboard visual sensors (such as low-cost monocular cameras) for environmental perception.

[0047] The current frame refers to the current moment.

[0048] Current vehicle speed refers to the vehicle's speed at the current moment.

[0049] The current vehicle steering angle refers to the steering angle of the vehicle at the current moment, reflecting the current driving direction of the vehicle.

[0050] It should be noted that this embodiment uses visual sensors such as monocular cameras mounted on the vehicle to visually perceive the road environment in which the vehicle is driving, thereby collecting the corresponding visual input images. At the same time, it obtains the current driving status parameters of the vehicle, thereby determining the data basis to support subsequent driving behavior planning and adjustment.

[0051] Step 102: Perform target detection on the visual input image of the current frame using a preset detection model, and output the semantic category and 3D position of multiple object targets.

[0052] A detection model refers to a model used for object detection in images. It has the ability to identify and locate instances of objects of interest in an image and output the attribute information of the object instance.

[0053] Object targets refer to physical objects associated with the autonomous driving environment of vehicles, such as vehicles, pedestrians, cyclists, and traffic cones.

[0054] The semantic category of an object refers to the category attribute of an object, such as which category the object belongs to, such as vehicle, pedestrian, cyclist, cone, etc.

[0055] 3D position refers to the 3D spatial coordinates of an object in the vehicle coordinate system.

[0056] It should be noted that this embodiment uses a detection model to perform target detection on the visual input image, thereby identifying the object targets in the image, and at least outputting two key types of information for each object target: object semantic category and 3D location.

[0057] In one specific implementation of this embodiment, the detection model is a deep-guided Transformer model.

[0058] It should be noted that this embodiment considers using a lightweight, depth-guided Transformer 3D object detection model to achieve efficient obstacle recognition. Its architecture can be found in [reference needed]. Figure 2 and Figure 3 As shown, it includes a feature extraction layer, a global encoder, a deep guided decoder, and a 3D attribute prediction head. Its input is a single-view image, which is processed by each neural network layer to output the category, size, position, and orientation of each detected object.

[0059] Feature extraction layer: Extracts visual and depth features from the visual input image; specifically, a convolutional neural network feature extraction backbone (such as ResNet-50) and a lightweight depth predictor can be used to generate the visual and depth features of the input image, respectively.

[0060] For visual features: Input a high-valued... , width is RGB image Then, feature maps of different scales are obtained from the last three stages of ResNet-50, and these feature maps with sufficient semantic information are used as the visual features of the input image. ;

[0061] For deep features: such as Figure 3 As shown, a lightweight depth predictor is used to obtain the depth features and foreground depth map of the input image; specifically, a bilinear pooling layer is used to process the three-scale feature map obtained by ResNet-50. Size uniformity Then, they are fused by adding them element by element. The fused image is then processed by two 3×3 convolutional layers to obtain the depth features. Obtain deep features The foreground depth map is then obtained by processing it through a 1×1 convolutional layer. This provides reliable deep features for the subsequent deep-guided Transformer architecture;

[0062] The global encoder comprises a parallel visual encoder and a depth encoder, based on Transformer encoder units. Specifically, the visual encoder consists of three Transformer encoder units, while the depth encoder consists of one Transformer encoder unit. Each Transformer encoder unit is composed of a multi-head self-attention layer and a feed-forward network layer connected sequentially. Each layer is equipped with residual connections and layer normalization mechanisms, forming a lightweight encoding structure. Through the global self-attention mechanism, the depth encoder explores long-term dependencies of depth values ​​from different foreground regions, providing non-local geometric cues in stereo space. The decoupling of the depth encoder and the visual encoder allows them to better learn features for themselves, encoding the input image from two perspectives: depth geometry and visual appearance.

[0063] Deep-guided decoder: includes a deep cross-attention layer, a query self-attention layer, a visual cross-attention layer, and a feedforward neural network layer. It initializes a series of object queries, and then fuses global geometric and semantic information through depth-aware queries. The queries eventually become deep-visual queries after a series of interactions.

[0064] Deep cross-attention layer: Each Object Queris searches for the most relevant depth geometry information in the image; the query vector acts as the "questioner" and the depth features act as the "knowledge base". Attention weights are calculated for each query, depth information is aggregated, and the updated query vector is output to the next layer. At this point, each query vector contains the most relevant depth cues extracted from the entire image, such as a specific depth region or object, and perceives the depth relationship between this object and other objects.

[0065] Query Self-Attention Layer: Enables information exchange between all depth-aware query vectors obtained from the deep cross-attention layer; each query (representing a potential object) establishes relationships between objects through interaction with all other queries (e.g., a car typically does not overlap with another car, and pedestrians are more likely to appear on the roadside). Outputs query vectors with enhanced relationships;

[0066] Visual cross-attention layer: Allows queries that already have depth geometric information to extract corresponding appearance semantic information from the image (whether the appearance at this depth location resembles a car or a person); similar to the depth cross-attention layer, it calculates the attention weight for each query and aggregates features.

[0067] Feedforward neural network layer: standard multilayer perceptron (MLP), further processes the features of each query, and finally outputs the final query, Depth-Visual Queries, which simultaneously integrates depth geometry and visual appearance;

[0068] The above decoders can be multiple, meaning they will be executed in cascade multiple times. In each iteration, the object query has the opportunity to better correct its focus and more accurately fuse depth and visual information. After processing by all decoder blocks, the final N query vectors are directly fed into a series of MLP detection heads.

[0069] 3D Attribute Prediction Head: By outputting instance-level 3D attributes through a lightweight MLP, the object queries output by the decoder, which have already incorporated depth and visual information, are transformed into specific 3D detection results. The prediction head can include multiple parallel, lightweight multilayer perceptrons, with each MLP branch independently predicting a specific attribute. This design allows the network to focus on learning the complex mapping relationships between different attributes. The specific model detection output can be seen in Table 1.

[0070] Table 1 Functions of each branch of the prediction head

[0071]

[0072] Step 103: Project the corresponding object target to the vehicle's obstacle avoidance area according to each 3D position, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle.

[0073] The obstacle avoidance zone refers to the vehicle's dynamic safety zone, which is activated by detecting obstacles within the zone. It is understood that the obstacle avoidance zone can be rectangular or fan-shaped; this embodiment does not impose any limitations on this. The size of the zone can be dynamically adjusted based on factors such as vehicle size, speed, and driving scenario. For example, an area with a semi-major axis of 4m and a semi-minor axis of 3m, centered on the vehicle, can be constructed as the obstacle avoidance zone.

[0074] A target obstacle refers to an object within the obstacle avoidance area, which is considered a threat to the vehicle's current driving safety.

[0075] The relative distance between a vehicle and an object refers to the distance relationship between the vehicle and the object.

[0076] It should be noted that, using 3D position information, the detected object is projected onto the obstacle avoidance area plane. When the projection falls within the obstacle avoidance area, it is identified as a target obstacle. The distance between the target obstacle and the vehicle, i.e., the relative distance between the vehicle and the object, is determined based on the projection position of each target obstacle, providing data for subsequent assessment of the risk of the target obstacle.

[0077] Step 104: Determine the relative speed between the vehicle and the obstacle based on the current vehicle speed, and assess the risk by combining the semantic category of the object with the relative distance between the vehicle and the obstacle, and determine the obstacle risk level of each obstacle.

[0078] Vehicle-object relative speed refers to the speed relationship between a vehicle and an object.

[0079] Obstacle risk level refers to an indicator that quantifies the degree of threat a target obstacle poses to the driving safety of a vehicle.

[0080] It should be noted that the risk assessment of the target obstacle is performed, and the obstacle risk level = f(vehicle-object relative distance, vehicle-object relative speed, object semantic category), where f(·) represents the risk assessment model. In this embodiment, the risk level is assigned by mapping the information of the three attributes of distance, speed, and category. The specific mapping relationship can be set according to different driving scenarios. For example, a pedestrian obstacle that suddenly appears is considered to have a high risk level, a vehicle traveling slowly in the same direction is considered to have a medium risk level, and a stationary cone is considered to have a low risk level. It can be understood that there is no restriction on the level of risk classification.

[0081] In one specific embodiment of this example, determining the relative vehicle-object speed of each target obstacle based on the current vehicle speed includes:

[0082] Based on the visual input image of the current frame, perform temporal filtering prediction to determine the instantaneous velocity of each object target;

[0083] The relative speed between the vehicle and the object is calculated by using the instantaneous speed of each object and the current speed of the vehicle.

[0084] The instantaneous velocity of an object refers to the speed at which the object moves.

[0085] It should be noted that when determining the relative speed between the vehicle and the object, the instantaneous speed of each object is first determined, and then the corresponding relative speed between the vehicle and the object is calculated by comparing the instantaneous speed of each object with the current vehicle speed. In a specific implementation, the instantaneous speed of the object can be predicted by temporal filtering. That is, the initial speed of the object is calculated by comparing the position change of the same object in adjacent frames (such as the visual input image of the current frame and the visual input image of the previous frame), and the initial speed of the object is smoothed by a filtering algorithm to determine the instantaneous speed of the object. This method is easy to implement and has a small computational load.

[0086] Step 105: Based on the risk level of each obstacle and the associated relative distance between the vehicle and the object, the relative speed between the vehicle and the object and the semantic category of the object, the current vehicle speed and the current vehicle steering angle are adjusted by a preset semantic-based obstacle avoidance behavior rule base to determine the target vehicle speed and the target vehicle steering angle.

[0087] In one specific embodiment of this example, step 105 includes the following sub-steps:

[0088] The finite state machine determines the target obstacle avoidance trigger conditions for successful matching of each obstacle risk level based on a preset semantic-based obstacle avoidance behavior rule library. Based on each target obstacle avoidance trigger condition, the target obstacle avoidance behavior strategy is determined by combining the relative distance between the vehicle and the object, the relative speed between the vehicle and the object, and the semantic category of the object. The current vehicle speed and the current vehicle steering angle are adjusted according to the priority order of the association of each obstacle risk level, and the target vehicle speed and the target vehicle steering angle are determined.

[0089] Finite state machines are models that automate behavioral decisions by defining predefined states, events, and actions. In autonomous driving, they can determine the strategies that a vehicle should execute in different driving situations. Existing technologies can be consulted today, and they will not be elaborated on here.

[0090] A semantic-based obstacle avoidance behavior rule base refers to a database that maps the relationship between obstacle avoidance triggering conditions and obstacle avoidance behavior strategies. Obstacle avoidance triggering conditions refer to the conditions for determining whether a specific obstacle avoidance behavior strategy is triggered. Obstacle avoidance behavior strategies refer to the driving action decision content planned in response to specific obstacle avoidance triggering conditions.

[0091] The target obstacle avoidance trigger condition refers to the condition selected and matched from the predefined obstacle avoidance trigger conditions in the semantic-based obstacle avoidance behavior rule library.

[0092] The target obstacle avoidance behavior strategy refers to the strategy selected and matched from the predefined obstacle avoidance behavior strategies in the semantic-based obstacle avoidance behavior rule base.

[0093] The target vehicle speed refers to the final speed that the vehicle needs to achieve after adjusting the current vehicle speed.

[0094] The target vehicle steering angle refers to the final steering angle that the vehicle needs to perform after the current vehicle steering angle is adjusted.

[0095] It should be noted that the states in the finite state machine include line following, detour, overtaking, following, and stopping. A semantic-based obstacle avoidance behavior rule base is set for the vehicle. By maintaining a finite state machine, semantic-based obstacle avoidance behavior decisions are implemented. The target vehicle speed (output_velocity) is obtained by adjusting the current vehicle speed using obstacle avoidance strategies (such as following or emergency stopping). The target vehicle steering angle (output_steering) is obtained by adding lateral offset to the steering angle calculated from line following, along with obstacle avoidance strategies (such as detour or overtaking). (output_velocity, output_steering) is then sent to the underlying vehicle drive-by-wire and steering actuator to achieve driving. It can be understood that during line following (straight driving), the vehicle's angular velocity is 0 or a small value by default, and the lateral offset is also 0. When encountering an obstacle and needing to avoid it, the obstacle avoidance strategy will give the vehicle a left or right offset.

[0096] For example, a semantic-based obstacle avoidance behavior decision-making mechanism can be seen in Table 2:

[0097] Table 2. Semantic-based obstacle avoidance behavior decision-making

[0098]

[0099] In one specific embodiment of this example, it further includes:

[0100] Perform semantic segmentation on the visual input image of the current frame and output the road mask;

[0101] The road edge point set is extracted based on the road mask, and the road edge line is determined by curve fitting using the road edge point set;

[0102] Based on the road edge line, construct the linear equation of the road edge line with reference to the vehicle;

[0103] Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

[0104] It should be noted that when the vehicle is in lane-following mode, the visual input image is semantically segmented using a semantic segmentation model (such as DeepLabv3+) to obtain a road mask. The road mask is scanned line by line to extract edge points, and their coordinates are recorded, thus forming a set of road edge points. Polynomial fitting is used to perform curve fitting on the set of road edge points to obtain a continuous road edge line. The slope of the tangent line to the road edge line is calculated using the vehicle's coordinate system as a reference coordinate system to determine the linear equation of the edge line. For simplicity, we assume that the edge line is a straight line within a certain distance in front of the vehicle, and its equation is... ,in, It's the slope. It is the intercept. It is the ordinate of the edge point. It is the x-coordinate of the edge point;

[0105] During lane following, to maintain a fixed distance between the vehicle and the road edge line, a lane following controller, such as the Stanley controller, is needed to combine heading and lateral errors to calculate the final steering angle command. The calculated steering angle is then... The signal is sent to the vehicle's steering actuator (such as a steer-by-wire system), enabling the vehicle to follow the lane.

[0106] Among them, the heading error is the angle between the left edge of the road and the vehicle, and in the vehicle coordinate system, it is the slope of the fitted straight line. The angle itself contains angular information, therefore the inclination angle of a straight line , Since the function is arctangent, the vehicle's heading angle is always 0 in the vehicle's coordinate system. The heading error is the angle between the road direction and the vehicle's current heading. We want the vehicle to maintain a fixed offset from the left edge line. (e.g., 1.5 meters), therefore, lateral error The calculation process is as follows ,if This indicates the car is too close to the edge line and needs to be adjusted to the right; if This indicates that the distance is too far and needs to be adjusted to the left.

[0107] In this embodiment of the invention, neural network-driven pure visual 3D object detection (providing instance-level semantic information) is combined with a semantic rule-based driving obstacle avoidance behavior planning generation method. During the planning process, on the one hand, point cloud processing is avoided, and obstacle avoidance is performed directly based on instances, enabling the obstacle avoidance strategy to obtain richer semantic information. Compared with point cloud-based planning, it exhibits more intelligent and natural behavior, thus being more suitable for real-time embedded system application scenarios. On the other hand, compared with end-to-end planning that relies on complex neural networks, it has natural advantages of high computational efficiency, small memory footprint, and strong interpretability, which overall helps to improve the reliability of autonomous driving obstacle avoidance planning.

[0108] Please see Figure 4 The second embodiment of the present invention provides an autonomous driving behavior planning device based on pure vision instance detection, comprising:

[0109] The data acquisition module 401 is used to acquire the visual input image of the vehicle in the current frame and determine the current vehicle speed and current vehicle steering angle in real time.

[0110] The object detection module 402 is used to perform target detection on the visual input image of the current frame using a preset detection model, and output the semantic category and 3D position of multiple object targets.

[0111] The obstacle determination module 403 is used to project the corresponding object target to the obstacle avoidance area of ​​the vehicle according to each 3D position, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle;

[0112] The risk assessment module 404 is used to determine the relative speed between the vehicle and the obstacle based on the current vehicle speed, and to assess the risk by combining the semantic category of the object with the relative distance between the vehicle and the obstacle, thereby determining the obstacle risk level of each obstacle.

[0113] The driving planning module 405 is used to determine the target vehicle speed and target vehicle steering angle by adjusting the current vehicle speed and current vehicle steering angle based on the obstacle risk level, the associated vehicle-object relative distance, vehicle-object relative speed and object semantic category through a preset semantic-based obstacle avoidance behavior rule base.

[0114] In one specific embodiment of this invention, the device further includes a line-following control module, used for:

[0115] Perform semantic segmentation on the visual input image of the current frame and output the road mask;

[0116] The road edge point set is extracted based on the road mask, and the road edge line is determined by curve fitting using the road edge point set;

[0117] Based on the road edge line, construct the linear equation of the road edge line with reference to the vehicle;

[0118] Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

[0119] In one specific embodiment of this example, the driving planning module 405 is specifically used for:

[0120] The finite state machine determines the target obstacle avoidance trigger conditions for successful matching of each obstacle risk level based on a preset semantic-based obstacle avoidance behavior rule library. Based on each target obstacle avoidance trigger condition, the target obstacle avoidance behavior strategy is determined by combining the relative distance between the vehicle and the object, the relative speed between the vehicle and the object, and the semantic category of the object. The current vehicle speed and the current vehicle steering angle are adjusted according to the priority order of the association of each obstacle risk level, and the target vehicle speed and the target vehicle steering angle are determined.

[0121] In one specific embodiment of this example, determining the relative vehicle-object speed of each target obstacle based on the current vehicle speed includes:

[0122] Based on the visual input image of the current frame, perform temporal filtering prediction to determine the instantaneous velocity of each object target;

[0123] The relative speed between the vehicle and the object is calculated by using the instantaneous speed of each object and the current speed of the vehicle.

[0124] In one specific implementation of this embodiment, the detection model is a deep-guided Transformer model.

[0125] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in Embodiment 1 of the present invention.

[0126] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in Embodiment 1 of the present invention.

[0127] Embodiment 5 of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in Embodiment 1 of the present invention.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0130] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0132] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An autonomous driving behavior planning method based on pure vision instance detection, characterized in that, include: Acquire the visual input image of the vehicle in the current frame and determine the current vehicle speed and steering angle in real time; The system performs target detection on the visual input image of the current frame using a preset detection model, and outputs the semantic category and 3D position of multiple object targets. Project the corresponding object target onto the obstacle avoidance area of ​​the vehicle according to each of the 3D positions, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle; The relative speed between the vehicle and the target obstacle is determined based on the current vehicle speed, and the risk is assessed by combining the semantic category of the object with the relative distance between the vehicle and the object, thus determining the obstacle risk level of each target obstacle. Based on the obstacle risk level and the associated vehicle-object relative distance, vehicle-object relative speed and object semantic category, the current vehicle speed and current vehicle steering angle are adjusted by a preset semantic-based obstacle avoidance behavior rule base to determine the target vehicle speed and target vehicle steering angle.

2. The autonomous driving behavior planning method based on pure vision instance detection according to claim 1, characterized in that, Also includes: Perform semantic segmentation on the visual input image of the current frame and output the road mask; Based on the road mask, a set of road edge points is extracted, and the set of road edge points is used for curve fitting to determine the road edge line; Based on the road edge line, and with the vehicle as a reference, construct the linear equation of the road edge line; Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

3. The autonomous driving behavior planning method based on pure vision instance detection according to claim 1, characterized in that, Based on the obstacle risk levels and associated vehicle-object relative distances, vehicle-object relative speeds, and object semantic categories, the obstacle risk levels are adjusted according to a preset semantic-based obstacle avoidance behavior rule base to determine the vehicle's current speed and steering angle, and the target vehicle speed and steering angle are determined, including: The system uses a finite state machine to determine the target obstacle avoidance trigger conditions that match the obstacle risk level based on a preset semantic-based obstacle avoidance behavior rule library. Based on the target obstacle avoidance trigger conditions, the system determines the target obstacle avoidance behavior strategy by combining the vehicle-object relative distance, vehicle-object relative speed and object semantic category. The system then adjusts the vehicle's current speed and steering angle according to the priority order associated with each obstacle risk level, and determines the target vehicle speed and steering angle.

4. The autonomous driving behavior planning method based on pure vision instance detection according to claim 1, characterized in that, Determining the relative vehicle-object speed of each target obstacle based on the current vehicle speed includes: Based on the visual input image of the current frame, perform temporal filtering prediction to determine the instantaneous velocity of each of the object targets; The relative speed between the vehicle and the object is calculated using the instantaneous speed of each object and the current speed of the vehicle.

5. The autonomous driving behavior planning method based on pure vision instance detection according to claim 1, characterized in that, The detection model is a deep-guided Transformer model.

6. An autonomous driving behavior planning device based on pure vision instance detection, characterized in that, include: The data acquisition module is used to acquire the visual input image of the vehicle in the current frame and determine the current vehicle speed and steering angle in real time. The object detection module is used to perform object detection on the visual input image of the current frame using a preset detection model, and output the semantic category and 3D position of multiple object targets. An obstacle determination module is used to project the corresponding object target onto the obstacle avoidance area of ​​the vehicle according to each of the 3D positions, determine the target obstacle within the obstacle avoidance area, and determine the vehicle-object relative distance of the target obstacle; The risk assessment module is used to determine the relative speed between the vehicle and the target obstacle based on the current vehicle speed, and to assess the risk by combining the semantic category of the object with the relative distance between the vehicle and the object, thereby determining the obstacle risk level of each target obstacle. The driving planning module is used to adjust the current vehicle speed and current vehicle steering angle of the vehicle based on the risk level of each obstacle and the associated relative distance, relative speed and semantic category of the vehicle and the object, and to determine the target vehicle speed and target vehicle steering angle by using a preset semantic obstacle avoidance behavior rule base.

7. The autonomous driving behavior planning device based on pure vision instance detection according to claim 6, characterized in that, It also includes a line-following control module, used for: Perform semantic segmentation on the visual input image of the current frame and output the road mask; Based on the road mask, a set of road edge points is extracted, and the set of road edge points is used for curve fitting to determine the road edge line; Based on the road edge line, and with the vehicle as a reference, construct the linear equation of the road edge line; Based on the linear equation of the edge line, the current vehicle speed and current vehicle steering angle are adjusted by the line-following controller to determine the heading error and lateral error.

8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in any one of claims 1-5.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the autonomous driving behavior planning method based on pure vision instance detection as described in any one of claims 1-5.

Images