Driving path planning method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of vehicles, and discloses a driving path planning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of driving environment images, inputting the plurality of driving environment images into a pre-constructed driving path planning model, and obtaining a target control instruction; and controlling a steering system, a power system and a braking system of the vehicle according to the target control instruction, so that the vehicle runs according to the target running path. According to the planning method, obstacle information of the surrounding environment of the vehicle can be accurately detected, three-dimensional target classification and positioning information output are carried out, and modeling of the surrounding environment by the automatic driving vehicle, system return of a travelable area and transverse and longitudinal system control of the vehicle based on a path planning result are realized; safer automatic driving service is provided for the user, sensor deployment is simple, data transmission is stable, the environment perception ability is high, and the use cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicles, and in particular, to a driving path planning method, a driving path planning device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Currently, Simultaneous Localization and Mapping (SLAM) is a key technology in fields such as robotics, autonomous driving, and augmented reality / virtual reality, aiming to build a map in real time and determine its own position in an unknown environment. Common sensors include lidar, cameras, IMUs (Inertial Measurement Units), ultrasonic waves, etc. Lidar has the characteristics of high precision but high cost; cameras have the characteristics of low cost but being greatly affected by light; IMUs can provide inertial data and are suitable for short-term positioning.

[0003] In related technologies, the perception of the vehicle environment is closely related to the driving control decision of the vehicle. The environmental perception ability of a single sensor is weak, the semantic information for building the driving environment is scarce, the point cloud of millimeter-wave radar is sparse and cannot accurately identify some small objects, and lidar cannot classify obstacles into targets. In addition, due to the large number of parameters in the training of related neural network models, the training is slow, which will further lead to the redundancy of the overall model software development, and the reasoning is complex when dealing with single-modal and multi-modal, and the performance of the network model is poor. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems in related technologies to some extent. To this end, the first object of the present invention is to propose a driving path planning method, which can accurately detect the obstacle information in the vehicle surrounding environment, perform three-dimensional target classification and output positioning information, realize the modeling of the surrounding environment of the autonomous driving vehicle, systematically transmit the feasible driving area, control the lateral and longitudinal systems of the vehicle based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the use cost.

[0005] The second object of the present invention is to propose a driving path planning device.

[0006] The third object of the present invention is to propose an electronic device.

[0007] The fourth object of the present invention is to propose a computer-readable storage medium.

[0008] To achieve the above object, an embodiment of the first aspect of the present invention provides a driving path planning method, including: obtaining multiple driving environment images, inputting the multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction; controlling the steering system, power system, and braking system of the vehicle according to the target control instruction so that the vehicle travels along the target driving path.

[0009] In addition, the driving path planning method according to the above embodiment of the present invention may further have the following additional technical features:

[0010] According to some embodiments of the present invention, the above method further includes: obtaining multiple groups of environment images through multiple groups of shooting devices preset on the vehicle, and obtaining multiple groups of environment bird's-eye view images according to the multiple groups of environment images; wherein, each group of environment images includes multiple environment images, and each group of environment images records the driving environment around the vehicle by 360°; dividing the multiple groups of environment bird's-eye view images according to a preset ratio to obtain multiple groups of training environment images, multiple groups of verification environment images, and multiple groups of test environment images.

[0011] According to some embodiments of the present invention, the driving path planning model includes a verified classification neural network model; the above method further includes: training a basic neural network model through multiple groups of training environment images; wherein, the basic neural network model is used to extract environmental elements in the training environment images and classify the environmental elements; introducing an attention module into the basic neural network model, embedding the classification label set of the environmental elements into the attention module to obtain an initial attention module; setting a preset classification task, inputting the preset classification task into the initial attention module, and associating the preset classification task with the classification label set to obtain a trained classification neural network model; wherein, the preset classification task indicates the importance of each label in the classification label.

[0012] According to some embodiments of the present invention, the above method further includes: verifying the trained classification neural network model through multiple groups of verification environment images, and obtaining a verified classification neural network model in response to the loss value of the loss function of the trained classification neural network model reaching a target threshold.

[0013] According to some embodiments of the present invention, the above method further includes: testing the verified classification neural network model through multiple groups of test environment images, and obtaining a verified classification neural network model in response to the classification accuracy and weight assignment accuracy of the environmental elements in the multiple groups of test environment images by the verified classification neural network model reaching a preset standard.

[0014] According to some embodiments of the present invention, the target control instruction includes at least one of a steering instruction, an acceleration instruction, and a deceleration instruction; inputting multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction, including: inputting multiple driving environment images into the driving path planning model, and obtaining a target environment bird's-eye view image according to the multiple driving environment images; extracting target environment features of environmental elements in the target environment bird's-eye view image, classifying the target environment features, and assigning weights to each target environment feature; generating a target driving path and a steering instruction, an acceleration instruction, or a deceleration instruction according to the classified target environment features.

[0015] According to some embodiments of the present invention, the target environment features include at least one of a driving turning indication feature, a driving progress indication feature, and an obstacle feature; wherein, the driving turning indication feature is used to indicate the generation of a steering instruction, the driving progress indication feature is used to indicate the generation of an acceleration instruction, and the obstacle feature is used to indicate the generation of a deceleration instruction; controlling the steering system, the power system, and the braking system of the vehicle according to the target control instruction, so that the vehicle travels along the target driving path, including: sending a steering instruction to the steering system, so that the steering system controls the vehicle to travel along the target driving path according to the steering instruction; and / or sending a progress instruction to the power system, so that the power system controls the vehicle to travel along the target driving path according to the power instruction; and / or sending a deceleration instruction to the braking system, so that the braking system controls the vehicle to travel along the target driving path according to the deceleration instruction.

[0016] The driving path planning method according to an embodiment of the present invention includes: acquiring multiple driving environment images, inputting the multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction; controlling the steering system, the power system, and the braking system of the vehicle according to the target control instruction, so that the vehicle travels along the target driving path. Thus, this method can accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the feasible driving area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the use cost.

[0017] The second object of the present invention is to propose a driving path planning device, which can accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the feasible driving area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the use cost.

[0018] To achieve the above object, an embodiment of the second aspect of the present invention provides a driving path planning device, including: an acquisition module configured to acquire multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control instruction; a control module configured to control the steering system, power system, and braking system of the vehicle according to the target control instruction, so that the vehicle travels along the target driving path.

[0019] The driving path planning device according to an embodiment of the present invention includes: an acquisition module configured to acquire multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control instruction; a control module configured to control the steering system, power system, and braking system of the vehicle according to the target control instruction, so that the vehicle travels along the target driving path. Thus, the device can accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the drivable area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the usage cost.

[0020] To achieve the above object, an embodiment of the third aspect of the present invention provides an electronic device, including: a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the above driving path planning method are implemented.

[0021] The electronic device according to an embodiment of the present invention, by executing the above driving path planning method, can accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the drivable area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the usage cost.

[0022] To achieve the above object, an embodiment of the fourth aspect of the present invention provides a computer-readable storage medium, where a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by the processor, the steps of the above driving path planning method are implemented.

[0023] According to the computer-readable storage medium of an embodiment of the present invention, through the above-mentioned driving path planning method, it is possible to accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the feasible driving area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide users with a safer autonomous driving service, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the usage cost.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a driving path planning method according to some embodiments of the present invention;

[0026] Figure 2 It is a schematic block diagram of the installation position of a shooting device of a vehicle according to some embodiments of the present invention;

[0027] Figure 3 It is a flowchart of obtaining a trained classification neural network model according to some embodiments of the present invention;

[0028] Figure 4 It is a schematic block diagram of a driving path planning device according to some embodiments of the present invention;

[0029] Figure 5 It is a schematic block diagram of an electronic device according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should have the ordinary meanings understood by those of ordinary skill in the art to which the present invention belongs. The "first", "second" and similar terms used in the embodiments of the present invention do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0032] As described in the background art section, currently, Simultaneous Localization and Mapping (SLAM) is a key technology in the fields of robotics, autonomous driving, augmented reality / virtual reality, etc., aiming to build a map in real time and determine its own position in an unknown environment. Commonly used sensors include lidar, cameras, IMUs, ultrasonic sensors, etc. Lidar has the characteristics of high precision but high cost; cameras have the characteristics of low cost but being greatly affected by light; IMUs can provide inertial data and are suitable for short-term positioning.

[0033] Front-end processing is responsible for data association and initial pose estimation. Commonly used methods include feature matching, direct methods, etc.; back-end optimization optimizes poses and maps through graph optimization or filtering algorithms (such as extended Kalman filters, particle filters); loop detection identifies visited areas and corrects cumulative errors. Commonly used methods include bag-of-words models and deep learning. Map construction generates 2D or 3D maps. Commonly used representation methods include point cloud maps, grid maps, topological maps, etc.

[0034] The applicant found in the process of implementing the present invention that, in the related art, the perception of the vehicle environment is closely related to the driving control decision-making of the vehicle. The environmental perception ability of a single sensor is weak, the semantic information for constructing the driving environment is scarce, the point cloud of the millimeter-wave radar is sparse and cannot accurately identify some small objects, and the lidar cannot classify obstacles by target. In addition, since the relevant neural network models have a large number of parameters during training, resulting in slow training, it will further lead to redundant software development of the overall model, and the inference is more complex when dealing with single-modal and multi-modal, and the performance of the network model is not good.

[0035] Therefore, the present invention can accurately detect obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the feasible driving area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide users with a safer autonomous driving service, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the usage cost.

[0036] The following describes a driving path planning method, a driving path planning device, an electronic device, and a computer-readable storage medium according to embodiments of the present invention with reference to the accompanying drawings.

[0037] Refer to Figure 1 , which is a flowchart of a driving path planning method according to some embodiments of the present invention.

[0038] As Figure 1 shown, the driving path planning method according to an embodiment of the present invention may include the following steps:

[0039] S101, obtain multiple driving environment images, and input the multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction.

[0040] Specifically, refer to Figure 2 , which is a schematic block diagram of the installation position of a vehicle's photographing device according to some embodiments of the present invention. Multiple driving environment images of the vehicle can be detected simultaneously by a surround-view camera installed on the vehicle. For example, the number of surround-view cameras can be 6, and the installation positions of the surround-view cameras can be in the front, left front, left rear, right front, right rear, and rear of the vehicle. After obtaining multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction, where the driving path planning model can be constructed based on a neural network model, and the target control instruction can be a control instruction for the vehicle to turn, accelerate, or decelerate.

[0041] S102, control the steering system, power system, and braking system of the vehicle according to the target control instruction so that the vehicle travels along the target driving path.

[0042] Specifically, after obtaining the target control instruction, the steering system, power system, and braking system of the vehicle are controlled according to the target control instruction, so that the vehicle travels along the target driving path. Among them, the steering system can control the vehicle to turn, the power system can control the vehicle to accelerate, and the braking system can control the vehicle to decelerate. For example, when the target control instruction is a steering instruction, the steering system of the vehicle is controlled to steer the vehicle; another example is that when the target control instruction is an acceleration instruction, the power system of the vehicle is controlled to accelerate the vehicle; and another example is that when the target control instruction is a deceleration instruction, the braking system of the vehicle is controlled to decelerate the vehicle. Thus, it is possible to systematically transmit the feasible driving area of the vehicle, control the lateral and longitudinal systems of the vehicle based on the path planning result, and provide users with safer autonomous driving services.

[0043] Map building and positioning are performed according to the driving environment of the autonomous vehicle, making the determined feasible driving area and control decisions more reliable. The pre-constructed driving path planning model can adapt to complex and variable environments, accurately detect obstacle information in the vehicle's surrounding environment, and output three-dimensional target classification and positioning information.

[0044] In some embodiments of the present invention, the above method further includes: obtaining multiple sets of environmental images through multiple sets of shooting devices preset on the vehicle, and obtaining multiple sets of environmental bird's-eye images according to the multiple sets of environmental images; wherein, each set of environmental images includes multiple environmental images, and each set of environmental images records the driving environment around the vehicle by 360°; the multiple sets of environmental bird's-eye images are divided according to a preset ratio to obtain multiple sets of training environmental images, multiple sets of verification environmental images, and multiple sets of test environmental images. Among them, the preset ratio can be calibrated according to the actual situation.

[0045] Specifically, multiple sets of environmental images are obtained through multiple sets of preset photographing devices on the vehicle. Among them, the photographing devices can be surround-view cameras, and the multiple sets of preset photographing devices can be 6 sets of photographing devices. It can be understood that 6 surround-view cameras on the vehicle are used to simultaneously obtain 6 driving environment images of the vehicle. Taking every 6 driving environment images as 1 set of environmental images, and splicing the 6 driving environment images in 1 set can obtain 1 set of environmental bird's-eye view images. In the same way, multiple sets of environmental images can be obtained from the starting point to the ending point of the journey, and thus multiple sets of environmental bird's-eye view images can be obtained. Each set of environmental images can record the 360° driving environment around the vehicle, which means that each set of environmental images can obtain the driving environment at each angle of the vehicle to monitor the full-range driving environment of the vehicle. After obtaining multiple sets of environmental bird's-eye view images, the multiple sets of environmental bird's-eye view images are divided according to a preset ratio to obtain multiple sets of training environmental images, multiple sets of verification environmental images, and multiple sets of test environmental images, which are equivalent to the training set, verification set, and test set in the neural network model. For example, the multiple sets of environmental bird's-eye view images are divided according to a ratio of 4:2:2. Assuming there are 100 sets of environmental bird's-eye view images in total, 40 sets of training environmental images, 20 sets of verification environmental images, and 20 sets of test environmental images can be obtained.

[0046] In some embodiments of the present invention, the driving path planning model includes a verified classification neural network model; the above method further includes: training a basic neural network model through multiple sets of training environmental images; wherein, the basic neural network model is used to extract environmental elements in the training environmental images and classify the environmental elements; an attention module is introduced into the basic neural network model, and the classification label set of the environmental elements is embedded into the attention module to obtain an initial attention module; a preset classification task is set, and the preset classification task is input into the initial attention module, and the preset classification task is associated with the classification label set to obtain a trained classification neural network model; wherein, the preset classification task indicates the importance of each label in the classification label.

[0047] Specifically, after obtaining multiple sets of training environment images, multiple sets of validation environment images, and multiple sets of test environment images, a basic neural network model is trained using the multiple sets of training environment images. The basic neural network model is used to extract environmental elements from the training environment images and classify the environmental elements. Among them, the environmental elements can be dynamic obstacles and static obstacles. Dynamic obstacles include moving vehicles, pedestrians, and walking animals, etc. Static obstacles include big trees fallen on the road, road signs, and road potholes, etc. Classifying the same environmental elements in the training environment images can obtain multiple categories, and tagging each category respectively can obtain the classification labels corresponding to each environmental element. An attention module is added to the basic neural network model, and the classification label set of the environmental elements is embedded into the attention module. At this time, the attention module changes, and the changed attention module is used as the initial attention module. Among them, the attention module is a mechanism widely used in deep learning, aiming to simulate the characteristics of human visual attention, that is, being able to focus on important parts while ignoring unimportant parts when processing information. The core idea of the attention module is to dynamically allocate weights, enabling the model to more effectively process key information in the input data, thereby improving the performance and efficiency of the model. Set the importance of each label in the classification labels, input the importance of each label in the classification labels into the initial attention module, and associate the preset classification task with the classification label set to focus on the labels with higher label importance, and obtain the trained classification neural network model. Among them, the classification neural network model can be a convolutional neural network, a recurrent neural network, a graph neural network, a multi-modal neural network, etc. For example, set the importance of pedestrians to be the greatest. During the automatic driving of the vehicle, when pedestrians and other obstacles appear on the road, the trained classification neural network model will pay more attention to pedestrians than other obstacles to ensure the safety of pedestrians.

[0048] As a specific embodiment, as Figure 3 shown, the flowchart of obtaining the trained classification neural network model of the present invention may include the following steps:

[0049] S301, Obtain multiple sets of environmental images through multiple sets of shooting devices preset on the vehicle.

[0050] S302, Obtain multiple sets of environmental bird's-eye view images according to the multiple sets of environmental images.

[0051] S303, Divide the multiple sets of environmental bird's-eye view images according to a preset ratio.

[0052] S304, Obtain multiple sets of training environment images, multiple sets of validation environment images, and multiple sets of test environment images.

[0053] S305, Train a basic neural network model using multiple sets of training environment images.

[0054] S306. Introduce an attention module into the basic neural network model, and embed the classification label set of environmental elements into the attention module to obtain an initial attention module.

[0055] S307. Set a preset classification task, input the preset classification task into the initial attention module, and associate the preset classification task with the classification label set to obtain a trained classification neural network model.

[0056] In some embodiments of the present invention, the above method further includes: verifying the trained classification neural network model through multiple groups of verification environmental images, and obtaining a verified classification neural network model in response to the loss value of the loss function of the trained classification neural network model reaching a target threshold. The target threshold can be calibrated according to the actual situation.

[0057] Specifically, after obtaining the trained classification neural network model, verify the trained classification neural network model through multiple groups of verification environmental images divided according to a preset ratio. That is to say, add multiple groups of verification environmental images divided according to a preset ratio to the trained classification neural network model. Determine whether the loss value of the loss function of the trained classification neural network model reaches the target threshold. When the loss value of the loss function of the trained classification neural network model reaches the target threshold, it can indicate that the training accuracy of the trained classification neural network model is relatively high. At this time, a verified classification neural network model can be obtained.

[0058] In some embodiments of the present invention, the above method further includes: testing the verified classification neural network model through multiple groups of test environmental images, and obtaining a verified classification neural network model in response to the classification accuracy and weight allocation accuracy of the environmental elements in the multiple groups of test environmental images by the verified classification neural network model reaching a preset standard. The preset standard can be calibrated according to the actual situation.

[0059] Specifically, after obtaining the verified classification neural network model, test the verified classification neural network model through multiple groups of test environmental images divided according to a preset ratio, and add multiple groups of test environmental images divided according to a preset ratio to the verified classification neural network model. Determine whether the classification accuracy and weight allocation accuracy of the environmental elements in the multiple groups of test environmental images by the verified classification neural network model reach the preset standard. When the classification accuracy and weight allocation accuracy of the environmental elements in the multiple groups of test environmental images by the verified classification neural network model reach the preset standard, it can indicate that the verification accuracy of the verified classification neural network model is relatively high, and a verified classification neural network model can be obtained. At this time, the verified classification neural network model can be applied.

[0060] In summary, the present invention uses a feature pyramid network to extract multi-scale features of an environmental image, encodes and models the features of the environmental image into features of a bird's-eye view image, and finally, the decoding method can complete the classification and positioning tasks of 3D object detection. The model is built in the early stage, then a dataset is collected, data annotation is carried out, the model is trained, tested according to the training weight results, and the model is tuned and optimized according to the test results. The above steps are repeated until the model version is locked. The model is packaged and integrated into system software, and the software is burned into a pre-constructed driving path planning model to provide users with a more intelligent autonomous driving service experience and safety guarantee, and can simultaneously achieve 3D classification and positioning of objects around the vehicle. An omnidirectional camera is deployed on the autonomous vehicle, and the data of the camera is synchronously transmitted in real time to the pre-constructed driving path planning model. After being calculated by the pre-constructed driving path planning model, a target control instruction is output, a signal is sent to the associated components to control the autonomous vehicle to drive.

[0061] By adding an attention module to strengthen the overall network model, and using the output of the feature processing detection and segmentation of the bird's-eye view image as the control signal of the controller, the actions of the steering system, braking system, and power system are controlled by the signal, so that the method for positioning and mapping of the autonomous vehicle provided by the present invention can better cope with the environmental changes of complex driving traffic during vehicle driving. Not only is the control method relatively flexible, but also the autonomous vehicle can be accurately decision-controlled according to the output of the positioning and mapping of the bird's-eye view image, which can meet the control decision requirements of existing autonomous vehicles.

[0062] In some embodiments of the present invention, the target control instruction includes at least one of a steering instruction, an acceleration instruction, and a deceleration instruction; inputting multiple driving environment images into the pre-constructed driving path planning model to obtain a target control instruction, including: inputting multiple driving environment images into the driving path planning model, obtaining a target environmental bird's-eye view image according to the multiple driving environment images; extracting target environmental features of environmental elements in the target environmental bird's-eye view image, classifying the target environmental features, and assigning weights to each target environmental feature; generating a target driving path and a steering instruction, an acceleration instruction, or a deceleration instruction according to the classified target environmental features.

[0063] Specifically, after obtaining the verified classification neural network model, since the driving path planning model includes the verified classification neural network model, it can be understood that after obtaining the driving path planning model, multiple driving environment images of the vehicle at the current moment are input into the driving path planning model, and the bird's-eye view image of the target environment at the current moment can be obtained, that is, the all-round information around the vehicle at the current moment can be obtained. The environmental elements in the environmental image can be obtained according to the bird's-eye view image of the target environment at the current moment, and then the target environmental features can be obtained from the environmental elements. The target environmental features can be the environmental elements that need special attention extracted from all environmental elements. Then, the target environmental features are classified to obtain multiple categories, and each type of target environmental feature is assigned a weight. According to the weight assignment, the importance of each type of target environmental feature can be known. A target driving path and steering instructions, acceleration instructions or deceleration instructions are generated according to the classified target environmental features. For example, when the target environmental feature is that there is a collapse in the front section of the road, the generated target driving path is to bypass at the front intersection, and steering instructions and deceleration instructions are generated. It can be understood that there are obstacles on the front road, and the obstacles need to be avoided and the vehicle cannot continue to drive forward. Another section of the road needs to be selected to continue driving. At this time, the vehicle needs to decelerate and turn.

[0064] In some embodiments of the present invention, the target environmental features include at least one of a driving turn indication feature, a driving progress indication feature, and an obstacle feature; wherein, the driving turn indication feature is used to indicate the generation of steering instructions, the driving progress indication feature is used to indicate the generation of acceleration instructions, and the obstacle feature is used to indicate the generation of deceleration instructions; controlling the steering system, the power system, and the braking system of the vehicle according to the target control instructions so that the vehicle travels along the target driving path includes: sending a steering instruction to the steering system so that the steering system controls the vehicle to travel along the target driving path according to the steering instruction; and / or sending a progress instruction to the power system so that the power system controls the vehicle to travel along the target driving path according to the power instruction; and / or sending a deceleration instruction to the braking system so that the braking system controls the vehicle to travel along the target driving path according to the deceleration instruction.

[0065] For example, when a road collapse occurs in the environment in front of the vehicle during driving, the vehicle needs to turn. At this time, the pre-constructed driving path planning model sends a steering instruction to the steering system, so that the steering system controls the vehicle to drive along the target driving path according to the steering instruction; or / and when a pedestrian appears in the environment in front of the vehicle during driving, the vehicle needs to reduce its speed to ensure the safety of the pedestrian. At this time, the pre-constructed driving path planning model sends a deceleration instruction to the braking system, so that the braking system controls the vehicle to drive along the target driving path according to the deceleration instruction; or / and when there are no obstacles in the environment in front of the vehicle during driving, the driving speed of the vehicle can be increased. At this time, the pre-constructed driving path planning model sends a travel instruction to the power system, so that the power system controls the vehicle to drive along the target driving path according to the power instruction. Or / and when the environment in front of the vehicle during driving is dark, the environment in front of the vehicle can be illuminated. At this time, the pre-constructed driving path planning model sends a headlight-on instruction to the headlight system, so that the headlight system controls the vehicle to drive along the target driving path according to the headlight-on instruction. Or / and when the noise in the environment in front of the vehicle during driving is relatively large, it is easy to interfere with the vehicle driving. At this time, the pre-constructed driving path planning model sends a noise reduction instruction to the noise reduction system, so that the noise reduction system controls the vehicle to drive along the target driving path according to the noise reduction instruction. Or / and when it rains in the environment in front of the vehicle during driving and the line of sight ahead is unclear, the pre-constructed driving path planning model sends an instruction to turn on the windshield wiper and the fog lamp to the rain-proof system, so that the rain-proof system controls the vehicle to drive along the target driving path according to the instruction to turn on the windshield wiper and the fog lamp, and can perform a systematic feedback on the drivable area, control the vehicle's lateral and longitudinal systems based on the path planning result, and provide users with a safer autonomous driving service.

[0066] Therefore, in an autonomous vehicle, the research purpose of the surround view camera to generate a bird's-eye view feature is to simultaneously detect the three-dimensional targets and positions of the autonomous vehicle environment, which requires modeling the vehicle's surrounding environment. Through the data of the in-vehicle surround view camera, it can quickly integrate from a single perspective image to a dense bird's-eye image. The arrangement position of the in-vehicle surround view camera is relatively simple, and only multiple positions need to be evenly distributed. For example, in addition to arranging the surround view camera in the front and rear of the vehicle, four cameras are simultaneously arranged in the front left, rear left, front right, and rear right of the vehicle, for a total of six cameras. The semantic information is richer than that of four cameras, and the splicing is clearer, which can better detect the environment around the vehicle and position the vehicle itself.

[0067] Each frame of data input into the pre-constructed driving path planning model contains images from multiple angle cameras. These images will be used as raw data and input into the pre-constructed driving path planning model after conversion processing, that is, the data is undistorted images. After being processed by the pre-constructed driving path planning model, a series of target control instructions will be quickly output, and it will take over the entire vehicle on behalf of the driver. Among them, it includes interacting with the gear, controlling gear shifting, controlling the power system to increase the vehicle speed, at the same time controlling the braking system to reduce the vehicle speed, and controlling the steering system to change the direction angle to avoid obstacles. And these systems and the pre-constructed driving path planning model interact with each other, and signal interaction is carried out immediately to issue reasonable instructions.

[0068] The network training model in the pre-constructed driving path planning model, as a processor in the process from input image to output result, has a set of multi-view image data at the input end, and includes the length and width of the image and the camera position of the image. In the first step, preprocessing will be performed on the input data, checking the time and sequence of each frame of data, and adjusting it if it is inconsistent. In the second step, feature extraction is performed on the data with consistent timing. The extracted image features enter the backbone network for training, and then the image features are output. Weighted calculation and normalization processing are performed on the image features. The obtained image features are processed by an attention module to increase the weights of those significant features, and then weighted calculation and normalization processing are performed on the image features again. At this time, a bird's-eye view feature of a three-dimensional variable can be output, and then the detection and segmentation results are output. Based on the above results, path planning and control decisions are made to control the entire vehicle to enter the drivable area, bringing a comfortable and safe driving experience to users.

[0069] In summary, the driving path planning method according to the embodiment of the present invention includes: obtaining multiple driving environment images, inputting the multiple driving environment images into the pre-constructed driving path planning model to obtain target control instructions; controlling the steering system, power system, and braking system of the vehicle according to the target control instructions so that the vehicle travels along the target driving path. Thus, this method can accurately detect the obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the drivable area, control the vehicle's lateral and longitudinal systems based on the path planning results, provide a safer autonomous driving service for users, with simple sensor deployment, stable data transmission, strong environmental perception ability, and reduced usage costs.

[0070] It should be noted that the method of the embodiment of the present invention can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by the cooperation of multiple devices. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiment of the present invention, and these multiple devices will interact with each other to complete the above method.

[0071] It should be noted that some embodiments of the present invention have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0072] Corresponding to the above embodiments, the present invention also proposes a driving path planning device.

[0073] As Figure 4 shown, the driving path planning device of the embodiment of the present invention includes: an acquisition module 410 and a control module 420.

[0074] Among them, the acquisition module 410 is configured to acquire multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control instruction; the control module 420 is configured to control the steering system, power system, and braking system of the vehicle according to the target control instruction, so that the vehicle travels along the target driving path.

[0075] In some embodiments of the present invention, the acquisition module 410 is further configured to obtain multiple groups of environment images through multiple groups of shooting devices preset on the vehicle, and obtain multiple groups of environment bird's-eye view images according to the multiple groups of environment images; wherein, each group of environment images includes multiple environment images, and each group of environment images records the driving environment around the vehicle by 360°; divide the multiple groups of environment bird's-eye view images according to a preset ratio to obtain multiple groups of training environment images, multiple groups of verification environment images, and multiple groups of test environment images.

[0076] In some embodiments of the present invention, the driving path planning model includes a verified classification neural network model; the obtaining module 410 is further configured to train a basic neural network model through multiple groups of training environmental images; wherein, the basic neural network model is used to extract environmental elements in the training environmental images and classify the environmental elements; an attention module is introduced into the basic neural network model, and the classification label set of the environmental elements is embedded into the attention module to obtain an initial attention module; a preset classification task is set, and the preset classification task is input into the initial attention module, and the preset classification task is associated with the classification label set to obtain a trained classification neural network model; wherein, the preset classification task indicates the importance of each label in the classification label set.

[0077] In some embodiments of the present invention, the obtaining module 410 is further configured to verify the trained classification neural network model through multiple groups of verification environmental images, and obtain a verified classification neural network model in response to the loss value of the loss function of the trained classification neural network model reaching a target threshold.

[0078] In some embodiments of the present invention, the obtaining module 410 is further configured to test the verified classification neural network model through multiple groups of test environmental images, and obtain a verified classification neural network model in response to the classification accuracy and weight assignment accuracy of the environmental elements in the multiple groups of test environmental images by the verified classification neural network model reaching a preset standard.

[0079] In some embodiments of the present invention, the target control instruction includes at least one of a steering instruction, an acceleration instruction, and a deceleration instruction; the control module 420 inputs multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction, and specifically is configured to: input multiple driving environment images into the driving path planning model, and obtain a target environmental bird's-eye view image according to the multiple driving environment images; extract the target environmental features of the environmental elements in the target environmental bird's-eye view image, classify the target environmental features, and assign weights to each type of target environmental feature; generate a target driving path and a steering instruction, an acceleration instruction, or a deceleration instruction according to the classified target environmental features.

[0080] In some embodiments of the present invention, the target environmental feature includes at least one of a driving turning indication feature, a driving progress indication feature, and an obstacle feature; wherein, the driving turning indication feature is used to indicate the generation of a steering command, the driving progress indication feature is used to indicate the generation of an acceleration command, and the obstacle feature is used to indicate the generation of a deceleration command; the control module 420 controls the steering system, the power system, and the braking system of the vehicle according to the target control command, so that the vehicle travels along the target driving path, specifically for: sending a steering command to the steering system, so that the steering system controls the vehicle to travel along the target driving path according to the steering command; and / or sending a progress command to the power system, so that the power system controls the vehicle to travel along the target driving path according to the power command; and / or sending a deceleration command to the braking system, so that the braking system controls the vehicle to travel along the target driving path according to the deceleration command.

[0081] It should be noted that for the details not disclosed in the driving path planning device of the embodiments of the present invention, please refer to the details disclosed in the driving path planning method of the embodiments of the present invention, and will not be elaborated herein.

[0082] In summary, according to the driving path planning device of the embodiments of the present invention, it includes: an acquisition module configured to acquire multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control command; a control module configured to control the steering system, the power system, and the braking system of the vehicle according to the target control command, so that the vehicle travels along the target driving path. Thus, this device can accurately detect the obstacle information in the vehicle's surrounding environment, perform three-dimensional target classification and positioning information output, realize the modeling of the surrounding environment of the autonomous vehicle, perform system feedback on the feasible driving area, control the vehicle's lateral and longitudinal systems based on the path planning result, provide a safer autonomous driving service for users, have simple sensor deployment, stable data transmission, strong environmental perception ability, and reduce the usage cost.

[0083] For the convenience of description, when describing the above system, various modules are described separately according to their functions. Of course, when implementing the present invention, the functions of each module can be implemented in one or more software and / or hardware.

[0084] The system of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein.

[0085] Corresponding to the above embodiments, the present invention also proposes an electronic device.

[0086] Reference Figure 5, a block diagram of an electronic device according to some embodiments of the present invention, showing a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 510, a memory 520, an input / output interface 530, a communication interface 540, and a bus 550. Among them, the processor 510, the memory 520, the input / output interface 530, and the communication interface 540 are communicatively connected to each other inside the device through the bus 550.

[0087] The processor 510 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0088] The memory 520 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 520 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 520 and are called and executed by the processor 510.

[0089] The input / output interface 530 is used to connect to an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0090] The communication interface 540 is used to connect to a communication module (not shown in the figure) to implement communication and interaction between this device and other devices. Among them, the communication module can communicate through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.).

[0091] The bus 550 includes a path for transmitting information between various components of the device (such as the processor 510, the memory 520, the input / output interface 530, and the communication interface 540).

[0092] It should be noted that although the above device only shows the processor 510, the memory 520, the input / output interface 530, the communication interface 540, and the bus 550, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0093] The electronic device of the above embodiment is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0094] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method of any of the above embodiments.

[0095] The above computer-readable storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSD)).

[0096] The computer instructions stored in the storage medium of the above embodiment are used to cause a computer to execute the method of any of the above exemplary method embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0097] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the shown operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be changed in the order of execution. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0098] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0099] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains. The "first", "second", and similar terms used in the embodiments of the present invention do not denote any order, quantity, or importance, but are only used to distinguish different components. The terms "comprising" or "including" and the like mean that the elements or items appearing before the term cover the elements or items listed after the term and their equivalents, without excluding other elements or items. The terms "connected" or "coupled" and the like are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", etc. are only used to indicate relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0100] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefits. This division is only for convenience of expression. The present invention aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A driving route planning method, characterized in that, Including: Obtain multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control instruction; Control the steering system, power system, and braking system of the vehicle according to the target control instruction, so that the vehicle travels along a target driving path.

2. The driving path planning method according to claim 1, wherein The method further includes: Obtain multiple groups of environment images through multiple groups of shooting devices preset on the vehicle, and obtain multiple groups of environment bird's-eye view images according to the multiple groups of environment images; wherein, each group of environment images includes multiple environment images, and each group of environment images records the driving environment around the vehicle by 360°; Divide the multiple groups of environment bird's-eye view images according to a preset ratio to obtain multiple groups of training environment images, multiple groups of verification environment images, and multiple groups of test environment images.

3. The driving path planning method according to claim 2, characterized in that, The driving path planning model includes a verified classification neural network model; The method further includes: Train a basic neural network model through the multiple groups of training environment images; wherein, the basic neural network model is used to extract environmental elements in the training environment images and classify the environmental elements; Introduce an attention module into the basic neural network model, and embed the classification label set of the environmental elements into the attention module to obtain an initial attention module; Set a preset classification task, input the preset classification task into the initial attention module, and associate the preset classification task with the classification label set to obtain a trained classification neural network model; wherein, the preset classification task indicates the importance of each label in the classification label.

4. The driving path planning method according to claim 3, wherein The method further includes: Verify the trained classification neural network model through the multiple groups of verification environment images, and obtain a verified classification neural network model in response to the loss value of the loss function of the trained classification neural network model reaching a target threshold.

5. The driving path planning method according to claim 4, characterized in that The method further includes: Test the verified classification neural network model through the multiple groups of test environment images, and obtain the verified classification neural network model in response to the classification accuracy and weight assignment accuracy of the environmental elements in the multiple groups of test environment images by the verified classification neural network model reaching a preset standard.

6. The driving path planning method according to claim 5, wherein, The target control instruction includes at least one of a steering instruction, an acceleration instruction, and a deceleration instruction; The step of inputting the multiple driving environment images into a pre-constructed driving path planning model to obtain a target control instruction includes: Input the multiple driving environment images into the driving path planning model, and obtain a target environment bird's-eye view image according to the multiple driving environment images; Extract the target environmental features of the environmental elements in the target environment bird's-eye view image, classify the target environmental features, and assign weights to each target environmental feature; Generate a target driving path and the steering instruction, the acceleration instruction, or the deceleration instruction according to the classified target environmental features.

7. The driving path planning method according to claim 6, wherein The target environmental features include at least one of a driving turn indication feature, a driving progress indication feature, and an obstacle feature; wherein, the driving turn indication feature is used to indicate the generation of the steering instruction, the driving progress indication feature is used to indicate the generation of the acceleration instruction, and the obstacle feature is used to indicate the generation of the deceleration instruction; Controlling the steering system, power system, and braking system of the vehicle according to the target control instruction so that the vehicle travels along a target driving path includes: Sending the steering instruction to the steering system so that the steering system controls the vehicle to travel along the target driving path according to the steering instruction; Or / and sending the progress instruction to the power system so that the power system controls the vehicle to travel along the target driving path according to the power instruction; Or / and sending the deceleration instruction to the braking system so that the braking system controls the vehicle to travel along the target driving path according to the deceleration instruction.

8. A driving route planning device, characterized in that Comprising: An acquisition module, configured to acquire multiple driving environment images, input the multiple driving environment images into a pre-constructed driving path planning model, and obtain a target control instruction; A control module, configured to control the steering system, power system, and braking system of the vehicle according to the target control instruction so that the vehicle travels along a target driving path.

9. An electronic device, characterized in that, Comprising: A processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the driving path planning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the driving path planning method according to any one of claims 1 to 7 are implemented.

Images