Path determination method, system and device and nonvolatile storage medium

Abstract

The invention discloses a path determination method, system and device and a nonvolatile storage medium. The method comprises the steps of receiving image data of a scene where a target vehicle is located at the current moment; inputting the image data into a preset convolutional neural network model to obtain a scene feature map; road information of a scene where the target vehicle is located is determined according to the scene feature map, and the road information represents the road condition of the scene where the target vehicle is located; the historical driving state of the target vehicle is obtained, and the historical driving state comprises the driving speed and the driving direction of the target vehicle in the historical time period before the current moment; and determining a future path of the target vehicle based on the road information and the historical driving state. The technical problem that the planned path is not accurate enough due to the fact that current path planning depends on static sensing information and cannot be continuously updated to adapt to a dynamically changing traffic environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a path determination method, system, device and non-volatile storage medium. Background Art

[0002] Most current path planning models are based on independent post-processing of perception results, lacking in-depth use of real-time perception information. This separation results in perception information not being fully used in path planning. Path planning can only be based on the static perception results of the current environment, and cannot continuously pay attention to the dynamic changes of the environment, resulting in the planned path being inflexible or inaccurate.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] Embodiments of the present invention provide a path determination method, system, device and non-volatile storage medium to at least solve the technical problem that current path planning relies on static perception information and cannot be continuously updated to adapt to the dynamically changing traffic environment, resulting in the planned path being inaccurate.

[0005] According to one aspect of an embodiment of the present invention, a path determination method is provided, comprising: receiving image data of a scene in which a target vehicle is located at a current moment; inputting the image data into a preset convolutional neural network model to obtain a scene feature map; determining road information of the scene in which the target vehicle is located based on the scene feature map, wherein the road information characterizes the road conditions of the scene in which the target vehicle is located; obtaining a historical driving status of the target vehicle, wherein the historical driving status includes a driving speed and a driving direction of the target vehicle in a historical period before the current moment; and determining a future path of the target vehicle based on the road information and the historical driving status.

[0006] Optionally, determining road information of a scene in which a target vehicle is located according to a scene feature map includes: identifying a target object in the scene feature map, wherein the target object includes at least one of the following: a road, a traffic sign, a vehicle; and determining road information based on the target object.

[0007] Optionally, determining road information based on the target object includes: when the target object includes a traffic sign, identifying a target traffic sign in the traffic sign; determining a target traffic rule corresponding to the target traffic sign based on a preset correspondence between the traffic sign and the traffic rule; and determining road information based on the target traffic rule.

[0008] Optionally, based on the road information and historical driving status, the future path of the target vehicle is determined, including: inputting the road information and historical driving status into a long short-term memory network model, and outputting the path points at the future time; based on the path points at the future time, determining the future path.

[0009] Optionally, based on the future path, a control signal is generated, wherein the control signal is used to control the direction and speed of travel of the target vehicle; and the control signal is sent to a control system of the target vehicle.

[0010] According to another aspect of an embodiment of the present invention, a path determination system is also provided, including: an image input module, used to receive image data of a scene in which a target vehicle is located at a current moment; a feature extraction module, used to input the image data into a preset convolutional neural network model to obtain a scene feature map, and determine road information of the scene in which the target vehicle is located based on the scene feature map, wherein the road information characterizes the road conditions of the scene in which the target vehicle is located; and a path planning module, used to obtain a historical driving status of the target vehicle, and determine a future path of the target vehicle based on the road information and the historical driving status.

[0011] Optionally, the output control module is used to generate a control signal based on the future path, wherein the control signal is used to control the direction and speed of the target vehicle, and send the control signal to the control system of the target vehicle.

[0012] According to another aspect of an embodiment of the present invention, a path determination device is also provided, including: a receiving module, used to receive image data of a scene in which a target vehicle is located at a current moment; an input module, used to input the image data into a preset convolutional neural network model to obtain a scene feature map; a first determination module, used to determine road information of the scene in which the target vehicle is located according to the scene feature map, wherein the road information characterizes the road conditions of the scene in which the target vehicle is located; an acquisition module, used to acquire a historical driving status of the target vehicle, wherein the historical driving status includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; and a second determination module, used to determine the future path of the target vehicle based on the road information and the historical driving status.

[0013] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above path determination methods.

[0014] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above path determination methods is executed.

[0015] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above path determination methods is implemented.

[0016] In an embodiment of the present invention, a path determination method is adopted, by receiving image data of a scene in which a target vehicle is located at a current moment; inputting the image data into a preset convolutional neural network model to obtain a scene feature map; determining road information of a scene in which a target vehicle is located based on the scene feature map, wherein the road information characterizes the road condition of the scene in which the target vehicle is located; obtaining a historical driving state of the target vehicle, wherein the historical driving state includes a driving speed and a driving direction of the target vehicle in a historical period before the current moment; determining a future path of the target vehicle based on the road information and the historical driving state, thereby achieving the purpose of determining a future path by detecting road conditions in real time, thereby realizing the technical effect of improving the accuracy of the determined future path, and further solving the technical problem that the current path planning relies on static perception information and cannot be continuously updated to adapt to the dynamically changing traffic environment, resulting in an inaccurate planned path. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a path determination method is shown;

[0019] Figure 2 is a schematic diagram of a flow chart of a path determination method provided according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a path determination system provided according to an embodiment of the present invention;

[0021] Figure 4 is a structural block diagram of a path determination device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] According to an embodiment of the present invention, a method embodiment of a path determination method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a path determination method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0026] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0027] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the path determination method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the path determination method of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0029] Figure 2 FIG. 1 is a flow chart of a path determination method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0030] Step S202, receiving image data of the scene where the target vehicle is located at the current moment.

[0031] In this step, the traffic scene image data in front of the target vehicle can be collected by a camera. For example, the environment image in front of the vehicle can be captured in real time by the camera to generate continuous image frames. The image data in front of the vehicle is captured in real time to facilitate the subsequent analysis of the traffic conditions in the scene to determine the future path.

[0032] Step S204: input the image data into a preset convolutional neural network model to obtain a scene feature map.

[0033] In this step, the image data can be input into the convolutional neural network model. For example, the image data is input into the CNN model to extract a variety of traffic environment information from the input image, such as detection of pedestrians, vehicles, traffic signs, etc., and semantic segmentation of roads. This module can simultaneously complete multiple visual perception tasks in a multi-task learning manner by sharing feature extraction layers. Specifically, a deep convolutional neural network can be used as a feature extractor to extract high-dimensional features of traffic scene images to obtain scene feature maps. CNN models usually include multiple convolutional layers, pooling layers, fully connected layers, and possible normalization layers. Input the image data into the CNN model and start the forward propagation process. The convolutional layer in the model uses a learnable filter (i.e., a convolution kernel) to scan the image and extract local features in the image, such as edges, textures, etc. Through multi-layer convolution and pooling operations, a multi-scale feature representation of the image can be constructed. In the deep layer of CNN, the model extracts more abstract and complex features, which contain target, region, and structural information in the image. Usually, these features are gradually abstracted and condensed through a series of convolution and pooling layers, and finally form a fixed-size feature map, namely the scene feature map. The scene feature map is the result of CNN's deep feature extraction of the input image. It converts the information in the image into a set of digital features that can represent the key visual information in the image, such as the position and shape of pedestrians, vehicles, traffic signs, etc., as well as the structure and segmentation of the road.

[0034] Step S206, determining the road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information represents the road condition of the scene where the target vehicle is located.

[0035] In this step, the road information of the scene where the target vehicle is located is determined based on the scene feature map. For example, the spatial resolution of the feature map can be restored and refined from the high-level feature map of the CNN using a specific decoder (such as a deconvolution layer, an upsampling layer, etc.). The restored feature map is processed by the road segmentation task head. This usually involves a fully convolutional network (FCN) or similar structure, where each pixel is classified as a road or non-road area. The model predicts the probability of each pixel belonging to the road based on the information in the scene feature map, such as the edge, texture, color, etc. of the road. The result of the road segmentation may contain some noise or discontinuous areas, so post-processing is required, such as using a conditional random field (CRF) for refinement, or using morphological operations (such as opening and closing operations) to clean up the segmentation results to ensure the coherence and accuracy of the road area. The geometric information of the road is extracted from the road segmentation map, such as the width, curvature, slope, etc. of the road. This step may involve image processing techniques such as contour detection and linear feature extraction to identify the boundaries and center lines of the road from the binary image. Combined with other information in the scene feature map, such as pedestrians, vehicles, traffic signs, etc., the understanding of road information is enhanced. For example, the identified traffic signs can provide additional information about road conditions (such as speed limits, no overtaking), and the positions of pedestrians and vehicles help understand the use of the road. Finally, the above information is integrated to form a comprehensive representation of the road conditions, including but not limited to the type of road (such as highways, urban roads), current lane information, obstacles ahead, traffic rules, etc. This information is crucial for the path planning and decision-making of autonomous driving vehicles, ensuring that the vehicle can travel safely and efficiently on the road. Through this series of steps, the road information of the scene where the target vehicle is located is extracted and determined from the scene feature map, which can not only identify the physical boundaries of the road, but also understand the dynamic conditions of the road, providing an important basis for subsequent path planning and autonomous driving decisions.

[0036] Step S208, obtaining the historical driving status of the target vehicle, wherein the historical driving status includes the driving speed and driving direction of the target vehicle in a historical period before the current moment.

[0037] In this step, the historical driving status of the target vehicle is obtained, wherein the historical driving status may include driving speed and driving direction, which are subsequently combined with the current traffic conditions to determine the future path. Driving data within a historical period can be collected from the vehicle's sensor suite, wherein the vehicle's instantaneous speed can be measured by a speed sensor, and a direction sensor, such as a gyroscope or a magnetometer, is used to determine the vehicle's driving direction. The GPS module provides the vehicle's geographic location information, which can be indirectly used to calculate the driving direction and speed. The wheel speed sensor measures the wheel rotation speed, and combined with the wheel diameter and the vehicle's steering angle, the actual driving speed and direction can be calculated.

[0038] Data recording and storage: Through the vehicle's internal storage or cloud storage, the historical driving status data is recorded and stored. The collected data can be organized into a time series data set. Each record contains the continuous driving status within a fixed historical period, providing input for the path planning model. The selection of this historical period needs to be determined based on actual needs and the prediction ability of the model, and is generally a few seconds to tens of seconds in length.

[0039] Through the above steps, the historical driving status of the target vehicle can be accurately obtained and utilized, which is of great significance for the autonomous driving system to predict vehicle behavior and plan safe paths. Models such as LSTM can generate future paths that are more in line with the actual driving conditions of the vehicle based on the vehicle's past driving patterns and combined with real-time environmental perception information, thereby improving the safety and robustness of autonomous driving.

[0040] Step S210, determining the future path of the target vehicle based on the road information and the historical driving status.

[0041] In this step, the future path of the target vehicle is determined based on the road information and historical driving status. Machine learning models, such as long short-term memory networks (LSTMs), can be used to process this information and generate a predicted path. The road information (road boundaries, road attributes, obstacle information, etc.) extracted from the scene feature map can be combined with the historical driving status of the target vehicle (driving speed, driving direction, etc.) to form a comprehensive input data set. These data need to be organized in a time series format so that the LSTM model can understand its time dependencies. The integrated data is input into the LSTM-based path planning model. LSTM is able to process sequence data and remember long-term dependencies, which is particularly important for predicting the future path of the vehicle because it needs to take into account the impact of past driving patterns on the future behavior of the vehicle. LSTM uses its internal gating mechanisms, such as input gates, forget gates, and output gates, to selectively remember and update the historical driving status information of the vehicle. Because the LSTM model is able to process sequence data, it can dynamically respond to changes in road information. For example, if a pedestrian or vehicle is detected ahead, the model will adjust the predicted path to avoid a collision; if the road suddenly narrows or a curve appears, the model will also adjust the path prediction accordingly. Based on the current road information and historical driving status, the LSTM model generates a path prediction for the next few seconds or minutes. The predicted path is usually a series of future position points or direction vectors that represent how the vehicle will travel on the road. The generated path prediction may need to be further optimized and corrected to ensure safety and feasibility. For example, an optimization algorithm is used to adjust the path to avoid conflicts with detected obstacles, or to ensure that the path complies with traffic regulations and road restrictions (such as lane lines, speed limits, etc.).

[0042] Through this series of steps, future path prediction based on road information and historical driving status can provide autonomous driving vehicles with dynamic and adaptable driving routes, improving their safety and decision-making capabilities in complex traffic environments.

[0043] Through the above steps, the purpose of real-time detection of road conditions to determine the future path is achieved, thereby achieving the technical effect of improving the accuracy of the determined future path, and further solving the technical problem that the current path planning relies on static perception information and cannot be continuously updated to adapt to the dynamically changing traffic environment, resulting in the planned path being inaccurate.

[0044] As an optional embodiment, determining road information of a scene in which a target vehicle is located according to a scene feature map includes: identifying a target object in the scene feature map, wherein the target object includes at least one of the following: a road, a traffic sign, a vehicle; and determining road information based on the target object.

[0045] Optionally, the road information of the scene where the target vehicle is located is determined based on the scene feature map, which involves a multi-task perception process. A convolutional neural network (CNN) can be used to detect and classify the target object in the scene feature map. This step usually uses an object detection algorithm such as YOLO, Faster R-CNN or SSD, which can identify multiple objects in the image, including roads, traffic signs, vehicles, etc., and give the location (bounding box) and category of each object. In addition to object detection, semantic segmentation is also required to understand the road information more finely. Semantic segmentation models (such as DeepLab, FCN, UNet, etc.) can classify each pixel of the image into a specific category, such as "road", "sidewalk", "lane line", "traffic sign", etc. This provides a basis for the subsequent determination of the specific attributes of the road. Extract the characteristics of the road from the results of semantic segmentation, including the width of the road, the location of the lane line, the curvature of the road, etc. This step may require the use of image processing techniques such as edge detection, contour analysis, Hough transform, etc., to further refine the segmentation results and extract the geometric features of the road. Based on the detected traffic signs and sign classification information, the traffic rules of the current road are identified, such as speed limit, no overtaking, right of way, etc. This helps to plan a path that is more in line with traffic rules. For detected vehicles and pedestrians, their moving direction and speed are analyzed, and their future positions are predicted to avoid potential collisions. All the above information is integrated to form a comprehensive understanding of the road conditions, including static road properties and dynamic traffic conditions. This can be used as input for path planning to help plan a driving path that complies with traffic rules and avoids obstacles.

[0046] Through the above steps, object detection, semantic segmentation and other visual tasks based on scene feature maps can provide key road and environment information for path planning, making path planning more accurate and safe, especially in complex and dynamic traffic scenarios.

[0047] As an optional embodiment, determining road information based on a target object includes: when the target object includes a traffic sign, identifying a target traffic sign in the traffic sign; determining a target traffic rule corresponding to the target traffic sign based on a preset correspondence between the traffic sign and the traffic rule; and determining road information based on the target traffic rule.

[0048] Optionally, the road information is determined based on the target object, especially when the scene feature map contains traffic signs, the traffic signs can be identified and the corresponding road rules can be determined, so as to further clarify the road information. Object detection algorithms (such as YOLO, Faster R-CNN, etc.) can be used to locate and identify all traffic signs in the scene feature map. This step focuses on determining the position and shape boundaries of the road signs. For each detected traffic sign, image recognition technology, such as convolutional neural network (CNN), is applied for refinement and classification to identify the specific type of traffic sign. For example, the identified sign can be a speed limit sign, a no-turn sign, a road information sign, etc. According to the identified traffic sign, a preset database of traffic signs and traffic rules is consulted. This database usually contains all possible traffic signs and their meanings, such as the maximum speed limit corresponding to the speed limit sign, the direction restriction corresponding to the no-turn sign, etc. Based on the parsed traffic rules, combined with the current position and driving direction of the vehicle, the road information is determined. For example, if a speed limit sign is identified, the speed limit rule needs to be applied to the current road information to adjust the vehicle's driving speed. If a no-turn sign is detected, the driver needs to plan a route to avoid the turn or prepare to change lanes in advance if appropriate. Identified road information signs (such as construction ahead, road narrowing) can warn the driver in advance or adjust the autonomous driving plan to cope with road changes.

[0049] During the driving process, the vehicle continuously detects and identifies traffic signs, updates road information and plans routes in real time. This enables the autonomous driving system to adapt to the ever-changing traffic environment and improve its safety and efficiency under complex conditions. Through the above steps, the system can not only identify traffic signs and logos, but also interpret their meanings and convert them into a clear description of road information, thereby guiding the vehicle to comply with traffic rules and make reasonable route planning decisions. This helps to enhance the vehicle's autonomous driving capabilities and safety.

[0050] As an optional embodiment, the future path of the target vehicle is determined based on road information and historical driving status, including: inputting the road information and historical driving status into a long short-term memory network model, and outputting the path points at the future time; and determining the future path based on the path points at the future time.

[0051] Optionally, the process of determining the future path of the target vehicle based on road information and historical driving status can be implemented through a long short-term memory network (LSTM) model. The road information (such as lane line position, road width, traffic signs, etc.) and historical driving status (including vehicle speed, acceleration, steering angle, and current position, etc.) can be first converted into a model-readable format, such as a numerical vector or feature matrix. These data may need to be standardized to ensure that the input of the model is within a suitable numerical range. The preprocessed data is then input into the LSTM model as a time series. The LSTM model is designed to handle this time-dependent input, memorizing and processing historical information through its internal gating mechanism (input gate, forget gate, and output gate), while responding to new information. Inside the LSTM, the input data undergoes the following calculation steps: The input gate determines which information needs to be stored in the cell state of the LSTM. The forget gate determines which information should be forgotten from the cell state to adapt to the current environmental changes. The output gate determines which information will be used as the output of the current time step, which will be used to generate the predicted path points. The LSTM model predicts waypoints at future moments based on input road information and historical driving status. These waypoints are based on the vehicle's past behavior patterns and current road conditions, including the vehicle's ideal position and direction on the road. Waypoints may be represented as two-dimensional coordinates (x, y) or angles and distances relative to the current vehicle position. The predicted future waypoints may need to be smoothed to generate a continuous and feasible driving path. This can be achieved by applying smoothing algorithms (such as spline curve fitting, moving average filtering, etc.) to ensure smooth transitions between waypoints and avoid abrupt turns or speed changes. Connect the smoothed waypoints to form a continuous future driving path. This may involve converting waypoints into a series of direction vectors or curves to guide the vehicle's driving direction and speed adjustments.

[0052] Through the above process, the LSTM model can predict and determine the future path based on the vehicle's historical driving status and real-time road information, providing dynamic and real-time path planning for autonomous vehicles. This path planning method that combines deep learning and time series analysis can significantly improve the adaptability and safety of vehicles in complex traffic environments.

[0053] As an optional embodiment, a control signal is generated based on the future path, wherein the control signal is used to control the direction and speed of the target vehicle; and the control signal is sent to a control system of the target vehicle.

[0054] Optionally, the generated future path can be converted into a control signal to guide the actual steering, acceleration or deceleration operation of the vehicle to ensure that the vehicle drives safely along the planned path. The predicted future path can be composed of a series of path points, which need to be converted into a continuous curve or trajectory through smoothing to adapt to the actual driving characteristics of the vehicle. The smoothed future path can be converted into a control signal using a path tracking algorithm (such as PID controller, LQR controller, model predictive control MPC, etc.). These algorithms calculate the steering angle and speed adjustment required for the vehicle to reach each point on the path. When generating control signals, the dynamic characteristics of the vehicle, such as maximum acceleration, maximum steering angle, vehicle response time, etc., need to be considered. Dynamic programming can ensure that the generated control signal does not exceed the physical limitations of the vehicle and can quickly adapt to changes in the path. Ultimately, the path tracking control algorithm will generate specific control signals, including but not limited to: steering angle signals for guiding the vehicle's steering system to adjust the driving direction. Speed ​​signals for adjusting the acceleration or deceleration of the vehicle to achieve the speed requirements on the predicted path. The generated control signals can be sent to the vehicle's control system, including the steering system, acceleration system, and braking system, through the vehicle's internal communication network (such as CAN bus). These systems will perform corresponding operations based on the control signals they receive, such as adjusting the steering angle, controlling the throttle or braking, to enable the vehicle to travel along the predicted path.

[0055] Through the above steps, the control signal generation and execution based on the future path can enable the autonomous driving vehicle to automatically adjust the driving direction and speed on complex and changeable roads to ensure safe and efficient arrival at the destination.

[0056] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0057] Through the description of the above implementation methods, those skilled in the art can clearly understand that the path determination method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0058] According to an embodiment of the present invention, a path determination system for implementing the above path determination method is also provided. Figure 3 is a schematic diagram of a path determination system provided according to an embodiment of the present invention. Figure 3 As shown, including:

[0059] The image input module is used to receive image data of the scene where the target vehicle is located at the current moment.

[0060] The feature extraction module is used to input the image data into a preset convolutional neural network model to obtain a scene feature map, and determine the road information of the scene where the target vehicle is located based on the scene feature map, wherein the road information represents the road conditions of the scene where the target vehicle is located.

[0061] The path planning module is used to obtain the historical driving status of the target vehicle and determine the future path of the target vehicle based on the road information and the historical driving status.

[0062] The above system may further include an output control module for generating a control signal based on the future path, wherein the control signal is used to control the direction and speed of the target vehicle, and sending the control signal to a control system of the target vehicle.

[0063] To give a specific embodiment, the path determination system may include an image input module, a multi-task perception module, a path planning module, and an output control module.

[0064] The image input module is responsible for receiving the traffic scene image data in front of the vehicle. This module uses the camera to capture the environment image in front of the vehicle in real time, generates continuous image frames, and provides them to the multi-task perception module for processing.

[0065] The multi-task perception module is based on a convolutional neural network (CNN) to extract a variety of traffic environment information from the input image, such as the detection of pedestrians, vehicles, traffic signs, etc., as well as the semantic segmentation of roads. This module completes multiple visual perception tasks simultaneously in a multi-task learning manner by sharing feature extraction layers.

[0066] The path planning module is based on the long short-term memory network (LSTM). It plans the future driving path based on the detection results and road segmentation results output by the multi-task perception module and the historical status of the vehicle. The LSTM network can predict dynamic changes during driving and plan a smooth and safe path through time series modeling.

[0067] The output control module is responsible for converting the path information generated by the path planning module into control signals to guide the vehicle's actual steering, acceleration or deceleration operations to ensure that the vehicle drives safely along the planned path.

[0068] According to an embodiment of the present invention, a path determination device for implementing the above path determination method is also provided. Figure 4 is a structural block diagram of a path determination device provided according to an embodiment of the present invention. Figure 4 As shown, the device includes: a receiving module 402, an input module 404, a first determining module 406, an acquiring module 408 and a second determining module 410. The device is described below.

[0069] The receiving module 402 is used to receive image data of the scene where the target vehicle is located at the current moment.

[0070] The input module 404 is connected to the receiving module 402 and is used to input the image data into a preset convolutional neural network model to obtain a scene feature map.

[0071] The first determination module 406 is connected to the input module 404 and is used to determine the road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information represents the road condition of the scene where the target vehicle is located.

[0072] The acquisition module 408 is connected to the first determination module 406 and is used to acquire the historical driving state of the target vehicle, wherein the historical driving state includes the driving speed and driving direction of the target vehicle in a historical period before the current moment.

[0073] The second determination module 410 is connected to the acquisition module 408 and is used to determine the future path of the target vehicle based on the road information and the historical driving status.

[0074] It should be noted that the receiving module 402, the input module 404, the first determination module 406, the acquisition module 408 and the second determination module 410 correspond to steps S202 to S210 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0075] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0076] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the path determination method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned path determination method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0077] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: receive image data of the scene where the target vehicle is located at the current moment; input the image data into a preset convolutional neural network model to obtain a scene feature map; determine the road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information characterizes the road conditions of the scene where the target vehicle is located; obtain the historical driving status of the target vehicle, wherein the historical driving status includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; determine the future path of the target vehicle based on the road information and the historical driving status.

[0078] Optionally, the processor may also execute program code of the following steps: determining road information of the scene in which the target vehicle is located based on a scene feature map, including: identifying a target object in the scene feature map, wherein the target object includes at least one of the following: a road, a traffic sign, a vehicle; and determining road information based on the target object.

[0079] Optionally, the processor may also execute program code for the following steps: determining road information based on the target object, including: identifying a target traffic sign in a traffic sign when the target object includes a traffic sign; determining a target traffic rule corresponding to the target traffic sign based on a preset correspondence between traffic signs and traffic rules; and determining road information based on the target traffic rule.

[0080] Optionally, the processor may also execute program code for the following steps: determining the future path of the target vehicle based on road information and historical driving status, including: inputting the road information and historical driving status into a long short-term memory network model, and outputting path points at future times; determining the future path based on the path points at future times.

[0081] Optionally, the processor may also execute program codes of the following steps: generating a control signal based on the future path, wherein the control signal is used to control the direction and speed of the target vehicle; and sending the control signal to a control system of the target vehicle.

[0082] By adopting an embodiment of the present invention, a path determination method is provided, which receives image data of a scene where a target vehicle is located at a current moment; inputs the image data into a preset convolutional neural network model to obtain a scene feature map; determines road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information characterizes the road condition of the scene where the target vehicle is located; obtains the historical driving state of the target vehicle, wherein the historical driving state includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; determines the future path of the target vehicle based on the road information and the historical driving state, thereby achieving the purpose of determining the future path by detecting the road condition in real time, thereby realizing the technical effect of improving the accuracy of the determined future path, and further solving the technical problem that the current path planning relies on static perception information and cannot be continuously updated to adapt to the dynamically changing traffic environment, resulting in the planned path being not accurate enough.

[0083] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0084] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the path determination method provided in the above embodiment.

[0085] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0086] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: receiving image data of a scene in which the target vehicle is located at a current moment; inputting the image data into a preset convolutional neural network model to obtain a scene feature map; determining road information of the scene in which the target vehicle is located based on the scene feature map, wherein the road information characterizes the road conditions of the scene in which the target vehicle is located; obtaining a historical driving status of the target vehicle, wherein the historical driving status includes a driving speed and a driving direction of the target vehicle in a historical period before the current moment; and determining a future path of the target vehicle based on the road information and the historical driving status.

[0087] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining road information of a scene in which a target vehicle is located based on a scene feature map, including: identifying a target object in the scene feature map, wherein the target object includes at least one of the following: a road, a traffic sign, a vehicle; and determining road information based on the target object.

[0088] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining road information based on the target object, including: when the target object includes a traffic sign, identifying a target traffic sign in the traffic sign; determining a target traffic rule corresponding to the target traffic sign based on a preset correspondence between traffic signs and traffic rules; and determining road information based on the target traffic rule.

[0089] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining the future path of the target vehicle based on road information and historical driving status, including: inputting the road information and historical driving status into a long short-term memory network model, and outputting path points at future times; determining the future path based on the path points at future times.

[0090] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: generating a control signal based on the future path, wherein the control signal is used to control the direction and speed of travel of the target vehicle; and sending the control signal to a control system of the target vehicle.

[0091] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: receiving image data of a scene in which a target vehicle is located at a current moment; inputting the image data into a preset convolutional neural network model to obtain a scene feature map; determining road information of the scene in which the target vehicle is located based on the scene feature map, wherein the road information characterizes the road conditions of the scene in which the target vehicle is located; obtaining a historical driving status of the target vehicle, wherein the historical driving status includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; and determining the future path of the target vehicle based on the road information and the historical driving status.

[0092] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A path determination method, characterized in that: include: Receive image data of the scene where the target vehicle is located at the current moment; Inputting the image data into a preset convolutional neural network model to obtain a scene feature map; Determining road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information represents the road condition of the scene where the target vehicle is located; Acquire the historical driving state of the target vehicle, wherein the historical driving state includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; Based on the road information and the historical driving status, a future path of the target vehicle is determined.

2. The method according to claim 1, characterized in that Determining the road information of the scene where the target vehicle is located according to the scene feature map includes: Identify a target object in the scene feature map, wherein the target object includes at least one of the following: a road, a traffic sign, and a vehicle; Based on the target object, the road information is determined.

3. The method according to claim 2, characterized in that The determining the road information based on the target object includes: In the case where the target object includes a traffic sign, identifying a target traffic sign in the traffic sign; Determining a target traffic rule corresponding to the target traffic sign based on a preset correspondence between the traffic sign and the traffic rule; Based on the target traffic regulation, the road information is determined.

4. The method according to claim 1, characterized in that: The determining the future path of the target vehicle based on the road information and the historical driving state includes: Input the road information and the historical driving status into a long short-term memory network model, and output the path points at a future time; The future path is determined based on the path points at the future time.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Based on the future path, generating a control signal, wherein the control signal is used to control the direction and speed of the target vehicle; The control signal is sent to a control system of the target vehicle.

6. A path determination system, characterized in that: The path determination method according to any one of claims 1 to 5 above comprises: An image input module is used to receive image data of the scene where the target vehicle is located at the current moment; A feature extraction module, used to input the image data into a preset convolutional neural network model to obtain a scene feature map, and determine the road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information represents the road conditions of the scene where the target vehicle is located; The path planning module is used to obtain the historical driving status of the target vehicle and determine the future path of the target vehicle based on the road information and the historical driving status.

7. The system according to claim 6, characterized in that Also includes: The output control module is used to generate a control signal based on the future path, wherein the control signal is used to control the direction and speed of the target vehicle, and send the control signal to the control system of the target vehicle.

8. A path determination device, characterized in that: include: A receiving module, used to receive image data of the scene where the target vehicle is located at the current moment; An input module, used to input the image data into a preset convolutional neural network model to obtain a scene feature map; A first determination module, configured to determine road information of the scene where the target vehicle is located according to the scene feature map, wherein the road information represents the road condition of the scene where the target vehicle is located; An acquisition module, used for acquiring the historical driving state of the target vehicle, wherein the historical driving state includes the driving speed and driving direction of the target vehicle in a historical period before the current moment; The second determination module is used to determine the future path of the target vehicle based on the road information and the historical driving status.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the path determination method according to any one of claims 1 to 5.

10. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the path determination method according to any one of claims 1 to 5.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the path determination method according to any one of claims 1 to 5 is implemented.

Images