Navigation method and device

Abstract

The embodiment of the invention provides a navigation method and device which are used for providing lane-level navigation guidance, so that more accurate navigation guidance is provided, and the accuracy of vehicle driving is improved. The method comprises the steps that firstly, navigation data and lane sensing data are obtained, the navigation data comprise navigation paths when a vehicle runs in a map and can comprise paths generated on the basis of a driving starting point and a destination of the vehicle, and the lane sensing data comprise information of at least one lane obtained on the basis of information collected in the environment where the vehicle is located; the information of at least one lane determined in environmental perception data collected by a sensor in the vehicle can be specifically included; then navigation guidance is generated according to the navigation data and the lane sensing data, the navigation guidance is displayed, the navigation guidance can be used for identifying a drivable lane corresponding to the navigation path in the environment, and the drivable lane comprises a vehicle drivable path obtained according to the lane sensing data.

Description

Technical Field

[0001] This application relates to the field of vehicles, and in particular, to a navigation method and apparatus. Background Art

[0002] With the rapid development of cities, there are more and more lanes and the road types are becoming more and more complex. The interface of the full-scenario navigation software has not changed much. Especially in complex road scenarios, it is becoming increasingly difficult for users to use the map navigation data to compare with the real world, and it is very easy to miss the intersection or the opportunity to change lanes.

[0003] From the data side, high-definition (HD) maps are rich in information and highly similar to the real world. However, the construction cost is high and the freshness is difficult to guarantee. High-precision guidance data cannot be provided in urban areas, and the high-order navigation guidance needs of users cannot be met.

[0004] Therefore, how to provide higher-precision navigation guidance for users has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a navigation method and apparatus for providing lane-level navigation guidance, thereby providing more accurate navigation guidance and improving the accuracy of vehicle navigation.

[0006] In view of this, in a first aspect, this application provides a navigation method, including: First, obtaining navigation data and lane perception data. The navigation data includes the navigation path when the vehicle is driving on the map, which may include the path generated based on the driving starting point and destination of the vehicle. The lane perception data includes the information of at least one lane determined based on the information collected in the environment where the vehicle is located, specifically, it may be determined based on the environment perception data collected by the sensors in the vehicle; Subsequently, generating a navigation guidance according to the navigation data and the lane perception data, and displaying the navigation guidance. The navigation guidance is used to provide guidance for the vehicle to drive along the navigation path, and the navigation guidance can be used to identify the drivable lanes corresponding to the navigation path in the environment. The drivable lanes corresponding to the environment include the path of the vehicle driving obtained according to the lane perception data, and this path may be the lane line or the lane without lane lines that the vehicle can actually drive on.

[0007] In the embodiments of this application, lane-level navigation guidance can be generated through environmental perception, thereby providing more accurate navigation guidance. And in the method provided by this application, by perceiving the lanes in the environment and generating guidance, it is possible to achieve higher-precision lane-level guidance without relying on high-precision maps. For example, in some scenarios where the map lane lines are not clear or there are no lane lines, the drivable path of the vehicle can also be identified through environmental perception, providing more accurate navigation guidance and improving the accuracy and safety of vehicle driving.

[0008] In a possible implementation, the foregoing method may further include: lane perception data can be determined based on the data collected by the vehicle's sensors. For example, the information of the lane can be determined according to the lane lines included in the lane perception data, or the lane perception data can be the lane perception data obtained after the lane perception data is input into a pre-trained lane perception model, etc.; the lane perception data may specifically include information about at least one lane in the vehicle's environment, and may specifically include information such as the lane lines, lane positioning, or the number of lanes of at least one lane. The at least one lane may specifically include the lane in which the vehicle is currently traveling, the lanes adjacent to the vehicle, or the lanes in front of the vehicle, etc. Therefore, in the method provided by the embodiments of the present application, the information of the lanes in the environment where the vehicle is located can be perceived based on the perception of the vehicle's environment.

[0009] In a possible implementation, the lane perception data may specifically include the data collected by the sensors in the vehicle. For example, the sensors may include but are not limited to image sensors, radars, infrared sensors, or depth sensors, etc. That is, the lane perception data may also be but is not limited to the data of the environment sensed by sensors such as image sensors, radars, infrared sensors, or depth sensors. Therefore, in the embodiments of the present application, the data sensed by the sensors can be used to perceive the specific situation of the vehicle's environment, so as to determine the lane-level navigation guidance based on the perception result without relying on the HD map.

[0010] In a possible implementation, the foregoing navigation data may specifically include map data and a navigation path. The map data is the data of the map of the vehicle's driving area, and the navigation path is the path planned for the vehicle to drive in the corresponding area of the map. The navigation data may specifically be generated according to the user's trigger, or may be generated based on the vehicle's autonomous driving function, etc., and may be specifically determined according to the actual application scenario.

[0011] In a possible implementation, generating the navigation guidance according to the navigation data and the lane perception data may include: using the map data and the vehicle's historical motion information as the input of a pre-trained prediction model to output the predicted path of the vehicle. The prediction model is used to output the predicted path of the vehicle based on the input data. The vehicle's historical motion information may include the information generated by the vehicle's motion in the historical period, such as the vehicle's speed, acceleration, angle, angular velocity, etc.; then generating the navigation guidance according to the predicted path, the navigation path, and the lane perception data.

[0012] In the embodiments of the present application, a pre-trained prediction model can be used to predict the driving path of the vehicle, so that the navigation guidance more matching the vehicle's driving path can be generated based on the predicted path of the vehicle, improving the navigation accuracy of driving the vehicle.

[0013] In a possible implementation manner, generating navigation guidance based on the predicted path, the navigation path, and the lane perception data may include: If the navigation path in the navigation data includes a path where the vehicle changes lanes, such as turning at an intersection or changing lanes while going straight, obtain the first path and the second path of the navigation path. The first path is the path for the vehicle to enter the target lane from the current lane. For example, when the vehicle needs to turn, the first path may be the driving path of the vehicle when entering the bend in the navigation path. Or, when the vehicle needs to change lanes, the first path may be the driving path of the vehicle when transferring from the current lane to the target lane, and the target lane is one of the at least one lane mentioned above. The second path includes the driving path planned for the vehicle after it travels to the target lane. For example, when the vehicle needs to turn, the second path is the driving path of the vehicle when exiting the bend in the navigation path. Or, when the vehicle needs to change lanes, the second path may be the driving path of the vehicle after transferring from the current lane to the target lane, and the target lane is the lane where the vehicle travels after turning or changing lanes. Subsequently, determine a plurality of waypoints based on the predicted path, the first path, and the second path. The plurality of waypoints may include points that the vehicle may pass through during driving. In addition, the predicted path can be used to bind the first path and the second path in the navigation path. For example, fuse the predicted path into the first path and the second path, so that when determining the waypoints subsequently, more realistic navigation points can be calculated as waypoints. The plurality of waypoints may include an in-bend waypoint and an out-bend waypoint. The in-bend waypoint is a point determined based on the first path, and the out-bend waypoint includes a point determined based on the second path, so as to facilitate navigation guidance for the vehicle's in-bend and out-bend. Generate navigation guidance according to the plurality of waypoints.

[0014] In the implementation manner of this application, navigation guidance can be generated at the granularity of the waypoints that the vehicle may pass through, so as to generate more accurate navigation guidance based on the waypoints that the vehicle may pass through.

[0015] In a possible implementation manner, generating navigation guidance according to the plurality of waypoints may include: fitting the plurality of waypoints to generate a steering guidance curve. The steering guidance curve is used to indicate the lane in which the vehicle travels in the environment, and the curve model is a model for fitting a curve from a plurality of points. Subsequently, generate navigation guidance according to the steering guidance curve.

[0016] Therefore, in the implementation manner of this application, a smoother guiding line can be fitted and generated, thereby improving the user's visual experience of the navigation guidance and improving the accuracy of the user's recognition of the navigation guidance.

[0017] Optionally, when fitting the steering guidance curve, a curve model can be used to fit a smoother guiding line to improve the user's visual experience.

[0018] Optionally, when determining multiple line points, sampling can also be performed from the initially determined multiple line points to sample multiple line points that are more adaptable to the actual driving path of the vehicle, and then fitting these multiple sampled points to generate a steering guidance curve. Therefore, in the embodiments of the present application, the method of line point sampling can be used to avoid generating unreachable curves.

[0019] In a possible implementation manner, when the vehicle turns as described above and fits multiple line points to generate a steering guidance curve, it may include: obtaining a set of polyline points, which may include the current position of the vehicle, points on the unpaved path in the predicted path, and line points that the vehicle has not reached among the multiple line points, and forming a polyline among these multiple types of line points; subsequently, fitting the set of polyline points according to a curve model to generate a steering guidance curve, and different types of line points can be fitted to generate a more accurate curve.

[0020] In a possible implementation manner, when fitting multiple line points according to a curve model to generate a steering guidance curve as described above, it further includes: when the vehicle travels to the entry point of a curve, fitting the line points in the first path to generate a steering guidance curve pointing to the exit line point; when the vehicle travels to the exit point of a curve, fitting the line points in the second path to generate a steering guidance curve from the vehicle to the end of the predicted path. Therefore, the method provided by the present application can provide more fine-grained guidance based on line points for both entering and exiting curves, and can provide more accurate navigation guidance.

[0021] In a possible implementation manner, the prediction model described above may specifically include:

[0022] A map encoder for extracting features from map data to obtain map features;

[0023] A motion encoder for extracting features from the historical motion information of the vehicle to obtain motion features;

[0024] A fusion encoder for extracting features from the map features and motion features to obtain global features;

[0025] A reference path generator for generating a reference path of the vehicle according to the predicted path of the previous frame;

[0026] A path decoder for generating a predicted path according to the global features and the reference path.

[0027] Therefore, in the embodiments of the present application, the prediction model can be used to extract features from multiple dimensions to obtain an accurate predicted path.

[0028] In a possible implementation manner, the prediction model described above may further include:

[0029] A visual encoder is used to extract features from the input lane perception data (such as information perceived from the environment like images or point clouds) to obtain visual features.

[0030] A path decoder is further used to generate a predicted path based on the visual features, global features, and path features.

[0031] Therefore, in the embodiments of the present application, the path prediction accuracy is also improved by combining the visual encoder with the environmental perception information.

[0032] In a possible implementation, the aforementioned reference path generator is further used to obtain a reference path by combining the previous frame's predicted path and the motion constraint path, that is, the drivable path of the vehicle. The motion constraint path is a path generated based on the physical motion model of the vehicle's driving, and the motion constraint path is used as a constraint for outputting the reference path. Among them, the previous frame's predicted path is the path output by the prediction model when the navigation guidance was generated or updated last time, and the reference path is the path that the vehicle may drive during the current path prediction process. That is, under the constraint of the motion constraint path, a reference path that conforms to the vehicle's motion principle can be obtained, avoiding generating paths that the vehicle cannot reach.

[0033] In the embodiments of the present application, the input of the reference trajectory generator may further include a predicted path generated based on the motion model, which is equivalent to introducing the motion model as a constraint to make the predicted path an accessible path and improve the prediction accuracy.

[0034] In a possible implementation, the aforementioned generation of navigation guidance based on navigation data, map data, and lane perception data may further include: determining the lane change decision of the vehicle according to the navigation data, map data, and lane perception data, where the lane change decision is used to indicate whether the vehicle changes lanes; generating navigation guidance according to the lane change decision.

[0035] In the embodiments of the present application, it is possible to more accurately determine whether the vehicle changes lanes in the actual application scenario based on the navigation data, map data, and lane perception data, and thus generate accurate lane-level navigation guidance based on the decision of whether to change lanes.

[0036] In a possible implementation, the aforementioned determination of the vehicle's lane change decision according to the navigation data, map data, and lane perception data may include: determining the lane in which the vehicle is currently driving according to the map data and lane perception data; obtaining the probability of the vehicle transferring to each of the N lanes from the current lane according to the lane perception data and navigation data, where N is a positive integer, and the N lanes are determined from the lane perception data; determining the transfer path of the vehicle according to the probability of the vehicle transferring to each of the N lanes from the current lane; obtaining the lane change decision according to the transfer path.

[0037] Therefore, the embodiment of the present application is equivalent to introducing a hidden Markov chain to determine the possible variable road paths formed, and introducing a transition probability to determine the possible paths of the vehicle, thereby improving the accuracy of obtaining the transition paths. Moreover, even for complex road conditions, such as scenarios where the lane line perception results are inaccurate, such as multi-lane, worn lane lines, and occlusion by other vehicles, the lane in which the vehicle is traveling can be very accurately identified, and very accurate navigation guidance can be achieved.

[0038] In a possible implementation manner, the foregoing obtaining the probability of each lane when the vehicle transfers N lanes from the current lane according to the lane perception data and the navigation data may include: using the map data and the lane perception data as the input of a classification network, dividing the map into multiple regions through the classification network, where the multiple regions include high-confidence regions, and the high-confidence regions include regions with a confidence level higher than a preset value; expanding the map according to the high-confidence regions to obtain an expanded map, for example, expanding the low-confidence regions in the map based on the high-confidence regions, so as to improve the information richness of the low-confidence regions; obtaining the probability of each lane when the vehicle and the navigation path transfer N lanes from the current lane according to the expanded map. Therefore, in the embodiment of the present application, even for some maps with low information richness, the information richness of the map can be improved by expanding the map based on the high-confidence regions, the dependence on high-precision maps can be reduced, and accurate navigation guidance can still be achieved under a standard definition (SD) map.

[0039] In a possible implementation manner, the foregoing generating navigation guidance according to the lane change decision may include: if the lane change decision is to change lanes and drive, generating navigation guidance according to the transfer path; if the lane change decision is to keep driving in the lane, keeping the generated guiding line indicating lane keeping as the navigation guidance. For the decision of changing lanes, the navigation guidance can be generated based on the transfer path determined according to the transfer probability. For the decision of keeping driving in the lane, the generated guiding line indicating lane keeping can be kept as the navigation guidance.

[0040] In a possible implementation manner, a head-up display (HUD) or a display screen is further provided in the foregoing vehicle to display the navigation guidance, including: displaying the navigation guidance through the HUD, and the display position of the navigation guidance matching the environment; or, displaying the scene image in the lane perception data on the display screen, and superimposing and displaying the navigation guidance in the lane of the scene image. Therefore, in the method provided by the present application, the navigation guidance can be superimposed and displayed through the scene map displayed on the HUD or the display screen, so as to display the navigation guidance on the actual scene and improve the usability of the navigation guidance.

[0041] In a second aspect, the present application provides a navigation device, including:

[0042] An acquisition module that acquires navigation data and lane perception data. The map data includes data of the map of the vehicle driving area, the navigation data includes the navigation path when the vehicle drives in the map, and the lane perception data includes information of at least one lane determined based on the information collected in the environment where the vehicle is located.

[0043] A processing module for generating navigation guidance according to the navigation data and the lane perception data.

[0044] A display module for displaying the navigation guidance. The navigation guidance is used to provide guidance for the vehicle to drive along the navigation path, and is used to identify the drivable lanes corresponding to the navigation path in the environment. The drivable lanes corresponding to the environment include the driving path of the vehicle obtained according to the lane perception data.

[0045] In a possible implementation manner, the foregoing processing module is further configured to: determine the lane perception data based on the data collected by the vehicle's sensors. For example, the information of the lane can be determined according to the lane lines included in the lane perception data, or the lane perception data can be the lane perception data obtained after being input into a pre-trained lane perception model. The lane perception data specifically may include information of at least one lane in the environment where the vehicle is located, and specifically may include information such as lane lines, lane positioning, or the number of lanes of at least one lane. The at least one lane specifically may include the lane where the vehicle is currently driving, the lanes adjacent to the vehicle, or the lanes in front of the vehicle. Therefore, in the method provided by the embodiments of the present application, the information of the lanes in the environment where the vehicle is located can be perceived based on the perception of the environment where the vehicle is located.

[0046] In a possible implementation manner, the lane perception data may specifically include the data collected by the sensors in the vehicle. For example, the sensors may include but are not limited to image sensors, radars, infrared sensors, or depth sensors. That is, the lane perception data may also be but is not limited to the data of the environment sensed by sensors such as image sensors, radars, infrared sensors, or depth sensors. Therefore, in the implementation manner of the present application, the data sensed by the sensors can be used to perceive the specific situation of the environment where the vehicle is located, so as to determine the lane-level navigation guidance based on the perception result without relying on the HD map.

[0047] In a possible implementation manner, the foregoing navigation data may specifically include map data and a navigation path. The map data is the data of the map of the vehicle driving area, and the navigation path is the path planned for the vehicle to drive in the corresponding area of the map. The navigation data may specifically be generated according to the user's trigger, or may be generated based on the vehicle's autonomous driving function, etc., and may be specifically determined according to the actual application scenario.

[0048] In a possible implementation, the foregoing specifically is used for: taking map data and the historical movement information of the vehicle as the input of a prediction model, outputting the predicted path of the vehicle, where the prediction model is used to output the driving path of the vehicle based on the input data; generating navigation guidance according to the predicted path, navigation data, and lane perception data.

[0049] In a possible implementation, the foregoing processing module is specifically used for: if the navigation path in the navigation data includes a path where the vehicle changes lanes, obtaining a first path and a second path of the navigation path, where the first path is the path for the vehicle to enter the target lane from the current lane. For example, when the vehicle needs to turn, the first path is the driving path of the vehicle when entering the curve in the navigation path, or when the vehicle needs to change lanes, the first path can be the driving path of the vehicle when transferring from the current lane to the target lane, and the target lane is one of the foregoing at least one lane. The second path includes the driving path planned for the vehicle after driving into the target lane. For example, when the vehicle needs to turn, the second path is the driving path of the vehicle when exiting the curve in the navigation path, or when the vehicle needs to change lanes, the second path can be the driving path of the vehicle after transferring from the current lane to the target lane; determining multiple waypoints based on the predicted path, the first path, and the second path, where the multiple waypoints include an in-curve waypoint and an out-curve waypoint, the in-curve waypoint includes the point obtained based on the first path, and the out-curve waypoint includes the point obtained based on the second path; generating navigation guidance according to the multiple waypoints.

[0050] In a possible implementation, the foregoing processing module is specifically used for: fitting among the multiple waypoints to generate a steering guidance curve, where the steering guidance curve is used to indicate the lane in which the vehicle travels in the environment.

[0051] Optionally, when the processing module fits the steering guidance curve, a curve model can be used to fit a smoother guiding line to improve the user's visual experience.

[0052] In a possible implementation, when the vehicle turns, the foregoing processing module is specifically used for: obtaining a set of broken-line points, where the set of broken-line points includes the broken line formed between the current position of the vehicle, the points on the un-traveled path in the predicted path, and the un-reached waypoints among the multiple waypoints; fitting among the set of broken-line points to generate a steering guidance curve.

[0053] In a possible implementation, the foregoing processing module is specifically used for: when the vehicle travels to the in-curve point, fitting the waypoints in the first path according to the curve model to generate a steering guidance curve pointing to the out-curve waypoint; when the vehicle travels to the out-curve point, fitting from the waypoints in the second path according to the curve model to generate a steering guidance curve from the vehicle to the end of the predicted path.

[0054] In a possible implementation manner, the foregoing prediction model specifically includes:

[0055] A map encoder, configured to extract features from map data to obtain map features;

[0056] A motion encoder, configured to extract features from the historical motion information of the vehicle to obtain motion features;

[0057] A fusion encoder, configured to extract features from the map features and the motion features to obtain global features;

[0058] A reference path generator, configured to generate a reference path of the vehicle according to the predicted path of the previous frame;

[0059] A path decoder, configured to generate a predicted path according to the global features and the reference path.

[0060] In a possible implementation manner, the foregoing prediction model further includes:

[0061] A vision encoder, configured to extract features from the input image or point cloud data to obtain vision features. The image may be data collected by an image sensor, and the point cloud data may include data collected by a radar;

[0062] The path decoder is further configured to generate a predicted path according to the vision features, the global features, and the path features.

[0063] In a possible implementation manner, the foregoing reference path generator is further configured to combine the predicted path of the previous frame and the motion constraint path to obtain a reference path. The motion constraint path is a path generated based on the physical motion model of the vehicle driving, and the motion constraint path is used as a constraint for outputting the reference path. Among them, the predicted path of the previous frame is the path output by the prediction model when generating or updating the navigation guidance last time, and the reference path is the path that the vehicle may drive during the current path prediction process. That is, under the constraint of the motion constraint path, a reference path that conforms to the motion principle of the vehicle can be obtained, avoiding generating a path that the vehicle cannot reach.

[0064] In a possible implementation manner, the foregoing processing module is specifically configured to: determine a lane change decision of the vehicle according to navigation data, map data, and lane perception data, where the lane change decision is used to indicate whether the vehicle changes lanes; generate a navigation guidance according to the lane change decision.

[0065] In a possible implementation, the foregoing processing module is specifically configured to: determine the lane in which the vehicle is currently traveling according to the map data and the lane perception data; obtain the probability of the vehicle transferring to each of N lanes from the current lane according to the lane perception data and the navigation data, where N is a positive integer, and the N lanes are determined from the lane perception data; determine the transfer path of the vehicle according to the probability of the vehicle transferring to each of N lanes from the current lane; and obtain a lane change decision according to the transfer path.

[0066] In a possible implementation, the foregoing processing module is specifically configured to: use the map data and the lane perception data as the input of a classification network, divide the map into multiple regions through the classification network, where the multiple regions include high-confidence regions, and the high-confidence regions include regions with a confidence level higher than a preset value; expand the map according to the high-confidence regions to obtain an expanded map; and obtain the probability of the vehicle transferring to each of N lanes from the current lane according to the expanded map and the navigation path.

[0067] In a possible implementation, the foregoing processing module is specifically configured to: if the lane change decision is to change lanes and drive, generate a navigation guide according to the transfer path; if the lane change decision is to keep driving in the lane, keep the generated guiding line indicating lane keeping as the navigation guide.

[0068] In a possible implementation, a head-up display (HUD) or a display screen is further provided in the foregoing vehicle. The display module is specifically configured to: display the navigation guide through the HUD, and the display position of the navigation guide matches the environment; or, display a scene image of the environment where the vehicle is located on the display screen, and superimpose and display the navigation guide in the lane of the scene image.

[0069] In a third aspect, the present application provides an electronic device, which includes a display device, a memory, and one or more processors. The memory stores the code of the graphical user interface of the application program, and the one or more processors are configured to execute the code of the graphical user interface (GUI) stored in the memory to display the graphical user interface on the display device. The graphical user interface includes:

[0070] Display navigation guidance, where the navigation guidance is generated based on navigation data, map data, and scene data. The map data includes data of the map of the vehicle driving area, and the navigation data includes the navigation path when the vehicle drives in the map, which may include a path generated based on the driving starting point and destination of the vehicle. The scene data is information collected in the environment where the vehicle is located, and specifically may include environmental perception data collected by sensors in the vehicle. The navigation guidance can be used to identify the drivable lanes corresponding to the navigation path in the environment. The drivable lanes corresponding to the environment include the path of the vehicle's driving perceived according to the scene data, or the lane lines with or without lane divisions that the vehicle can actually drive on, and the navigation guidance is displayed superimposed on the perceived lanes.

[0071] Therefore, in the embodiments of the present application, the navigation guidance can be superimposed and displayed in the lanes in the GUI interface, so as to achieve more accurate lane-level guidance.

[0072] Optionally, the foregoing display device may specifically include an HUD or a display screen.

[0073] In a possible implementation manner, if the foregoing display device includes an HUD, the projection position of the vehicle's drivable lane on the windshield can be determined based on the user's line of sight point and scene data, and the navigation guidance is displayed at the projection position, thereby improving the usability of the navigation guidance.

[0074] In a possible implementation manner, if the foregoing display device includes a display screen, a scene image, that is, an image of the vehicle's drivable area, can be displayed on the display screen, and the navigation guidance is superimposed and displayed in the lanes in the scene image, thereby improving the usability of the navigation guidance.

[0075] In addition, the generation method of the navigation guidance can refer to the foregoing first aspect or any optional implementation manner of the first aspect, and will not be elaborated here.

[0076] In a fourth aspect, an embodiment of the present application provides a navigation device, including: a processor and a memory, where the processor and the memory are interconnected by a line, and the processor calls program code in the memory to execute functions related to processing in the method shown in any item of the foregoing first aspect. Optionally, the navigation device may be a chip.

[0077] In a fifth aspect, an embodiment of the present application provides a navigation device. The display device may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute functions related to processing in the foregoing first aspect or any optional implementation manner of the first aspect.

[0078] Sixth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which when running on a computer, cause the computer to execute the method in the first aspect or any optional implementation manner in the first aspect above.

[0079] Seventh aspect, an embodiment of the present application provides a computer program product containing instructions, which when running on a computer, cause the computer to execute the method in the first aspect or any optional implementation manner in the second aspect above.

[0080] Eighth aspect, an embodiment of the present application provides a vehicle, which includes the device described in the third aspect and at least one display device.

[0081] Optionally, the display device may specifically include a HUD or a display screen, etc. Description of the Drawings

[0082] Figure 1 It is a schematic structural diagram of a vehicle provided by the present application;

[0083] Figure 2 It is a schematic structural diagram of another vehicle provided by the present application;

[0084] Figure 3 It is a schematic structural diagram of a HUD provided by the present application;

[0085] Figure 4 It is a schematic structural diagram of another HUD provided by the present application;

[0086] Figure 5 It is a schematic flowchart of a navigation method provided by the present application;

[0087] Figure 6 It is a schematic structural diagram of a prediction model provided by the present application;

[0088] Figure 7 It is a schematic diagram of a navigation guidance generation method provided by the present application;

[0089] Figure 8 It is a schematic flowchart of another navigation method provided by the present application;

[0090] Figure 9 It is a schematic flowchart of another navigation method provided by the present application;

[0091] Figure 10 It is a schematic flowchart of another navigation method provided by the present application;

[0092] Figure 11 It is a schematic diagram of another navigation guidance generation method provided by the present application;

[0093] Figure 12Schematic diagram of another prediction model provided by this application;

[0094] Figure 13 Schematic diagram of another navigation guidance generation method provided by this application;

[0095] Figure 14 Schematic diagram of another navigation guidance generation method provided by this application;

[0096] Figure 15 Schematic diagram of another navigation guidance generation method provided by this application;

[0097] Figure 16 Schematic diagram of a GUI of a navigation guidance provided by this application;

[0098] Figure 17 Schematic diagram of a GUI of a navigation guidance provided by this application;

[0099] Figure 18 Schematic diagram of a GUI of a navigation guidance provided by this application;

[0100] Figure 19 Schematic diagram of a GUI of a navigation guidance provided by this application;

[0101] Figure 20 Schematic diagram of the structure of a navigation device provided by this application;

[0102] Figure 21 Schematic diagram of the structure of another navigation device provided by this application. Detailed implementation manners

[0103] Next, the technical solutions in the embodiments of this application will be described with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0104] The method provided by this application can be applied to a vehicle driving scenario to provide lane-level navigation guidance, providing navigation guidance that is more in line with the actual application scenario.

[0105] First, the method provided by this application can be applied to a device with a display device, such as a vehicle or an electronic device, etc. The display device can specifically include an electronic device display screen, a head-up display (HUD), a central control screen, an instrument screen, etc.

[0106] The method provided by this application can be applied to the driving scenario of a vehicle, which can specifically include a user driving a vehicle, an autonomous driving vehicle, or an assisted driving vehicle, etc.

[0107] The structure of the vehicle provided by this application or the vehicle to which it is applied will be introduced below. The vehicle has one or more display devices. During the driving process of the vehicle, lane-level navigation guidance can be displayed on the one or more display devices, so as to provide more appropriate navigation guidance for vehicle driving compared with the real environment.

[0108] Exemplarily, Figure 1 is a schematic structural diagram of a vehicle provided by an embodiment of this application. Figure 1 is a schematic functional block diagram of vehicle 100 provided by an embodiment of this application. Vehicle 100 can be configured in a fully or partially autonomous driving mode. For example: Vehicle 100 can obtain the environmental information around it through the perception system 120, and obtain an autonomous driving strategy based on the analysis of the surrounding environmental information to achieve full autonomous driving, or present the analysis result to the user to achieve partial autonomous driving.

[0109] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 can be interconnected by wired or wireless means. The models that need to be verified mentioned below in this application can include the models for implementing various systems or subsystems in the vehicle.

[0110] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0111] The data that needs to be displayed on the display screen mentioned in the following embodiments of this application can include the data generated by various systems in the vehicle during the operation of the vehicle, such as the operating status of each system or the collected data, etc.

[0112] The communication system 111 may include a wireless communication system 111, which can wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 111 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 111 can communicate with a wireless local area network (WLAN) using WiFi. In some embodiments, the wireless communication system can directly communicate with devices using an infrared link, Bluetooth, or ZigBee. The wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which can include public and / or private data communications between vehicles and / or roadside stations.

[0113] The entertainment system 112 may include a center console screen, a microphone, and speakers. Users can listen to the radio and play music in the vehicle based on the entertainment system 112; or connect the mobile phone to the vehicle to achieve screen mirroring of the mobile phone on the center console screen. The center console screen can be touch-sensitive, and users can operate it by touching the screen. In some cases, the user's voice signal can be obtained through the microphone, and certain controls of the vehicle 100 by the user can be achieved based on the analysis of the user's voice signal, such as adjusting the temperature inside the vehicle. In other cases, music can be played for the user through the speakers.

[0114] The navigation system 113 may include map services provided by a map provider to provide navigation for the driving route of the vehicle 100. The navigation system 113 can be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map services provided by the map provider can be standard definition (SD) maps, or high definition (HD) maps, and can also include SD+ maps, SD Pro maps, or ADAS maps, etc. This application does not make any limitations in this regard.

[0115] In addition, in some driving scenarios, the environmental information or navigation information of the vehicle can also be displayed on the center console screen. This center console screen can be one of the display screens mentioned in the following embodiments of this application.

[0116] The perception system 120 may include several types of sensors that sense information about the environment around the vehicle 100. For example, the perception system 120 may include a Global Positioning System 121 (the Global Positioning System may be a GPS system, or a Beidou system, or other positioning systems), an Inertial Measurement Unit (IMU) 122, a Light Detection and Ranging (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors for monitoring the internal systems of the vehicle 100 (such as an in-vehicle air quality monitor, a fuel gauge, an engine oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). Such detection and identification are key functions for the safe operation of the vehicle 100. The lane perception data mentioned in the embodiments of this application may include, but is not limited to, the data collected by the sensors provided in the perception system.

[0117] The Global Positioning System 121 can be used to determine the geographical location of the vehicle 100.

[0118] The Inertial Measurement Unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the Inertial Measurement Unit 122 may be a combination of an accelerometer and a gyroscope.

[0119] The Light Detection and Ranging (LiDAR) 123 can use lasers to sense objects in the environment where the vehicle 100 is located. In some embodiments, the Light Detection and Ranging (LiDAR) 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0120] The millimeter-wave radar 124 can use radio signals to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or forward direction of the objects.

[0121] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0122] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information obtained by the camera device 126 may include static image information or video stream information.

[0123] The decision-making control system 130 includes a computing system 131 that analyzes and makes decisions based on the information obtained by the perception system 120. The decision-making control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, as well as a steering system 133, an accelerator 134, and a braking system 135 for controlling the vehicle 100.

[0124] The computing system 131 can process and analyze various information obtained by the perception system 120 to identify targets, objects, and / or features in the surrounding environment of the vehicle 100.

[0125] The vehicle controller 132 can be used to coordinately control the power battery and the engine 141 of the vehicle to improve the power performance of the vehicle 100.

[0126] The steering system 133 can be used to adjust the forward direction of the vehicle 100.

[0127] The accelerator 134 is used to control the operating speed of the engine 141 and thus control the speed of the vehicle 100.

[0128] The braking system 135 is used to control the deceleration of the vehicle 100. The braking system 135 can use friction to slow down the rotation speed of the wheels 144.

[0129] In addition, the decision-making control system 130 may further include an instrument panel ( Figure 1 not shown in the figure), which is usually set near the steering wheel of the vehicle or at a position in the vehicle convenient for the user to observe. The instrument panel can also be a type of display screen mentioned in this application.

[0130] In some possible scenarios, the data displayed on the center control screen and the instrument panel can be combined and displayed on one display screen, or in other words, the center control screen and the instrument panel can be the same display screen or areas divided in the same display screen.

[0131] The drive system 140 includes components that provide power for the movement of the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144.

[0132] Examples of the energy source 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other power sources. The energy source 142 can also provide energy for other systems of the vehicle 100.

[0133] The transmission system 143 can transmit the mechanical power from the engine 141 to the wheels 144. The transmission system 143 may include a gearbox, a differential, and a drive shaft.

[0134] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151 that can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0135] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include, for example, a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), a microcontroller unit (MCU), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 may be located on a device remote from the vehicle and communicate wirelessly with the vehicle.

[0136] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100.

[0137] In addition to instructions 153, memory 152 may also store data, such as road maps, route information, the location, direction, speed of the vehicle, and other similar vehicle data, as well as other information. Such information may be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0138] Computing platform 150 can control the functions of vehicle 100 based on inputs received from various subsystems (e.g., drive system 140, perception system 120, and decision control system 130).

[0139] Optionally, one or more of the above components may be installed or associated separately from vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0140] Optionally, the above components are only an example. In actual applications, the components in each of the above modules may be added or deleted according to actual needs. Figure 1 It should not be construed as a limitation to the embodiments of the present application.

[0141] An autonomous vehicle moving on a road, such as vehicle 100 above, can identify objects within its surrounding environment to determine adjustments to its current speed. The objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine the speed adjustment that the autonomous vehicle is to make.

[0142] Optionally, vehicle 100 or a sensing and computing device associated with vehicle 100 (e.g., computing system 131, computing platform 150) can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each of the identified objects depends on the behavior of the others, so all of the identified objects can also be considered together to predict the behavior of a single identified object. Vehicle 100 is capable of adjusting its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered to determine the speed of vehicle 100, such as the lateral position of vehicle 100 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, etc.

[0143] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., a sedan in an adjacent lane on the road).

[0144] The above vehicle 100 can be a sedan, truck, motorcycle, bus, boat, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, etc., and the embodiments of the present application do not make special limitations.

[0145] Figure 2FIG. 0 shows a schematic diagram of the system architecture of vehicle 100 based on some embodiments. Vehicle 100 includes multiple vehicle integration units (VIUs) 11, a telematic box (T-BOX) 12, a cockpit domain controller (CDC) 13, a mobile data center (MDC) 14, a vehicle domain controller (VDC) 15, etc. These units can be used to support the vehicle in implementing various control or information interaction functions, etc.

[0146] Vehicle 100 further includes various types of sensors disposed on the vehicle body, including: lidar 21, millimeter-wave radar 22, ultrasonic radar 23, and imaging device 24. The sensors mentioned in the embodiments of the present application can include these various types of sensors. Each type of sensor can include multiple sensors. It should be understood that although Figure 2 shows the position layout of different sensors on vehicle 100, Figure 2 the number and position layout of the sensors in

[0147] are only for illustration. Those skilled in the art can reasonably select the types, numbers, and position layouts of the sensors according to needs. Figure 1 and Figure 2 the vehicle structure shown, one or more display screens can be provided in the vehicle, such as a center console screen (i.e., a display screen usually set at the center console position), an instrument screen (i.e., a display screen usually set on the instrument panel or a display screen for displaying the instrument panel), a head-up display (HUD), a rearview mirror display screen, or a rear row display screen, etc. Optionally, the HUD provided in the vehicle of the present application can specifically be an Augmented Reality Head Up Display (AR-HUD) to more vividly display navigation guidance.

[0148] Taking the display of navigation guidance in the following HUD as an example, when displaying navigation guidance in the HUD, the content that the driver needs to pay attention to when driving the vehicle can be projected onto the windshield centered on the driver's line of sight, enabling the driver to know this information without having to lower their head or turn their head as much as possible, reducing the occurrence of accidents and improving driving safety. Through the method provided in the present application, a navigation guidance more fitting to the actual road can be displayed in the windshield through the HUD, and a navigation guidance more adapted to the actual road can be generated without relying on a high-precision map, which can adapt to more scenarios, such as scenarios with complex road conditions. To further improve the driving experience and display more comprehensive navigation guidance, an AR-HUD can specifically be used to display the navigation guidance.

[0149] Refer to Figure 3 , for ease of understanding, the imaging structure of the HUD will be introduced below. Among them, an HUD 30 can be set in a vehicle, which includes multiple parts. Exemplarily, it can include a correction mirror 301, a concave mirror 302, an image generation unit 303, etc. as shown in Figure 3 .

[0150] Among them, the image generation unit 303 can be used to generate an optical signal of an image to be displayed;

[0151] The concave mirror 302 reflects the optical signal to the correction mirror 301;

[0152] The correction mirror 301 reflects the incident optical signal to the windshield of the vehicle.

[0153] The concave mirror 302 and the correction mirror 301 form an optical amplification circuit, so as to amplify and display the optical signal generated by the image generation unit 303 on the windshield.

[0154] For an AR-HUD, compared with a traditional HUD, its virtual image distance (VID) is usually farther, such as the distance d shown in Figure 3 , and the field of view (FOV) is wider. The FOV can be as shown in Figure 4 , that is, the included angle formed by the center of the driver's eyes (i.e., the sight point 40 shown in Figure 4 ) and the virtual image range. Specifically, it can be divided into a horizontal field of view angle θ and a vertical field of view angle α. From the user's perspective, the AR-HUD display range seen by the user is larger, and the range indicated by the actual scene is also larger. That is, in the method provided in this application, an AR-HUD can be used to display navigation guidance, so as to display lane-level navigation guidance with a larger field of view and more in line with the actual scene for the user.

[0155] Next, the method flow provided in this application will be introduced first.

[0156] Refer to Figure 5 , a schematic flow chart of a navigation method provided in this application.

[0157] It should be noted that the method provided in this application can be applied to devices such as vehicles or in-vehicle terminals. For ease of understanding, the devices to which the method provided in this application is applied are collectively referred to as electronic devices below. It should be understood that the electronic devices mentioned below can specifically include vehicles or other devices with a display device or devices connected to a display device, etc., which can be adjusted according to the actual application scenario, and this application does not limit this.

[0158] In addition, it should be noted that during the vehicle's travel, the navigation guidance can be updated in real time. In the implementation manner of this application, by way of example, the generation method of one frame of navigation guidance is introduced by way of example. The generation frequency or update frequency of the navigation guidance can be specifically determined according to the actual application scenario, and this application does not limit this.

[0159] 501. Obtain navigation data and lane perception data.

[0160] Among them, the navigation data is specifically divided into map data and a navigation path. The map data can include data of the map of the vehicle's driving area. The map data can be pre-stored in the electronic device or loaded from the server when the navigation is turned on, etc.; the navigation path can include the path when the vehicle travels in the map. The path can be a path generated based on the user's trigger operation or a path recommended based on the user's historical data, etc., and can be used to guide the path that the vehicle needs to travel.

[0161] The lane perception data can include information on at least one lane obtained by lane perception based on data collected by sensors provided in the vehicle. The lane perception data can also be referred to as perception data or other names, etc.

[0162] In a possible implementation manner, lane perception data can be determined by perception based on data collected by the vehicle's sensors. For example, the information of the lane can be determined according to the lane lines included in the lane perception data, or the lane perception data can be obtained by taking the lane perception data as the input of a pre-trained lane perception model and then outputting. The lane perception data can specifically include information on at least one lane in the environment where the vehicle is located, and can specifically include information such as lane lines, lane positioning, or the number of lanes of at least one lane. The at least one lane can specifically include the lane where the vehicle is currently traveling, the lane adjacent to the vehicle, or the lane in front of the vehicle, etc. Therefore, in the method provided by the embodiments of this application, the information of the lanes in the environment where the vehicle is located can be perceived based on the perception of the environment where the vehicle is located.

[0163] In a possible implementation, the lane perception data may specifically include data collected by sensors in the vehicle. For example, the sensors may include, but are not limited to, image sensors, radars, infrared sensors, or depth sensors, etc. That is, the lane perception data may also include, but is not limited to, data of the environment sensed by sensors such as image sensors, radars, infrared sensors, or depth sensors. Therefore, in the embodiments of the present application, the data sensed by the sensors can be used to perceive the specific situation of the environment where the vehicle is located, so as to determine the lane-level navigation guidance based on the perception result without relying on the HD map. In addition, optionally, the historical movement information of the vehicle can also be collected so that when predicting the vehicle path, lane-level positioning, etc. later, more accurate calculation results can be obtained by combining the movement data of the vehicle, thereby improving the accuracy of the navigation guidance. The historical movement information of the vehicle may specifically include information such as the vehicle speed, steering angle, and driving direction.

[0164] 502. Generate lane-level navigation guidance based on the navigation data and the lane perception data.

[0165] Subsequently, the map data and the lane perception data can be used to perceive the environment where the vehicle is located, and the lane-level guidance can be generated based on the perceived scene, navigation data, and map data, so as to provide a more accurate and more appropriate navigation guidance for the user in the actual scene.

[0166] Among them, the navigation guidance can be used to instruct the vehicle to drive along the navigation path. The road in the environment where the vehicle is located can be perceived based on the lane perception data and the map data. When guiding the navigation path, the guidance can be carried out according to the perceived lane; that is, the navigation guidance can be used to identify the drivable lanes corresponding to the navigation path in the environment, realizing lane-level navigation guidance.

[0167] That is, in the method provided in the present application, the road that the vehicle can drive on in the real scene can be identified based on the lane perception data and the map data. Specifically, it can be a lane with lane lines marked, or a lane without lane lines that the vehicle can actually drive on; the lane-level navigation guidance is generated based on the perceived lane, so as to provide a navigation guidance more adapted to the real scene without relying on the high-precision map.

[0168] Exemplarily, some possible generation methods of the navigation guidance are introduced in the present application. There can be multiple ways to generate the navigation guidance. For example, for different scenarios, an appropriate method can be adaptively selected to generate the navigation guidance. Some possible generation methods are introduced below.

[0169] It should be noted that there are various ways to generate navigation guidance in this application. For the convenience of understanding, the following will introduce the generation methods of navigation guidance from the aspects of path prediction and lane change decision respectively. It should be understood that the methods based on the prediction model and lane change decision mentioned below can be implemented in combination, or different generation methods can be selected for different scenarios to generate navigation guidance, etc. Specifically, the appropriate generation method can be selected according to the actual application scenario. That is to say, this application provides multiple different generation methods to generate navigation guidance, and the combination methods between different generation methods are not limited.

[0170] Method 1: Based on path prediction

[0171] This application provides a possible implementation manner. A pre-trained prediction model can be used to predict the trajectory of the vehicle. For example, map data and the historical movement information of the vehicle (i.e., vehicle movement data within a historical time period) are used as the input of the prediction model, and the predicted path of the vehicle is output. Then, navigation guidance is generated by combining the predicted path, map data, and lane perception data. In the implementation manner of this application, a neural network can be combined to predict the predicted path of the vehicle, that is, the path that the vehicle can travel within a future time period, so as to generate more accurate navigation guidance by combining the perception data and map data of the actual scenario based on the predicted path.

[0172] First, the structure of the possible prediction model will be introduced.

[0173] Refer to Figure 6 , this prediction model may include, but is not limited to, one or more of the following modules:

[0174] Map encoder 601, which is used to extract features from the input map data to obtain map features;

[0175] Motion encoder 602, which is used to extract features from the historical movement information of the vehicle to obtain motion features. The historical movement information may include information generated by the vehicle movement within the previous time period, such as the vehicle speed, steering angle, historical path, and other information;

[0176] Fusion encoder 603, or it can also be called a map - motion encoder, which is used to extract features from the map features and motion features to obtain global features;

[0177] Reference path generator 604, which is used to generate the reference path of the vehicle according to the predicted path of the previous frame. Among them, the navigation guidance during the vehicle driving process usually changes in real time, and the navigation guidance can be updated in units of frames. The predicted path of the previous frame is the predicted path output by the prediction model when generating or updating the navigation guidance of the previous frame. The reference path is the possible driving trajectory of the vehicle within a future time period based on the historical movement information of the vehicle;

[0178] A path decoder 605 for generating a predicted path based on the global feature and the reference path.

[0179] In the embodiments of the present application, a pre-trained prediction model can be used to take the information of each dimension as the input of the prediction model and output the current predicted path that the vehicle can travel, so that when generating navigation guidance subsequently, more accurate navigation guidance can be generated based on the predicted path.

[0180] To improve the accuracy of the predicted path, in the method provided by the present application, lane perception data can be further combined to enhance the perception of the environment, so as to improve the accuracy of the predicted path by combining the enhanced environmental perception. That is, the present application also provides an optional embodiment, and the foregoing prediction model may further include:

[0181] A visual encoder 606 for extracting features from the input lane perception data (such as data of an image or point cloud of the collected environment) to obtain visual features;

[0182] The path decoder 605 is further configured to generate a predicted path according to the visual features, the global features, and the path features.

[0183] Therefore, in the embodiments of the present application, the environment can be perceived through the lane perception data of the environment where the vehicle is located, so as to generate a more accurate predicted path by combining the environmental perception result.

[0184] In addition, to further improve the reachability of the predicted path, the present application can also combine the motion model of the vehicle to obtain a predicted path that more conforms to the physical motion model of the vehicle. That is, the present application also provides a possible embodiment, and a motion constraint path can also be generated according to the physical motion model of the vehicle. The reference path generator is further configured to combine the predicted path of the previous frame and the motion constraint path to obtain a reference path. The motion constraint path is a path generated based on the physical motion model of the vehicle driving, and the motion constraint path is used as a constraint for outputting the reference path, so as to avoid outputting a path that the vehicle cannot reach.

[0185] Further optionally, the method for generating navigation guidance may specifically include: for a lane-changing path, that is, if the predicted path is a scenario where the vehicle needs to change lanes, such as when the vehicle changes lanes while going straight or when the vehicle changes lanes when passing through an intersection, etc., the first path and the second path in the navigation path can be obtained. The first path is the entry path when the vehicle enters the target lane (i.e., the lane the vehicle needs to change to) in the navigation path, and the second path is the exit path when the vehicle travels towards the target lane in the navigation path. Subsequently, multiple waypoints are determined based on the predicted path, the first path, and the second path. The multiple waypoints may at least include an entry waypoint and an exit waypoint. The entry waypoint is the point in the entry path (determined based on the first path) when the vehicle enters the bend, and the exit waypoint is the point in the exit path (determined based on the second path) when the vehicle exits the bend. When the vehicle is traveling, the navigation guidance can be generated in real time based on the multiple waypoints. Usually, the navigation path only plans the vehicle's driving path according to the map, so there may be a deviation between the navigation path and the vehicle's actual driving path. Therefore, in the solution provided in this application, the path of the vehicle traveling in the actual application scenario is predicted. When determining the waypoints, the first path can be adsorbed into the predicted path, or the first path can be fused with the predicted path, or the path parallel to the first path in the predicted path can be used as the first path, etc., so that the navigation path is adsorbed into a more accurate predicted path, and thus the first path that can represent the entry path that the vehicle may travel in the actual application scenario is obtained. When generating the navigation guidance, the navigation guidance can be generated by combining the first path that fits the actual scenario, so as to achieve a more accurate guidance.

[0186] For example, as Figure 7 shown in the figure, the entry path and the exit path are shown. When the vehicle needs to turn, the current position of the vehicle is taken as point 1, point 2 is the predicted point when the vehicle turns, and point 3 is the point after the vehicle exits the bend. The path formed by point 1 and point 2 is the entry path, and the path formed by point 2 and point 3 is the exit path. For ease of understanding, the entry path can be understood as the path formed by pointing from the current position of the vehicle to the midpoint of the bend when entering the bend. The bend mentioned in this application refers to the vehicle driving road that requires the vehicle to have a certain steering angle when the vehicle turns, steers, or changes lanes. The exit path is the path pointing from the end point of the bend to the straight lane after exiting the bend.

[0187] Therefore, in the embodiments of this application, the navigation guidance can be generated in units of points, and the lane-level guidance can be updated in real time according to the vehicle waypoints when the vehicle is traveling, so as to provide clearer navigation guidance for the vehicle when it changes lanes.

[0188] Optionally, when determining multiple line points, sampling can also be performed from the initially determined multiple line points to sample multiple line points that are more suitable for the actual driving path of the vehicle, and then fitting the multiple sampled points to generate a steering guidance curve. Therefore, in the embodiments of the present application, the method of line point sampling can be used to avoid generating unreachable curves.

[0189] Optionally, in order to make the generated navigation guidance smoother and thus provide more accurate guidance, when generating navigation guidance, a curve model can be used to generate curve guidance, so that the navigation path fits better with the actual driving path of the vehicle. For example, when generating navigation guidance based on multiple line points as described above, curve fitting can be performed on the multiple line points according to the curve model to generate a steering guidance curve, and navigation guidance can be generated based on the steering guidance curve. For example, using the steering guidance curve as the navigation guidance, or using the steering guidance curve as a part of the navigation guidance, etc.

[0190] Specifically, when the vehicle changes lanes, in order to provide more accurate navigation guidance, a more clearly directed navigation guidance can be obtained by fitting a polyline. For example, first, a polyline point set (which can also be called a line point set, a line point polyline, etc.) can be obtained. The polyline point set can include the current position of the vehicle, points on the un-traveled path in the predicted path, and points among the multiple line points that the vehicle has not reached. A polyline is formed among the multiple line points. Among them, the points on the un-traveled path in the predicted path are the points on the path that the vehicle has not reached in the predicted path, and the points among the multiple line points that the vehicle has not reached are the points among the previously determined multiple line points that the vehicle has not reached; perform polyline fitting on the polyline point set and generate a steering guidance curve based on the collected polyline. Therefore, in the embodiments of the present application, a smooth curve can be generated by fitting a polyline, so that a reachable curve can be generated, and the accuracy of the generated navigation guidance can be improved.

[0191] Optionally, in order to make the in-curve navigation and out-curve navigation of the vehicle more accurate, when generating a steering guidance curve during the vehicle's driving process, a curve pointing to the in-curve point or out-curve point is generated based on the fitting of line points, so that the driving destination point of the vehicle is more accurate. For example, when the vehicle travels to the in-curve point, perform curve fitting on the line points in the first path according to the curve model to generate a steering guidance curve pointing to the out-curve point. When the vehicle travels to the out-curve point, perform curve fitting on the line points in the second path according to the curve model to generate a steering guidance curve pointing to the target lane. In the embodiments of the present application, when the vehicle changes lanes, curve fitting can be performed based on line points to generate a guidance curve for a curve, thereby improving the guidance accuracy for a curve. For complex road conditions, such as the situation where there are multiple parallel lanes or no lane lines after turning, the method provided by the present application can be used to achieve more accurate navigation guidance.

[0192] Method 2: Based on lane change decision

[0193] The present application also provides a possible implementation manner. According to navigation data, map data, and lane perception data, a lane-changing decision of the vehicle can be determined, and the lane-changing decision is used to indicate whether the vehicle changes lanes; a navigation guidance is generated according to the lane-changing decision. Therefore, in the implementation manner of the present application, it is possible to determine whether the vehicle changes lanes by combining the environment where the vehicle is located and the path indicated by the navigation path, and generate a navigation guidance based on the lane-changing decision, which can be applied to the scenario where the vehicle needs to change lanes and drive.

[0194] Optionally, when determining the lane-changing decision, the probability of the vehicle traveling from the current position to each lane can be calculated, and the probability corresponding to each transfer path that the vehicle may generate can be determined based on this probability, and it can be determined whether the vehicle changes lanes according to this probability. Specifically, the lane in which the vehicle is currently traveling can be determined according to the map data and the lane perception data; the probability of the vehicle transferring from the current lane to each of N lanes can be obtained according to the lane perception data and the navigation data, where N is a positive integer, and the N lanes are the lanes in the environment where the vehicle is located, such as the lanes determined from the lane perception data; the transfer path of the vehicle can be determined according to the probability of the vehicle transferring from the current lane to each of the N lanes; and the lane-changing decision can be obtained according to the transfer path. Equivalent to the method provided in the embodiment of the present application, the probability corresponding to each possible transfer path of the vehicle can be calculated based on a Markov chain, so as to select a suitable transfer path based on the probability of each transfer path, and thus more accurately determine the lane-changing decision.

[0195] In addition, the method provided by the present application may not rely on a high-precision map. Therefore, there may be some inaccurate data in the map area, such as the lane positioning in the map data does not match the actual lane position, or there are some drivable roads that do not exist in the map data. Therefore, in view of these situations, the present application divides the map into multiple regions, including high-confidence regions, that is, regions where the matching degree between the positioning in the map data and the positioning in the collected lane perception data is higher than a certain value. Specifically, the map data and the lane perception data can be used as the input of a classification network, and the classification result of each region, that is, the confidence level of each region, can be output; the regions can be divided into high-confidence regions and low-confidence regions, and the confidence level of the high-confidence regions is higher than that of the low-confidence regions. For example, regions with a confidence level higher than the confidence level threshold can be used as high-confidence regions, and regions with a confidence level not higher than the confidence level threshold can be used as low-confidence regions; subsequently, the map is expanded based on the high-confidence regions to obtain an expanded map. For example, high-confidence regions usually have higher accuracy, so an expanded and more comprehensive map can be obtained by combining the high-confidence regions with the lane perception data; and the probability of the vehicle transferring from the current lane to each of N lanes is calculated based on the expanded map.

[0196] That is, in the method provided in the embodiments of the present application, a classification network is used to match the lane positioning in the map with the lane positioning in the actual scenario, and high-confidence regions with relatively high matching degrees and low-confidence regions with relatively low matching degrees are output; in order to improve the accuracy of the overall map, the map is extended based on the high-confidence regions to obtain an extended map with a higher matching degree between the lane positioning and the lane positioning in the actual scenario. For example, for maps in some areas with low map accuracy or unclear lane lines, through the method provided in the present application, accurate lane positioning can be achieved without relying on high-precision maps, and thus more accurate navigation guidance can be generated based on the more accurate lane positioning.

[0197] Optionally, when generating navigation guidance, if the current lane change decision is to change lanes, the navigation guidance can be generated according to the transfer path, and the guidance displayed on the navigation guidance display interface is updated; if the current lane change decision is to keep driving in the current lane, the straight-line guiding line can be maintained as the navigation guiding guidance, and the straight-line guiding line is the navigation guidance that instructs the vehicle to keep going straight in the current lane.

[0198] 503. Display the navigation guidance.

[0199] After the navigation guidance is generated, the navigation guidance can be displayed on the display device. For example, the navigation guidance can be projected into the vehicle through the HUD, or the actual scenario or the rendered image of the actual scenario can be displayed on the display screen, and the navigation guidance is superimposed and displayed.

[0200] In the embodiments of the present application, the lane information of the vehicle driving in the actual scenario can be perceived based on the perception data collected by the vehicle sensors, and lane-level navigation guidance is generated to improve the navigation guidance that is more adapted to the real scenario. Moreover, the lanes in the real scenario are perceived through the data collected by the sensors. Without relying on high-precision maps, more accurate lane positioning can also be achieved, and more accurate navigation guidance can be realized.

[0201] The foregoing provides an overview of the method provided in the present application. Next, in combination with specific application scenarios, the method provided in the present application will be introduced in more detail.

[0202] The method provided in the present application can be applied to the driving scenarios of vehicles, such as autonomous driving, assisted driving, or user driving, etc. Assisted driving can include assisting the user in driving the vehicle, and can also include the auxiliary functions during autonomous driving, and autonomous driving means that the vehicle automatically drives.

[0203] Exemplarily, the architecture of the method provided in the present application can refer to Figure 8 . For example, the method provided in the present application can be deployed in a vehicle controller or an in-vehicle terminal connected to the vehicle. As shown in Figure 8 , the method provided in the present application is deployed in the AR engine.

[0204] First, the input data may specifically include SD basic navigation data, positioning data, lane perception data, etc. The input data may be data read from storage, may also include data from the vehicle's intelligent driving system or automated driving system (ADS), or may directly collect data through the interfaces provided by the vehicle. For example, as described above Figure 1 or Figure 2 As shown in the vehicle structure, when the vehicle is running, data can be transmitted between various VIUs in the vehicle, or transmitted to the vehicle's controller, etc., so that the data required for this application can be collected through the interfaces in the vehicle, such as data collected by vehicle sensors, such as information on the vehicle's surrounding environment collected by lidar, millimeter-wave radar, infrared sensors, depth sensors, or image sensors.

[0205] The AR engine is the module in the device used to drive AR display. As Figure 8 shown, it can be used to generate map guidance information, for navigation guidance modes, such as guiding straight, at intersections or lane changes, lane-level positioning, and corresponding guiding lines, etc.

[0206] After obtaining the navigation guidance information, the navigation guidance can be displayed on the AR device. Specifically, the navigation guidance can be displayed on the display screen, such as displaying the real-time environment image of the vehicle and superimposing the AR navigation guidance on the real-time environment image; or, the navigation guidance can be displayed on the AU-HUD.

[0207] In the following embodiments of this application, exemplarily, taking the display of navigation guidance on the AU-HUD as an example for exemplary introduction, which is not a limitation. The AR-HUD mentioned below can also be replaced by display screen display, and can be specifically adjusted according to the actual application scenario.

[0208] For example, the method flow provided by this application can be as Figure 9 shown.

[0209] First, the input data may include vehicle motion parameters (such as vehicle speed, steering angle, etc.), navigation information (including map data), and lane perception data (such as data collected by vehicle sensors), etc.

[0210] Using the navigation information and lane perception information, complex intersections can be modeled, and local maps can also be constructed. And based on the results of complex intersection modeling and local map construction, navigation guidance is generated. For the scenario without lane lines, lane estimation and prediction, guiding timing decision-making, and guiding line generation, etc. can be carried out; for the scenario with lane lines, lane-level positioning and lane change decision-making, etc. can be achieved. And the navigation guidance is displayed in real time on the AR-HUD.

[0211] The following will provide a more comprehensive process introduction to the method provided in this application in combination with specific application scenarios.

[0212] Refer to Figure 10 the flowchart of another navigation method provided in this application, which is described as follows.

[0213] It should be noted that for the sake of easy understanding, the foregoing multiple navigation guidance methods are respectively applied to different scenarios. For example, the foregoing method one can be applied to the intersection scenario, and the foregoing solution two can be applied to the scenario of straight driving and lane changing. Of course, in actual application scenarios, method one can also be applied to the straight driving and lane changing scenario, and method two can be applied to the intersection scenario, etc., or method one and method two can be applied in combination in different scenarios. Specifically, it can be adjusted according to the actual application scenario, and this application does not make any limitations in this regard.

[0214] 1001. Obtain vehicle motion data, navigation data, and lane perception data.

[0215] First, after triggering navigation for a vehicle trip, during the vehicle's driving, the data required for navigation can be collected in real time. The required data can be specifically divided into vehicle motion data, navigation data, lane perception data, etc.

[0216] The vehicle motion data can specifically collect elements such as the position and orientation of the vehicle in the world coordinate system, or data collected from sensors such as GNSS, IMU, and WSS wheel speedometers, such as the vehicle's speed, acceleration, angle, angular velocity, etc.

[0217] The navigation data can include the navigation path in the map generated based on the map data at the start of navigation. Specifically, it can include but is not limited to the planned path of the vehicle in the SD map, lane passing attributes (such as whether the lane allows turning or only allows straight driving, etc.).

[0218] The environmental perception data can specifically come from environmental perception data such as images or point clouds, and can specifically include lane lines, lane curbs, types of lane lines (such types include but are not limited to dotted lines, solid lines, semi-solid and semi-dotted lines, white lines, yellow lines, or double lines, etc.).

[0219] During the vehicle's driving, the state information of the vehicle's location and current status can be collected in real time through the sensors set in the vehicle, so as to update the navigation guidance in real time based on the navigation path.

[0220] 1002. Determine whether the vehicle has reached an intersection. If so, execute step 1003, otherwise execute step 1009.

[0221] During the vehicle driving process, it can be determined whether the vehicle has reached an intersection. If the vehicle has reached the intersection, the navigation guidance can be generated in the aforementioned first way, that is, step 1003 is executed. If the vehicle has not reached the intersection, the navigation guidance can be generated in the aforementioned second way, that is, step 1009 is executed.

[0222] Specifically, the method for determining whether the vehicle has reached the intersection can specifically determine whether the vehicle is at the intersection by the vehicle's positioning in the map, identifying whether it is an intersection based on the collected environmental perception data, or combining the positioning result of the vehicle in the map with the environmental perception result, etc., which can be specifically determined according to the actual application scenario. For example, it can be determined whether the vehicle's positioning in the map is at the intersection, or the environmental perception data can be used as the input of a pre-trained recognition network to identify whether the vehicle is at the intersection, or the recognition result of the vehicle positioning can be fused with the recognition result based on the environmental perception data to more accurately identify whether the vehicle is at the intersection.

[0223] 1003. Kinematic path prediction.

[0224] For the aforementioned collected vehicle motion data, based on the vehicle's kinematic model, such as the Constant Turn Rate and Velocity (CTRV) model, the Constant Turn Rate and Acceleration (CTRA) model, etc., the high-dimensional combined space of physical quantities such as the vehicle's short-term speed, acceleration, angular velocity, and angular acceleration can be predicted, and then the predicted path that the vehicle may generate can be calculated based on these parameters.

[0225] Taking the Constant Turn Rate and Velocity (CTRV) model as an example, assuming that the pose of the vehicle at time t is (x t , y t , θ t ), then the pose of the vehicle at time t + 1 (x t+1 , y t+1 , θ t+1 ) can be expressed as:

[0226]

[0227] Among them, v t represents the speed of the vehicle at time t, w t represents the angular velocity of the vehicle at time t. The pose change situation between time t and time t + 1 can be used to determine the path of the vehicle from time t to time t + 1.

[0228] For example, the effects achieved based on the motion model can be as follows Figure 11 As shown, the vehicle motion model constructs a mathematical equation based on the vehicle sensor inputs of the t-th frame and previous frames, and derives and calculates the possible future motion trajectories of the vehicle without training, realizing knowledge-driven path prediction and avoiding outputting unreachable paths.

[0229] 1004. Streaming path prediction.

[0230] Map data, environment perception data, and kinematic prediction paths can be used as inputs to the prediction model, and the predicted paths are output.

[0231] The inputs to the streaming trajectory prediction model include multiple parts: the vehicle sensor inputs of the t-th frame and previous frames (including but not limited to images or point clouds collected by on-vehicle cameras or LiDAR), multiple output trajectories of the streaming trajectory prediction model of the (t - 1)-th frame, the output trajectory of the vehicle motion model of the t-th frame, map information, and the motion information of the vehicle over a period of time in the past. The output of the streaming trajectory prediction model is multiple possible predicted paths of the ego vehicle in the future for a period of time. The streaming trajectory prediction model usually includes methods based on deep learning and requires prior model training.

[0232] For example, as Figure 12 shown, the predicted path of the vehicle motion model at the t-th frame and the predicted path of the (t - 1)-th frame can be used as inputs to the prediction model, and the predicted path of the t-th frame is output; when predicting the predicted path of the next frame (t + 1), the predicted path obtained based on the vehicle motion model can be used as an input to the prediction model, and the predicted path of the (t + 1)-th frame is output, and so on.

[0233] Specifically, the prediction model can refer to the foregoing Figures 6 to 7 and the corresponding descriptions, and the similarities will not be elaborated here. Below, exemplarily, the functions performed by each module of the prediction model will be introduced in detail.

[0234] Map encoder: Receives vectorized accurate map information, segments the road network information in the map by the minimum length (such as 5 meters), and saves it as a tensor of N map ×D map where N map is the number of map elements divided into segments, and D map is the number of attributes of each map element segment (including start point coordinates, lane category, etc.). A graph neural network is used as the map encoder to perform hierarchical encoding on the above tensor to obtain map features of N map ×D hidden .

[0235] Motion encoder: Receives a tensor of shape T×D carVehicle historical motion information tensor, where T is the historical trajectory time length and D car is the number of historical motion attributes (including coordinates, speed, orientation angle, etc.). Use LSTM as the motion encoder to encode the motion information and obtain the vehicle motion features of T×D hidden .

[0236] Map - Motion Encoder (i.e., Fusion Encoder 603): Concatenate the obtained map features and vehicle motion features to get an input of (N map +T)×D hidden , and then use the self - attention mechanism to process it to obtain global features of the form (N map +T)×D hidden .

[0237] Reference Trajectory Generator: Receive the previous - frame predicted path of K×T future ×2, where K is the number of predicted paths, T future is the prediction time length, 2 represents two - dimensional coordinate information, and the predicted path of the current - frame motion model of M×T future ×2. The reference trajectory generator concatenates the two, converts them to the current vehicle coordinate system, eliminates redundant predictions and complements missing predictions to obtain a reference trajectory of (K + M)×T future ×2

[0238] Image Encoder: Receive the RGB vehicle foreground image of H×W×3 and use a convolutional neural network to process the image to obtain image features of h×w×c

[0239] The path decoder uses the reference trajectory of (K + M)×T future ×2 to generate a query of (K + M)×D hidden through an MLP. First, use the query to perform cross - attention with the global features obtained by the map - motion encoder to obtain query feature 1 of (K + M)×D hidden ; then, according to the vehicle camera parameters and assuming the trajectory height coordinate, project the reference trajectory onto the vehicle foreground image. According to the position of the trajectory in the foreground image, use feature pooling and deformable convolution to extract the image features of each reference trajectory and perform a projection transformation to obtain trajectory features of (K + M)×D hidden . Add this feature to query feature 1 and perform a projection transformation to obtain query feature 2 of (K + M)×D hidden . The above feature extraction process is repeated R times, and finally the (K + M)×D hidden is output through an MLP as the predicted path of (K + M)×T future ×2 and their confidence levels of (K + M)×T future×1. During the training process, the predicted path and the ground truth trajectory are trained using the least mean square error and the winner-take-all strategy. During the inference process, the (K + M) × T future ×2 predicted paths are used to select the K trajectories with the highest confidence as the input for the next moment.

[0240] 1005. Navigation waypoint matching.

[0241] In the method provided by this application, by combining the predicted pose of vehicle movement, the navigation waypoints are adaptively reconstructed, and the error between the navigation waypoints and the real environment is reduced.

[0242] For example, as Figure 13 shown, according to the section labels of the SD map, the waypoints in the navigation path at the intersection can be split into the in-curve waypoints (such as Figure 13 midpoint 1-2) and the out-curve waypoints (such as Figure 13 midpoint 2-3). Then, the movement of the vehicle passing through the intersection can be divided into three stages, as Figure 13 shown in (a), (b), and (c) in it:

[0243] For (a): Before the vehicle enters the curve, the in-curve waypoint is attached to the vehicle. Before entering the curve, the vehicle can drive along the in-curve road according to the predicted path, that is, the in-curve vector direction has a high confidence. At this time, the in-curve waypoint is horizontally translated to the end of the predicted path, and the out-curve waypoint remains unchanged due to no additional reference information. The two are recombined to generate the recombined waypoint (such as Figure 13 point 4-5-6 in (a)).

[0244] For (b): During the vehicle turning, the waypoint is detached from the vehicle. During the turning process, the vehicle drives in the open area of the intersection and does not drive along the in-curve waypoint or the out-curve waypoint. At this time, the recombined waypoint remains unchanged (such as Figure 13 point 4-5-3 in (b)), and the vehicle is guided to the correct out-curve waypoint.

[0245] For (c): When the vehicle completes the turning, the out-curve waypoint is attached to the predicted path of the vehicle. After the turning is completed, the vehicle drives along the out-curve road, that is, the out-curve vector direction has a high confidence. At this time, the out-curve waypoint is horizontally translated to the end of the predicted path, and the in-curve recombined waypoint remains unchanged. The two are recombined to generate the recombined waypoint (such as Figure 13 point 4-6-7 in (c)).

[0246] Furthermore, for the navigation data, when the vehicle enters a new intersection, the initial recombined waypoint is the original navigation waypoint, and it is split into the in-curve waypoint polyline P e and the out-curve waypoint polyline P o .

[0247] For the predicted path output in step 1004, project its end to the recombined line points, and denote the projected point as p r , the vector direction to which this projected point p r belongs is θ r .

[0248] When the projected point belongs to the in-bend line point and the vector direction of the projected point is close to the end orientation of the predicted path, calculate the lateral error σ v between the vehicle's real-time position p r = p v - p r , and laterally translate the in-bend line point to the end of the predicted path. Then, the translated in-bend line point polyline P' e = P e + σ p .

[0249] Recombine with to generate the recombined line point polyline P. For P' e , denote as the ray with as the endpoint and the vector as the direction, denote as the line segment with the point and the point as the endpoints. For P o , denote as the ray with as the endpoint and the vector as the direction, denote as the line segment with the point and the point as the endpoints. Calculate the intersection point P between , then the recombined line point polyline t where nt and mt respectively represent the intersection indices of P in t .

[0250] When the vector direction of the projected point is not close to the vehicle's orientation, keep the recombined line points unchanged.

[0251] When the projected point belongs to the out-bend line point and the vector direction of the projected point is close to the end orientation of the predicted path, calculate the lateral error σ v between the vehicle's real-time position p r = p v - p r , and laterally translate the out-bend line point to the end of the predicted path. Then, the translated out-bend line point polyline P'​o = P o + σ p 。

[0252] Recombine with to generate the recombined line point polyline P. For P e , denote as the ray with as the endpoint and the vector as the direction, denote the line segment with the points and as the endpoints. For P o , denote as the ray with as the endpoint and the vector as the direction, denote the line segment with the points and as the endpoints. Calculate the intersection point P between , then the recombined line point polyline t , where nt and mt respectively represent the intersection indices of P in t in .

[0253] 1006. Guide line point search.

[0254] More specifically, for the recombined line points in step 1005 and the predicted path in step 1004, calculate the first projection point of the current vehicle pose on the recombined line points and the second projection point of the end of the predicted path on the recombined line points. Perform weighted fusion on the navigation line points between the predicted path, the first projection point, and the second projection point, and calculate to obtain the proximal guide line point P n . Calculate the intersection point of the end of the vehicle's proximal guide line point and the recombined line points. When the intersection point exists, use this intersection point as the starting point. When the intersection point does not exist, use the second projection point as the starting point, and take the line points after this starting point in the recombined line points as the distal guide line point P f . Combine the proximal guide line point and the distal guide line point to obtain the guide line point P of the vehicle at the current moment = {p n , P f}.

[0255] In addition, as Figure 14 shown, for the local guide line in the navigation guidance, the polyline l formed by the local guide line control points = {p t , p t ′ , p i , p i+1 , …, pn} consists of the current position p of the vehicle t , the predicted trajectory position p t ′, the recombined line points {p i , p i+1 , …, p n} (0 ≤ i ≤ n) that the vehicle has not reached.

[0256] Map p t ′ to the recombined line points to obtain the sequence of forward road line points {p c , p c+1 , …, p n} (as shown in Figure (a), points {p1, p2}). Calculate the intersection points of the search space (the possible future movement trajectories of the vehicle) and the forward road line points as new control points to avoid the reverse distortion of the guiding line. For example, a right-turn curve has a left-turn tendency. If there is a legal intersection point p k when i = k, combine p t , p′ t , p k with the forward road line points after this intersection point to form the guiding line control point polyline l = {p t , p′ t , p k , p k+1 , …, p n}; if there is no legal intersection point, i = c, combine p t , p′ t with the forward road line points to form the guiding line control point polyline l = {p t , p′ t , p c , p c+1 , …, p n}.

[0257] 1007. Generation of the guiding line.

[0258] After collecting the line points, the guiding line points can be sampled in ways such as equidistant sampling and curvature threshold sampling. Based on the sampled points, guiding line control points are formed to ensure the stability of the guiding line between frames. The specific curve models that can be used can include but are not limited to curve models such as B-spline curves and Bezier curves to generate a smooth steering guiding line. Taking the B-spline curve as an example, the guiding line curve equation C(t) can be expressed as:

[0259]

[0260] where B i,deg (t) is the basis function, deg is the curve order, knot is the knot vector, and P i (P i ∈ P) represents the control points.

[0261] After generating the guiding line of the navigation guidance, the navigation guidance can be displayed, that is, step 1015 is executed.

[0262] 1008. High-confidence area division.

[0263] In the case of non-intersection scenarios, lane-level matching and positioning can be performed, so as to accurately identify the next driving path of the vehicle.

[0264] First, the map represented by the map data can be divided into multiple regions, and regions with relatively high confidence can be screened out, such as regions with a confidence higher than the confidence threshold, or the matching degree between the vehicle's positioning in the map and the vehicle positioning collected by the vehicle sensor is higher than a certain value. For example, when usually on the left or right edge lanes of the road or the perception data with good visibility can accurately identify all lane lines, etc., the initialization of the lane-level matching and positioning model is completed, and at this time, the SD map route data and the positioning data are aligned.

[0265] Among them, the determination of the high-confidence area can be identified using a lightweight image classification network such as MobileNetV3, and at this time, relatively accurate lane-level initialization information can be obtained. The current positioning position is denoted as point A xy , and the position bound to the SD route is denoted as point B xy , from the perception information, it can be known that the current lane number is X, the left offset distance W within the current lane l , the current lane width W all , and from the positioning information, it can be known that the current road direction is h.

[0266] The left boundary point of the current road is denoted as C xy , point A xy and point C xy The distance between:

[0267] dist = W all *(X - 1) + W l

[0268] Then point C xy :

[0269] C x = A x + dist * (cos(h - 90) - sin*(h - 90))

[0270] C y = A y + dist * (cos(h - 90) + sin*(h - 90))

[0271] Furthermore, the lateral relative position of the SD route on the current road, that is, the offset α, is determined. xy。

[0272] α xy = C xy - B xy

[0273] Therefore, based on the method provided in this application, the lanes in the actual application scenario can be accurately identified, so as to compensate the lane positioning accuracy in the map and obtain accurately positioned lane information.

[0274] 1009. SD map expansion.

[0275] It can be inferred from the acquisition principle of map production that the Euclidean distance of the offset α xy will not change within a limited time and space ahead. Combining navigation information such as perceived lane lines, positioning, and the number of lanes, the construction of the current lane-level positioning map is completed based on the skeleton information of the SD map route. The current position is denoted as point A xy , and the position point bound to the SD route is denoted as B xy . From the offset α xy combined with the current road direction h, it can be inferred that the intersection point of the normal direction of the current vehicle driving direction and the left boundary of the road is point C.

[0276] C x = B x + ||α xy || * (cos(h - 90) - sin*(h - 90))

[0277] C y = B y + ||α xy || * (cos(h - 90) + sin*(h - 90))

[0278] The number of current navigation lanes is denoted as N, and the center points of each lane are P1, P2…P N , and the width of the current lane is W all

[0279]

[0280] P i y = C y + (i - 1) * W all * (cos(h + 90) + sin*(h + 90))

[0281] The center points of each lane obtained by the current method and the perceived lane lines are weighted and fused to determine the final center points of each lane.

[0282] 1010. Lane probability matching.

[0283] The perception, navigation, and positioning results all have a certain impact on the formation of the probability of the current lane position, and the probability distribution of the vehicle's lane position changes with the vehicle's movement and perception and navigation information. The probability is composed of an initial probability, an observation probability, and a transition probability. The initial probability is evenly distributed by the number of lanes:

[0284]

[0285] The observation probability is composed of parts such as the positioning observation probability P1, the lane number observation probability P2, and the lane line type observation probability P3:

[0286]

[0287] The lane number observation probability P2 and the lane line type observation probability P3 can be obtained by training a Bayesian network. The total observation probability:

[0288] P o = P1 * P2 * P3

[0289] The transition probability uses the cosine distance. Denote the angle between the line connecting the center points of each lane at the previous moment and the center points of each lane at the current moment and the positioning displacement line segment as θ:

[0290]

[0291] Therefore, in the embodiment of the present application, the transition probability between lanes of the vehicle can be used to facilitate subsequent prediction of the possible transfer position of the vehicle based on the probability.

[0292] 1011. Sequence estimation.

[0293] In the method provided by the present application, by combining vehicle movement data, lane perception data, and navigation data, the single-frame lane line perception and positioning problem is transformed into an optimal probability problem under spatio-temporal continuous observation. Regarding the N lanes of navigation as N hidden state variables, based on the hidden Markov process, the Viterbi algorithm is used to calculate the current optimal lane-level positioning of the vehicle and output the probabilistic result.

[0294] Generally, the output result of the lane-level positioning is the absolute lane number where it is located. In the embodiment of the present application, probabilistic output is adopted to finely represent the lane information where it is located, thereby achieving a high accuracy rate for lane change decisions.

[0295] For example, as Figure 15As shown in the figure, the single process of lane-level matching positioning can be divided into several steps: high-confidence area initialization, SD route expansion mapping, probabilistic matching positioning, and optimal estimation of the state sequence. When outside the credible interval, means such as perception can be used to obtain lane-level positioning information. The embodiment of this application is based on the hidden Markov process, which transforms the single-frame lane line perception and positioning problem into an optimal probability problem under spatio-temporal continuous observation. The perception, navigation, and positioning results jointly determine the state transition probability, reduce the possibility of false positioning caused by missed detection, occlusion, etc., and obtain the optimal estimation of the specific state sequence, that is, obtain a better estimation of the lane where the vehicle is located.

[0296] 1012. Determine whether to change lanes. If so, execute step 1013; if not, execute step 1014.

[0297] Combining the lane passing attributes (left turn, straight, right turn, etc.) provided by the navigation data, the solid and dashed line information provided by the lane perception data, and the lane-level positioning result obtained in the previous step 1011, conduct traffic rule verification to make a decision on whether the vehicle needs to change lanes currently.

[0298] 1013. Generate a lane change guiding line.

[0299] When the lane change decision is to change lanes, combining the lane perception data and navigation data in the environmental perception data, fusing the driving points of the vehicle along the current lane, the lane center points of the target lane, and the distal navigation points, generate guiding line control points, including but not limited to using curve models such as B-spline curves and Bezier curves, to obtain a smooth lane change guiding line.

[0300] 1014. Generate a straight driving guiding line.

[0301] When the lane change decision is to keep the current lane, project the current pose of the vehicle onto the navigation route, take out the navigation points in front of the vehicle, and weighted-fuse the lane perception data as the guiding line control points, including but not limited to using curve models such as B-spline curves and Bezier curves, to generate a smooth straight driving guiding line.

[0302] 1015. Display navigation guidance.

[0303] After generating the navigation guidance, the navigation guidance can be displayed. Specifically, it can be displayed in the AR-HUD or on the display screen.

[0304] For example, for intersection guidance, the AR-HUD display interface for a right turn guidance can be as Figure 16 shown, and the AR-HUD display interface for a left turn guidance can be as Figure 17 shown.

[0305] For another example, for lane change guidance, the AR-HUD display interface for a right lane change guidance can be asFigure 18 As shown, the AR-HUD display interface for the left lane change guidance can be as Figure 19 shown.

[0306] The method provided by this application solves the problem that the SD map accuracy is insufficient to support high-precision navigation through trajectory prediction and local guidance line generation. In scenarios such as no lane lines or blocked lane lines, it can still stably provide intersection guidance and lane change guidance. For example, for intersection guidance, it overcomes the problems of no lane line constraints at complex intersections, difficult placement of guidance lines, and easy off-road; for lane change guidance, it overcomes the problems of mismatch between lane-level perception and navigation information and incorrect lane changes; and when applied to AR-HUD, compared with the limited perspective of the center console, it is necessary to construct and generate guidance lines adapted to the vehicle position in real time, rather than directly performing a projection transformation on the SD map.

[0307] The above introduced the method flow provided by this application. Next, the device for executing the method flow provided by this application will be introduced.

[0308] This application also provides an electronic device, which includes a display device, a memory, and one or more processors. The memory stores the code of the graphical user interface of the application program. The one or more processors are used to execute the code of the graphical user interface (GUI) stored in the memory to display the graphical user interface on the display device. The graphical user interface includes:

[0309] Display navigation guidance, where the navigation guidance is generated according to navigation data, map data, and scene data. The map data includes the data of the map in the vehicle driving area, the navigation data includes the navigation path when the vehicle is driving on the map, which may include the path generated based on the driving starting point and destination of the vehicle, and the scene data is the information collected in the environment where the vehicle is located, specifically including the environmental perception data collected by the sensors in the vehicle; the navigation guidance can be used to identify the drivable lanes corresponding to the navigation path in the environment. The drivable lanes corresponding to the environment include the path of the vehicle driving perceived according to the scene data, or the divided or undivided lanes that the vehicle can actually drive on, and the navigation guidance is superimposed and displayed in the perceived lanes.

[0310] Therefore, in the embodiment of this application, the navigation guidance can be superimposed and displayed in the lanes in the GUI interface, so as to achieve more accurate lane-level guidance.

[0311] The aforementioned electronic device can specifically be a vehicle, an in-vehicle terminal, or other navigation devices, etc.

[0312] Optionally, the aforementioned display device can specifically include HUD or a display screen.

[0313] In a possible implementation, if the foregoing display device includes an HUD, such as a conventional HUD or an AR-HUD specifically, the projection position of the drivable lane of the vehicle on the windshield can be determined based on the user's line of sight point and scene data, and navigation guidance can be displayed at the projection position, thereby improving the usability of the navigation guidance. For example, as described above Figures 16 to 19 As shown, the navigation guidance can be projected on the windshield of the vehicle. Referring to the foregoing Figures 3 to 4 , when projecting, the position of the lane corresponding to the navigation guidance projected on the windshield can be determined in combination with the user's line of sight point, that is, the position where the user observes the lane through the windshield, and the navigation guidance is superimposed and displayed at this position. Equivalent to the method provided in this application, the more accurate navigation guidance obtained by the method corresponding to the foregoing in this application is projected onto the actual lane through the HUD, improving the user's viewing experience of the navigation guidance and improving the user experience. Figures 5 to 15

[0314] In a possible implementation, if the foregoing display device includes a display screen, a scene image, that is, an image of the drivable area of the vehicle, can be displayed on the display screen, and navigation guidance is superimposed and displayed in the lane in the scene image, thereby improving the usability of the navigation guidance.

[0315] In addition, the generation method of the navigation guidance displayed in the GUI can refer to the foregoing Figures 5 to 15 corresponding description, which will not be elaborated here.

[0316] Referring to Figure 20 , a schematic structural diagram of a navigation device provided in this application, the navigation device may include:

[0317] An acquisition module 2001, which has the function of acquiring navigation data, map data, and lane perception data. The map data includes data of the map of the vehicle driving area, the navigation data includes the navigation path when the vehicle drives in the map, and the lane perception data is the information collected in the environment where the vehicle is located;

[0318] A processing module 2002, which is used to generate navigation guidance according to the navigation data, map data, and lane perception data;

[0319] A display module 2003, which is used to display navigation guidance. The navigation guidance is used to instruct the vehicle to drive according to the navigation path, and the navigation guidance is used to identify the drivable lane corresponding to the navigation path in the environment. The drivable lane corresponding to the environment includes the path that the vehicle can drive through according to the lane perception data.

[0320] ​In a possible implementation, the aforementioned processing module 2002 is further configured to: sense and determine lane perception data based on the data collected by the vehicle's sensors. For example, the lane information can be determined according to the lane lines included in the lane perception data, or the lane perception data can be the output obtained after the lane perception data is input into a pre-trained lane perception model, etc.; the lane perception data can specifically include information about at least one lane in the vehicle's environment, which can specifically include information such as lane lines, lane positioning, or the number of lanes of at least one lane. The at least one lane can specifically include the lane where the vehicle is currently traveling, the lanes adjacent to the vehicle, or the lanes in front of the vehicle, etc. Therefore, in the method provided by the embodiments of the present application, the information about the lanes in the environment where the vehicle is located can be sensed based on the perception of the vehicle's environment.

[0321] In a possible implementation, the lane perception data can specifically include the data collected by the sensors in the vehicle. For example, the sensors can include, but are not limited to, image sensors, radars, infrared sensors, or depth sensors, etc. That is, the lane perception data can also be, but is not limited to, the environmental data sensed by sensors such as image sensors, radars, infrared sensors, or depth sensors. Therefore, in the implementation manner of the present application, the specific situation of the vehicle's environment can be sensed by using the data sensed by the sensors, and thus the lane-level navigation guidance can be determined based on the sensing result without relying on the HD map.

[0322] In a possible implementation, the aforementioned navigation data can specifically include map data and a navigation path. The map data is the data of the map of the vehicle's driving area, and the navigation path is the path planned for the vehicle to drive in the corresponding area of the map. The navigation data can specifically be generated according to the user's trigger, or can be generated based on the vehicle's autonomous driving function, etc., and can be specifically determined according to the actual application scenario.

[0323] In a possible implementation, the aforementioned processing module 2002 is specifically configured to: use the map data and the vehicle's historical motion information as the input of a prediction model, and output the vehicle's predicted path. The prediction model is used to output the vehicle's driving path based on the input data; generate a navigation guidance according to the predicted path, the navigation data, and the lane perception data.

[0324] In a possible implementation manner, the foregoing processing module 2002 is specifically configured to: if the navigation path in the navigation data includes a path where the vehicle changes lanes, obtain a first path and a second path of the navigation path. The first path is the path for the vehicle to enter the target lane from the current lane. For example, when the vehicle needs to turn, the first path is the driving path of the vehicle when entering the curve in the navigation path. Or, when the vehicle needs to change lanes, the first path can be the driving path of the vehicle when transferring from the current lane to the target lane. The target lane is one of the foregoing at least one lane. The second path includes the driving path planned for the vehicle after driving into the target lane. For example, when the vehicle needs to turn, the second path is the driving path of the vehicle when exiting the curve in the navigation path. Or, when the vehicle needs to change lanes, the second path can be the driving path of the vehicle after transferring from the current lane to the target lane; determine a plurality of waypoints based on the predicted path, the first path, and the second path. The plurality of waypoints include an in-curve waypoint and an out-curve waypoint. The in-curve waypoint includes a point determined based on the first path, and the out-curve waypoint includes a point determined based on the second path; generate navigation guidance according to the plurality of waypoints.

[0325] In a possible implementation manner, the foregoing processing module 2002 is specifically configured to: fit the plurality of waypoints to generate a steering guidance curve, and the steering guidance curve is used to indicate the lane in which the vehicle travels in the environment.

[0326] Optionally, when the processing module fits the steering guidance curve, a curve model can be used to fit a smoother guidance line to improve the user's visual experience.

[0327] In a possible implementation manner, when the vehicle turns, the foregoing processing module 2002 is specifically configured to: obtain a set of broken-line points, where the set of broken-line points includes a broken line formed between the current position of the vehicle, the points on the un-traveled path in the predicted path, and the un-reached waypoints among the plurality of waypoints; fit the waypoints in the set of broken-line points to generate a steering guidance curve.

[0328] In a possible implementation manner, the foregoing processing module 2002 is specifically configured to: when the vehicle travels to the in-curve point, fit the waypoints in the first path to generate a steering guidance curve pointing to the out-curve waypoint; when the vehicle travels to the out-curve point, fit the waypoints in the second path to generate a steering guidance curve pointing from the vehicle to the end of the predicted path.

[0329] In a possible implementation manner, the foregoing prediction model specifically includes:

[0330] A map encoder, configured to extract features from map data to obtain map features;

[0331] A motion encoder, which is used to extract features from the historical motion information of a vehicle to obtain motion features;

[0332] A fusion encoder, which is used to extract features from map features and motion features to obtain global features;

[0333] A reference path generator, which is used to generate a reference path of the vehicle according to the predicted path of the previous frame;

[0334] A path decoder, which is used to generate a predicted path according to the global features and the reference path.

[0335] In a possible implementation manner, the foregoing prediction model further includes:

[0336] A visual encoder, which is used to extract features from the input lane perception data to obtain visual features;

[0337] The path decoder is further used to generate a predicted path according to the visual features, the global features and the path features.

[0338] In a possible implementation manner, the foregoing reference path generator is further used to combine the predicted path of the previous frame and the motion constraint path to obtain a reference path, where the motion constraint path is a path generated based on the physical motion model of the vehicle driving, and the motion constraint path is used as a constraint for outputting the reference path.

[0339] In a possible implementation manner, the foregoing processing module 2002 is specifically used to: determine a lane change decision of the vehicle according to the navigation data and the lane perception data, where the lane change decision is used to indicate whether the vehicle changes lanes; generate a navigation guidance according to the lane change decision.

[0340] In a possible implementation manner, the foregoing processing module 2002 is specifically used to: determine the lane in which the vehicle is currently driving according to the map data and the lane perception data; obtain the probability of the vehicle transferring to each of N lanes from the current lane according to the lane perception data and the navigation data, where N is a positive integer, and the N lanes are determined from the lane perception data; determine the transfer path of the vehicle according to the probability of the vehicle transferring to each of N lanes from the current lane; obtain a lane change decision according to the transfer path.

[0341] In a possible implementation manner, the foregoing processing module 2002 is specifically used to: use the map data and the lane perception data as the input of a classification network, divide the map into multiple regions through the classification network, where the multiple regions include high-confidence regions, and the high-confidence regions include regions with a confidence level higher than a preset value; expand the map according to the high-confidence regions to obtain an expanded map; obtain the probability of the vehicle transferring to each of N lanes from the current lane according to the expanded map and the navigation path.

[0342] In a possible implementation manner, the foregoing processing module 2002 is specifically configured to: if the lane change decision is to change lanes and drive, generate navigation guidance according to the transfer path; if the lane change decision is to keep driving in the lane, keep the generated guiding line indicating lane keeping as the navigation guidance.

[0343] In a possible implementation manner, a head-up display (HUD) or a display screen is further provided in the foregoing vehicle. The display module is specifically configured to: display the navigation guidance through the HUD, and the display position of the navigation guidance matches the environment; or, display a scene image of the environment where the vehicle is located on the display screen, and superimpose and display the navigation guidance in the lane of the scene image.

[0344] Refer to Figure 21 , which is a schematic hardware structure diagram of a navigation device 210 provided by an embodiment of the present application. The navigation device can be deployed in a vehicle and can be used to execute the foregoing Figures 3 to 19 shown method steps. The navigation device can also be referred to as an electronic device.

[0345] Figure 21 The navigation device 210 shown can include: a processor 2101, a memory 2102, a communication interface 2103, and a bus 2104. The processor 2101, the memory 2102, and the communication interface 2103 can be connected through the bus 2104.

[0346] The processor 2101 is the control center of the navigation device 210. It can be a general-purpose central processing unit (CPU), or other general-purpose processors, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0347] As an example, the processor 2101 can include one or more CPUs, such as Figure 21 the CPUs 0 and 1 shown in

[0348] The memory 2102 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium, or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0349] In one possible implementation, the memory 2102 can exist independently of the processor 2101. The memory 2102 can be connected to the processor 2101 via the bus 2104 and is used to store data, instructions, or program codes. When the processor 2101 calls and executes the instructions or program codes stored in the memory 2102, it can implement the display method provided by the embodiments of the present application or extract the GUI provided by the embodiments of the present application.

[0350] In another possible implementation, the memory 2102 can also be integrated with the processor 2101.

[0351] The communication interface 2103 is used to connect the navigation device 210 to other devices via a communication network. The communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 2103 can include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0352] The bus 2104 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 21 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0353] It should be noted that Figure 21 the structure shown in the figure does not constitute a limitation on the navigation device 210. Except Figure 21 for the components shown, the navigation device 210 can include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0354] Optionally, the aforementioned Figure 21 navigation device shown in the figure is a chip.

[0355] The embodiment of the present application further provides a digital processing chip. The digital processing chip integrates circuits for implementing the foregoing processor 2101 or the functions of processor 2101, and one or more interfaces. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When a memory is not integrated in the digital processing chip, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the navigation device in the foregoing embodiments according to the program code stored in the external memory.

[0356] The embodiment of the present application further provides a computer program product, which when running on a computer, causes the computer to execute the steps in the method described in the foregoing Figure 3 illustrated embodiments.

[0357] The navigation device provided by the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit so that the chip executes the above Figure 3 illustrated embodiments. Optionally, the storage unit is a storage unit inside the chip, such as a register, a cache, etc. The storage unit may also be a storage unit outside the chip located in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0358] Specifically, the foregoing processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processing unit or processor can be used to execute the steps of the foregoing Figure 3 corresponding method.

[0359] In an embodiment of the present application, a vehicle is further provided, and the vehicle includes the above-mentioned Figure 12 or Figure 21 shown device.

[0360] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0361] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for the present application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disc of a computer, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0362] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0363] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0364] In the specification, claims, and above-mentioned drawings of the present application, the terms "first", "second", "third", "fourth", etc. are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way may be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0365] Finally, it should be noted that the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. A navigation method, characterized in that, Including: Obtaining navigation data and lane perception data, where the navigation data includes the navigation path when the vehicle is traveling, and the lane perception data includes information on at least one lane determined based on information collected in the environment where the vehicle is located; Generating a navigation guidance based on the navigation data and the lane perception data, and displaying the navigation guidance, where the navigation guidance is used to identify the drivable lanes corresponding to the navigation path in the environment, and the drivable lanes include the path on which the vehicle travels obtained based on the lane perception data.

2. The method according to claim 1, characterized in that, The generating the navigation guidance based on the navigation data and the lane perception data includes: Using map data and the vehicle's historical movement information as inputs to a prediction model, and outputting a predicted path of the vehicle, where the prediction model is used to output the driving path of the vehicle based on the input data, the historical movement information includes information generated by the vehicle's movement in a historical period, and the map data includes data of the map of the area where the vehicle travels in the navigation data; Generating the navigation guidance based on the predicted path, the navigation path, and the lane perception data.

3. The method according to claim 2, characterized in that, The generating the navigation guidance based on the predicted path, the navigation path, and the lane perception data includes: If the navigation path includes a path where the vehicle changes lanes, obtaining a first path and a second path based on the navigation path and the predicted path, where the first path is the path for the vehicle to enter the target lane from the current lane, and the second path includes the driving path planned for the vehicle after it travels to the target lane, and the target lane is one of the at least one lane; Determining a plurality of driving points based on the first path and the second path, where the plurality of driving points include an in-curve driving point and an out-curve driving point, the in-curve driving point includes a point determined based on the first path, and the out-curve driving point includes a point determined based on the second path; Generating the navigation guidance based on the plurality of driving points.

4. The method according to claim 3, characterized in that, The generating the navigation guidance based on the plurality of driving points includes: Fitting the plurality of driving points to generate a steering guidance curve, where the steering guidance curve is used to indicate the lane on which the vehicle travels in the environment; Generating the navigation guidance based on the steering guidance curve.

5. The method according to claim 4, characterized in that, When the vehicle turns, the fitting the plurality of driving points to generate a steering guidance curve according to the curve model includes: Obtaining a set of broken-line points, where the set of broken-line points includes the current position of the vehicle, points on the un-traveled path in the predicted path, and points among the plurality of driving points that the vehicle has not reached; Fitting the set of broken-line points to generate the steering guidance curve.

6. The method according to claim 4 or 5, characterized in that, The fitting the plurality of driving points to generate a steering guidance curve further includes: When the vehicle travels to the in-curve point, fitting based on the driving points in the first path to generate the steering guidance line, and the steering guidance line points to the out-curve driving point; When the vehicle travels to the exit point of the curve, a steering guidance curve is generated by fitting based on the waypoints in the second path, and the steering guidance curve is a curve pointing from the vehicle to the end of the predicted path.

7. The method according to any one of claims 1 - 6, characterized in that, The generating of the navigation guidance lane perception data according to the navigation data and the lane perception data further includes: Determining a lane change decision of the vehicle according to the navigation data and the lane perception data, where the lane change decision is used to indicate whether the vehicle changes lanes; Generating the navigation guidance according to the lane change decision.

8. The method according to claim 7, characterized in that, The determining of the lane change decision of the vehicle according to the navigation data and the lane perception data includes: Determining the lane in which the vehicle is currently traveling according to the map data and the lane perception data, where the map data includes data of the map of the vehicle traveling area in the navigation data; Obtaining the probability of the vehicle transferring to each of N lanes from the current lane according to the lane perception data and the navigation data, where N is a positive integer, and the N lanes are determined from the lane perception data; Determining the transfer path of the vehicle according to the probability of the vehicle transferring to each of N lanes from the current lane; Obtaining the lane change decision according to the transfer path.

9. The method according to claim 8, characterized in that, The obtaining of the probability of the vehicle transferring to each of N lanes from the current lane according to the lane perception data and the navigation data includes: Taking the map data and the lane perception data as the input of a classification network, and dividing the map into multiple regions through the classification network, where the multiple regions include high-confidence regions, and the high-confidence regions include regions with a confidence level higher than a preset value; Expanding the map according to the high-confidence regions to obtain an expanded map; Obtaining the probability of the vehicle transferring to each of N lanes from the current lane according to the expanded map and the navigation path.

10. The method according to claim 8 or 9, characterized in that, The generating of the navigation guidance according to the lane change decision includes: If the lane change decision is to change lanes and drive, generating the navigation guidance according to the transfer path; If the lane change decision is to keep driving in the lane, keeping the generated guiding line indicating lane keeping as the navigation guidance.

11. The method according to any one of claims 1 - 10, characterized in that, The obtaining process of the lane perception data includes: Obtaining the data collected by the sensors in the vehicle to obtain the scene data; Perceiving according to the scene data to obtain the lane perception data.

12. The method according to claim 11, characterized in that, The sensors of the vehicle include one or more of the following: Image sensor, radar, infrared sensor or depth sensor.

13. The method according to any one of claims 1 - 12, characterized in that, A head-up display (HUD) or a display screen is further provided in the vehicle, and the displaying of the navigation guidance includes: Displaying the navigation guidance through the HUD, and the display position of the navigation guidance matches the environment; Or, displaying a scene image of the environment where the vehicle is located on the display screen, and superimposing and displaying the navigation guidance in the lane of the scene image.

14. A navigation device, characterized in that, Including: An acquisition module that acquires navigation data and lane perception data. The navigation data includes the navigation path when the vehicle is driving, and the lane perception data includes information on at least one lane determined based on information collected in the environment where the vehicle is located; A processing module for generating a navigation guidance based on the navigation data and the lane perception data; A display module for displaying the navigation guidance, which is used to identify the drivable lanes corresponding to the navigation path in the environment. The drivable lanes include the path of the vehicle's travel obtained based on the lane perception data.

15. The device according to claim 14, characterized in that, The processing module is specifically configured to: Use the map data and the historical movement information of the vehicle as inputs to a prediction model, and output the predicted path of the vehicle. The prediction model is used to output the driving path of the vehicle based on the input data. The historical movement information includes information generated by the vehicle's movement in a historical period, and the map data includes data of the map of the vehicle's driving area in the navigation data; Generate the navigation guidance based on the predicted path, the navigation path, and the lane perception data.

16. The device according to claim 15, characterized in that, The processing module is specifically configured to: If the navigation path includes a path for the vehicle to change lanes, obtain a first path and a second path based on the navigation path and the predicted path. The first path is the path for the vehicle to enter the target lane from the current lane, and the second path includes the driving path planned for the vehicle after it travels to the target lane. The target lane is one of the at least one lane; Determine a plurality of waypoints based on the predicted path, the first path, and the second path. The plurality of waypoints include an in-curve waypoint and an out-curve waypoint. The in-curve waypoint includes a point determined based on the first path, and the out-curve waypoint includes a point determined based on the second path; Generate the navigation guidance based on the plurality of waypoints.

17. The device according to claim 16, characterized in that, The processing module is specifically configured to: Fit the plurality of waypoints to generate a steering guidance curve, which is used to indicate the lane in which the vehicle travels in the environment; Generate the navigation guidance based on the steering guidance curve.

18. The device according to claim 17, characterized in that, When the vehicle turns, the processing module is specifically configured to: Obtain a set of polyline points, which includes the current position of the vehicle, the points on the unpredicted path of the predicted path, and the polyline formed between the waypoints that the vehicle has not reached among the plurality of waypoints; Fit the set of polyline points to generate the steering guidance curve.

19. The device according to claim 17 or 18, characterized in that, The processing module is specifically configured to: When the vehicle travels to the in-curve point, fit the waypoints in the first path to generate the steering guidance line, which points to the out-curve waypoint; When the vehicle travels to the out-curve point, fit the waypoints in the second path to generate the steering guidance line, which is a curve pointing from the vehicle to the end of the predicted path.

20. The device according to any one of claims 14 - 19, characterized in that, The processing module is specifically configured to: Determine a lane change decision for the vehicle based on the navigation data and the lane perception data, where the lane change decision is used to indicate whether the vehicle changes lanes; Generate the navigation guidance according to the lane change decision.

21. The device according to claim 20, characterized in that, The processing module is specifically configured to: Determine the lane in which the vehicle is currently traveling based on the map data and the lane perception data, where the map data includes data of the map of the vehicle's driving area in the navigation data; Obtain the probability of the vehicle transferring to each of N lanes from the current lane based on the lane perception data and the navigation data, where N is a positive integer, and the N lanes are determined from the lane perception data; Determine the transfer path of the vehicle according to the probability of the vehicle transferring to each of N lanes from the current lane; Obtain the lane change decision according to the transfer path.

22. The device according to claim 21, characterized in that, The processing module is specifically configured to: Use the map data and the lane perception data as the input of a classification network, and divide the map into multiple regions through the classification network, where multiple regions include a high-confidence region, and the high-confidence region includes regions with a confidence level higher than a preset value; Expand the map according to the high-confidence region to obtain an expanded map; Obtain the probability of the vehicle transferring to each of N lanes from the current lane according to the expanded map and the navigation path.

23. The device according to claim 19 or 20, characterized in that, The processing module is specifically configured to: If the lane change decision is to change lanes and drive, generate the navigation guidance according to the transfer path; If the lane change decision is to keep driving in the lane, keep the generated guiding line indicating lane keeping as the navigation guidance.

24. The device according to any one of claims 14 - 23, characterized in that, The processing module is specifically configured to: Obtain the data collected by the sensors in the vehicle to obtain the scene data; Perform perception according to the scene data to obtain the lane perception data.

25. The device according to claim 24, characterized in that, The sensors of the vehicle include one or more of the following: Image sensor, radar, infrared sensor or depth sensor.

26. The device according to any one of claims 14 - 25, characterized in that, A head-up display (HUD) or a display screen is further provided in the vehicle. The display module is specifically configured to: Display the navigation guidance through the HUD, and the display position of the navigation guidance matches the environment; Alternatively, display the scene image of the environment where the vehicle is located on the display screen, and superimpose and display the navigation guidance in the lane of the scene image.

27. An electronic device, characterized in that, Includes a processor, the processor is coupled with a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method described in any one of claims 1 to 13 is implemented.

28. A vehicle, characterized in that, Includes an electronic device and at least one display device, the electronic device is used to implement the method described in any one of claims 1 to 13, and the at least one display screen is used to display data under the trigger of the electronic device.

29. The vehicle according to claim 28, characterized in that, The at least one display device includes a head-up display (HUD) or a display screen.

30. A computer-readable storage medium, comprising a program which, when executed by a processing unit, performs the method according to any one of claims 1 to 13.

31. A computer program product, comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by the processor, the method described in any one of claims 1 to 13 is implemented.

Images