Navigation arrow display method, vehicle-mounted equipment, readable storage medium and chip

Abstract

The invention provides a navigation arrow display method, vehicle-mounted equipment, a readable storage medium and a chip, and relates to the technical field of vehicle navigation. The method is applied to vehicle-mounted equipment in a vehicle, the vehicle is provided with a head-up display (HUD), and the method comprises the steps that it is detected that an obstacle exists in a first lane where the vehicle is located currently; determining the position of the obstacle in the view image; determining the display position of a navigation arrow according to the position of the obstacle in the view image; and displaying the navigation arrow by using the HUD according to the display position of the navigation arrow. In the technical scheme provided by the invention, the navigation arrow and the actual road condition have a relatively high fusion degree, and a relatively good user impression is achieved.

Description

Technical Field

[0001] The present application relates to the field of vehicle navigation technology, and in particular to a navigation arrow display method, a vehicle-mounted device, a readable storage medium and a chip. Background Art

[0002] With the development of computer technology, people have gradually combined augmented reality (AR) technology with head up displays (HUD) in vehicles to form a more powerful augmented reality head up display AR-HUD. The vehicle can display the AR navigation arrow through the AR-HUD while driving. At present, vehicles usually use navigation data and lane-related information to determine the position of the AR navigation arrow, and then use AR-HUD to display the AR navigation arrow at that position. However, under this display mode, the display effect of the AR navigation arrow is less robust, that is, the integration degree of the AR navigation arrow with the actual road conditions is not high, and the display effect is not good. Summary of the invention

[0003] The present application provides a navigation arrow display method, a vehicle-mounted device, a readable storage medium and a chip, which are used to solve the problem in the prior art that the AR navigation arrow is not highly integrated with the actual road conditions and the display effect is not good.

[0004] In order to achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a navigation arrow display method, which is applied to an on-board device in a vehicle, wherein the vehicle is provided with a head-up display HUD, and the method comprises: detecting the presence of an obstacle in a first lane where the vehicle is currently located; determining the position of the obstacle in a field of view image; and displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image.

[0006] The head-up display HUD may be a traditional head-up display or a virtual reality head-up display AR-HUD, which is not limited in this embodiment.

[0007] Through the method provided in this embodiment, the vehicle-mounted equipment can determine a suitable display position for the navigation arrow based on the positions of obstacles such as fences and water barriers in actual road conditions, and use the HUD to display the navigation arrow at that position, so that the navigation arrow has a high degree of integration with the actual road conditions, thereby improving the user's viewing experience.

[0008] In some embodiments, detecting that there is an obstacle in the first lane where the vehicle is currently located includes: acquiring a field of view image of the vehicle; and detecting that there is an obstacle in the first lane where the vehicle is currently located according to the field of view image. In other words, in this embodiment, the vehicle-mounted device determines whether there is an obstacle in the first lane by image recognition.

[0009] Of course, in some other embodiments, the vehicle-mounted device may also use radar and field of view images in combination to determine whether there is an obstacle in the first lane. For example, radar is first used to detect whether there is an obstacle near the vehicle. If there is an obstacle near the vehicle, field of view images are collected and identified to determine whether there is an obstacle in the first lane. The radar may be a laser radar, a millimeter wave radar, etc., which is not limited in this embodiment.

[0010] In some embodiments, a navigation arrow is displayed using the HUD according to the position of the obstacle in the field of view image, including:

[0011] When the obstacle in the first lane does not affect the passage, the HUD is used to display a navigation arrow indicating the passage through the first lane according to the position of the obstacle in the field of view image.

[0012] When the obstacle in the first lane affects the passage, but the second lane in the same direction as the first lane is passable, the HUD is used to display a navigation arrow indicating the passage from the first lane into the second lane according to the position of the obstacle in the field of view image.

[0013] Through the method provided in this embodiment, the vehicle-mounted equipment can further determine the impact of the obstacle on the traffic conditions of the first lane based on identifying the existence of the obstacle in the first lane, and display different navigation arrows according to different impact results. The display process is relatively intelligent.

[0014] In addition, when the first lane and other lanes in the same direction as the first lane are impassable due to obstacles, the on-board equipment can also display a warning message indicating that the road is impassable to remind the user.

[0015] In some embodiments, a navigation arrow is displayed using the HUD according to the position of the obstacle in the field of view image, including: determining the display position of the navigation arrow according to the position of the obstacle in the field of view image; and displaying the navigation arrow using the HUD according to the display position of the navigation arrow.

[0016] In some embodiments, when the navigation arrow is a forward arrow, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD according to the display position of the navigation arrow, including: determining a passage guide line of the navigation arrow according to the position of the obstacle in the field of view image; determining a position on the passage guide line that is a preset distance from the vehicle as the display position of the navigation arrow; and using the HUD to display the navigation arrow along the passage guide line according to the display position of the navigation arrow.

[0017] In this embodiment, the vehicle can determine the appropriate display position and traffic guide line for the navigation arrow according to the location of the obstacle, and guide the vehicle to pass safely along the traffic guide line. This method helps to drive the vehicle safely.

[0018] In some embodiments, the navigation arrow's guide line is determined based on the position of the obstacle in the field of view image, including: when the first lane in the field of view image is a straight lane, the passable area of ​​the first lane is determined based on the position of the obstacle, and the obstacle is not included in the passable area; the center line of the passable area is determined as the guide line of the first lane.

[0019] Through the method provided in this embodiment, when the first lane is a straight lane, the on-board equipment determines the center line of the passable area of ​​the first lane as the pass guide line, so that the navigation arrow can be displayed in the center of the passable area to guide the vehicle to safely pass through obstacles and avoid hitting obstacles.

[0020] In some embodiments, the passage guide line of the navigation arrow is determined according to the position of the obstacle in the field of view image, including: when the first lane in the field of view image is a curved lane, according to the position of the obstacle in the field of view image, the center line of the boundary lines on both sides of the passable area of ​​the first lane is determined, and each point on the center line is equidistant from the boundary lines on both sides of the passable area; and the center line is determined as the passage guide line of the navigation arrow.

[0021] Through the method provided in this embodiment, when the first lane is a curved lane, the traffic guide line determined by the on-board equipment can not only guide the vehicle to avoid obstacles, but also is close to the curvature of the curved lane, which not only helps to improve driving safety, but also helps to improve the display effect of the navigation arrow.

[0022] In some embodiments, when the navigation arrow is a left turn arrow or a right turn arrow, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD according to the display position of the navigation arrow, including: determining a passage guide line of the navigation arrow according to the position of the obstacle; determining a trajectory tangent of the vehicle when it is driving stably after turning left or right according to the navigation information and the vehicle's driving information; determining the intersection of the passage guide line and the trajectory tangent as the display position of the navigation arrow; using the HUD to display the navigation arrow along the direction of the vehicle's trajectory tangent according to the display position of the navigation arrow.

[0023] Through the method provided in this embodiment, under the effect of the traffic guide line, the vehicle-mounted equipment can display the left-turn arrow and the right-turn arrow in front of the actual traffic position of the vehicle, which has a better visual experience.

[0024] In some embodiments, when the obstacle is a fork in the road, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD according to the display position of the navigation arrow, including: determining the location of the fork in the road as the display position of the navigation arrow; and using the HUD to display the navigation arrow at the location of the fork in the road along the direction of the fork in the road.

[0025] Compared with displaying the navigation arrow in the middle of the fork in the road, displaying the navigation arrow starting from the fork point and along the direction of the fork in the road is easier for the driver to see and can provide better guidance.

[0026] In some embodiments, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, including: determining the initial display position of the navigation arrow according to the navigation information and position information of the vehicle; correcting the initial display position according to the position of the obstacle in the field of view image to obtain a corrected display position; wherein the corrected display position is the final display position of the navigation arrow.

[0027] In some embodiments, the obstacle includes at least one of a fence, a water barrier, a signpost, a fork in the road, a road construction fence, a pedestrian, and other vehicles.

[0028] In some embodiments, the method further includes: identifying whether the vehicle is traveling in the opposite direction based on the field of view image within a preset time; if the vehicle is traveling in the opposite direction, displaying a prompt message to remind the vehicle that it is traveling in the opposite direction to ensure driving safety.

[0029] In a second aspect, an embodiment of the present application provides a vehicle-mounted device, which is configured to execute the method shown in the first aspect above.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method shown in the first aspect above.

[0031] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method shown in the first aspect above is implemented.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, in which a computer program is stored, and when the computer program is executed by a processor, the method shown in the first aspect above can be implemented.

[0033] In a sixth aspect, an embodiment of the present application further provides a vehicle, comprising the vehicle-mounted device as shown in the second aspect above, and a head-up display HUD, wherein the vehicle-mounted device uses the HUD to display a navigation arrow.

[0034] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of the working principle of the HUD provided in an embodiment of the present application;

[0036] Figure 2 This is a schematic diagram of displaying an AR navigation arrow provided in an embodiment of the present application;

[0037] Figure 3 This is a schematic diagram of displaying danger warning information provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of a lane and lane lines provided in an embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a vehicle coordinate system provided in an embodiment of the present application;

[0040] Figure 6 is a schematic diagram of a pixel coordinate system provided in an embodiment of the present application;

[0041] Fig. 7A It is a schematic diagram of a part of the structure of a vehicle to which the navigation arrow display method provided in the embodiment of the present application is applicable;

[0042] Figure 7B is a structural diagram of an arrow determination module provided in an embodiment of the present application;

[0043] Figure 8is a schematic diagram of a steering angle provided in an embodiment of the present application;

[0044] Fig. 9 is a schematic flow chart of a navigation arrow display method provided in an embodiment of the present application;

[0045] Fig.10 Schematic diagram of the display style of the AR navigation arrow provided in the embodiment of the present application;

[0046] Figures 11A to 11C is a schematic diagram of different road traffic conditions provided by the embodiments of the present application;

[0047] Fig.12 is a schematic flowchart of an AR navigation arrow display method in a forward scenario provided by an embodiment of the present application;

[0048] Fig.13 It is a schematic diagram of determining the display position of an AR forward arrow provided by an embodiment of the present application;

[0049] Fig.14 is a schematic diagram of determining the display position of an AR forward arrow provided by another embodiment of the present application;

[0050] Fig.15 is a schematic flowchart of an AR navigation arrow display method in a turning scenario provided by an embodiment of the present application;

[0051] Fig.16 This is a schematic diagram of the display effect of the AR navigation arrow in the left turn scenario provided by an embodiment of the present application;

[0052] Fig.17 is a schematic flow chart of an AR navigation arrow display method in a fork road driving scenario provided by an embodiment of the present application;

[0053] Fig.18 It is a schematic diagram of the display effect of the AR navigation arrow display in the fork road driving scenario provided by an embodiment of the present application;

[0054] Figure 19A to Figure 19B is a schematic diagram of displaying prompt information provided by an embodiment of the present application;

[0055] Fig. 20 is a schematic diagram of displaying prompt information provided by another embodiment of the present application;

[0056] Fig.21 is a flowchart of a method for displaying an AR navigation arrow provided by another embodiment of the present application;

[0057] Fig. 22 It is a schematic diagram of correcting the display position of the AR navigation arrow provided in an embodiment of the present application;

[0058] Fig.23A is a schematic flow chart of a lane recognition method provided in an embodiment of the present application;

[0059] Fig. 23B is a flow chart of a method for determining a vehicle traveling in the wrong direction provided by an embodiment of the present application;

[0060] Fig.24 is a schematic diagram of the longitudinal distance between the target vehicle and the vehicle provided in an embodiment of the present application;

[0061] Fig.25 It is a schematic diagram of the structure of the chip provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0063] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0064] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0065] With the development of computer technology, people gradually combine augmented reality (AR) technology with head up display (HUD) in vehicles, forming a more powerful augmented reality head up display AR-HUD. AR-HUD can improve the user's driving experience and has a very wide range of applications in vehicles.

[0066] The following first provides an exemplary explanation of the display principle of the HUD.

[0067] Figure 1 Schematic diagram of the working principle of the HUD provided in the embodiment of the present application. The HUD mainly works based on the principle of optical reflection. For example, see Figure 1As shown, the HUD is arranged below the windshield in the vehicle cockpit, and can emit projection light to the windshield through a light source, and the projection light carries the relevant information to be displayed. After the projection light is reflected by the windshield, the reflected light just goes along the direction from the front field of view to the human eye and enters the driver's eyes. Since the human eye believes that light propagates in a straight line, the driver will think that the reflected light is emitted from the front field of view, and thus see the relevant information projected by the HUD in the front field of view.

[0068] Normally, the relevant information projected by the HUD includes the vehicle's instrument information (such as vehicle speed, fuel level, gear information, etc.), driving reminder information (such as vehicle collision warning information, lane departure warning information, etc.), navigation information (navigation arrows, route congestion, remaining driving time, arrival time, etc.), etc. It can be understood that compared with the traditional technology of displaying information through the car screen, the HUD can directly display the information in the front field of view. It can be understood that during the driving process of the vehicle, the driver can quickly obtain the information by directly observing the front field of view, avoiding the situation of frequently looking down at the car screen to obtain information, which can improve driving safety.

[0069] AR-HUD further adds the function of AR display on the basis of HUD. Specifically, AR-HUD can display AR images integrated with actual road conditions in the front field of view, enhance the driver's acquisition of field of view information, and provide a better user experience. Therefore, more and more cars are beginning to use AR-HUD function.

[0070] In some implementations, the vehicle can use AR-HUD to implement AR navigation, AR warning and other functions by combining navigation information, advanced driver assistance system (ADAS) information and driving field of view information (such as field of view images). Figure 2 As shown, the vehicle can use AR-HUD to display AR navigation arrows that are integrated with the actual road conditions in the front field of view based on the navigation information. Alternatively, see Figure 3 As shown, after the vehicle detects a pedestrian near the lane, it can use AR-HUD to display danger warning information at the location of the pedestrian.

[0071] At present, when a vehicle equipped with AR-HUD displays an AR navigation arrow, it usually uses navigation data, lane identification information, and lane line detection information to determine the type and position of the AR navigation arrow, and then displays the AR navigation arrow at that position. In an ideal state, the AR navigation arrow should appear at the appropriate time, look and feel consistent with the actual road, and be compatible with various complex actual road conditions. However, the current display method does not take into account obstacles such as fences, water barriers, and fork points at forks in the driving field of view. Therefore, the display effect of the AR navigation arrow is less robust, the display effect is not integrated with the actual road conditions, and the display effect is not good.

[0072] To this end, an embodiment of the present application provides a navigation arrow display method, through which a vehicle equipped with AR-HUD can display AR navigation arrows in combination with the positions of obstacles in actual road conditions, so that the AR navigation arrows have a high degree of integration with the actual road conditions, thereby improving the user experience.

[0073] First, some terms involved in the embodiments of the present application are explained below.

[0074] (1) Lanes and lane lines

[0075] A lane is an area where vehicles travel, usually between two lane lines. For ease of description, see Figure 4 As shown, in this embodiment, the lane where the vehicle is currently located is called the first lane, and other lanes in the same direction as the first lane are called the second lane, the third lane, etc., and the lane in the opposite direction of the first lane is called the opposite lane. It should be noted that in this embodiment, the first, second, and third limited lanes are used to distinguish different lanes, and this embodiment does not restrict the relative positions of the first lane, the second lane, and the third lane.

[0076] In this embodiment, the attributes of the lane line generally include line type and color, wherein the line type of the lane line generally includes dashed line and solid line, the color of the lane line generally includes yellow and white, and the number of lines in each lane line generally includes one or two. Figure 4 As shown in the figure, the two yellow lines in the middle of the road are the central double yellow lines, which are used to separate lanes in different directions. The white dotted lines in the road are used to separate different lanes in the same direction.

[0077] (2) Vehicle coordinate system

[0078] The vehicle coordinate system is a special coordinate system used to describe the motion of a vehicle. For example, see Figure 5As shown in the figure, the origin of the vehicle coordinate system is usually the center of mass of the vehicle, the direction of vehicle travel is the Z-axis direction, the vertical direction is the Y-axis direction, and the direction from the left side of the vehicle to the right side is the X-axis direction. During vehicle travel, for various objects around the vehicle, we are more concerned about the relative position of these objects to the vehicle, rather than their absolute position. Therefore, the position information of various objects (such as lanes, lane lines, obstacles, etc.) in the field of view image is usually determined based on the vehicle coordinate system.

[0079] (3) World coordinate system

[0080] The world coordinate system (WCS), also known as the measurement coordinate system, is a three-dimensional rectangular coordinate system that can be used as a reference to describe the spatial position of the camera and the object to be measured. The position of the world coordinate system can be freely determined according to the actual situation.

[0081] (4) Pixel Coordinate System

[0082] The pixel coordinate system is an image coordinate system with pixels as units, which is used to describe the position of each pixel in the image. For example, see Figure 6 As shown in the figure, the origin of the pixel coordinate system is usually the upper left vertex of the image plane, and the X-axis and Y-axis are parallel to the horizontal and vertical edges of the image respectively. The position of a pixel point in the image pixel coordinate system can be represented by (u, v). It should be noted that the vehicle coordinate system, the world coordinate system and the pixel coordinate system can be converted to each other.

[0083] The navigation arrow display method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0084] Fig. 7A FIG. 1 is a schematic diagram of a portion of a vehicle to which the navigation arrow display method provided in the embodiment of the present application is applicable. Fig. 7A As shown, the vehicle includes a field of view image acquisition module 701, a field of view image element recognition module 702, an information inference module 703, a navigation module 704, an on-board monitoring module 705, an arrow determination module 706, an alarm prompt module 707 and an AR-HUD 708.

[0085] The visual field image acquisition module 701 may be a camera for acquiring a visual field image. Exemplarily, the visual field image acquisition module 701 at least includes a camera arranged in front of the vehicle, and the visual field image at least includes a visual field image in front of the vehicle.

[0086] It should be noted that in order to obtain images of the outside of the vehicle, the camera may be located at an appropriate position outside the vehicle. For example, in the embodiment of the present application, the camera may be located near the windshield inside the vehicle, or may be arranged around the front bumper or radiator grille so as to capture images of the area in front of the vehicle. Alternatively, in order to obtain images of the side of the vehicle, the camera may be arranged near the internal side window. This embodiment does not limit the specific position of the camera.

[0087] The field of view image element recognition module 702 is used to recognize the acquired field of view image through an image recognition algorithm and determine the key information in the field of view image. For example, the field of view image is segmented through a bilateral segmentation network (BiSeNet) algorithm to obtain obstacle information such as vehicles, pedestrians, sidewalks, fences, water barriers, and road bifurcations. Alternatively, the "you only look once" (YOLO) algorithm is used to identify information such as the position and range of traffic lights and road signs in the field of view image. Alternatively, the optical character recognition (OCR) algorithm is used to further identify the text on the road sign. This embodiment does not limit the image recognition algorithm used by the field of view image element recognition module 702.

[0088] The information inference module 703 is used to further determine the actual vehicle driving information, road condition information, etc. based on the key information determined by the field of view image element recognition module 702. For example, based on the position of the vehicle in each lane at different times, it is inferred whether the vehicle is going in the opposite direction. Alternatively, it is determined whether there are obstacles in each lane based on the field of view image, and when there are obstacles in the first lane, it is determined whether the obstacles affect the passage of vehicles. Alternatively, based on the text on the signpost, traffic lights, zebra crossings, the direction of the vehicle ahead, and other information, it is inferred whether there is an intersection in the current field of view. Alternatively, based on the text on the signpost, it is inferred whether there is construction ahead.

[0089] The navigation module 704 is used to provide navigation information for the vehicle, which includes but is not limited to navigation points, intersection nodes, and types of intersections. Among them, navigation points refer to the future driving trajectory planned by the navigation application for the vehicle in combination with the location information and destination information of the vehicle. Intersection nodes are used to divide navigation points and indicate and distinguish different intersections. The types of intersections include but are not limited to forward intersections, left turn intersections, right turn intersections, U-turn intersections, forks in the road, etc.

[0090] The vehicle monitoring module 705 is used to determine the lane information and monitor the vehicle's posture information. In this embodiment, the lane information includes the attributes of the lane, the three-dimensional coordinates of the lane in the vehicle coordinate system, etc., where the attributes of the lane can be color, line type, etc. The vehicle's posture information includes position information and steering angle, etc. The position information can be longitude and latitude information, and the steering angle refers to the angle between the direction of the vehicle tire after turning left or right and the direction when no deflection occurs. For details, see Figure 8 shown.

[0091] The arrow determination module 706 is used to determine the type and position of the AR navigation arrow according to the information output by the above modules, so that the AR-HUD displays the AR navigation arrow of this type at this position.

[0092] Among them, the type of arrow can be determined according to the type of intersection, and different types of arrows have different indication functions. For example, the arrow type of the left-turn intersection is determined to be a left-turn arrow, and the left-turn arrow is used to indicate that the vehicle turns left at the intersection ahead. Or, the arrow type of the right-turn intersection is determined to be a right-turn arrow, and the right-turn arrow is used to indicate that the vehicle turns right at the intersection ahead. Or, the arrow type of the U-turn intersection is determined to be a U-turn arrow, and the U-turn arrow is used to indicate that the vehicle turns around at the intersection ahead. Or, the arrow type of the fork in the road is determined to be a fork in the road driving arrow, which is used to indicate that the vehicle drives to the fork in the road. In addition, the arrow determination module 706 can also determine the lane change arrow indicating that the vehicle changes lanes according to the actual road conditions, and the present embodiment does not specifically limit the type of arrow.

[0093] In addition, the position of the arrow can be determined based on the position of the vehicle, the position of the first lane where the vehicle is currently located, and the positions of obstacles in the first lane such as water barriers, fences, and fork points at intersections.

[0094] In some implementations, the arrow determination module 706 comprehensively analyzes the information output by the field of view image element recognition module 702, the information inference module 703, the navigation module 704, the vehicle monitoring module 705 and other modules to determine the type and position of the AR navigation arrow so that the AR-HUD displays this type of AR navigation arrow at that position.

[0095] In some other implementations, see Figure 7BAs shown, the arrow determination module 706 includes an arrow determination unit 706A and an optimization correction unit 706B. Among them, the arrow determination unit 706A determines the type and initial display position of the AR navigation arrow according to the information output by the navigation module 704 and the vehicle monitoring module 705. The optimization correction unit 706B first corrects the initial display position of the navigation arrow according to the information output by the field of view image element recognition module 702 and the information inference module 703, and determines the corrected display position of the arrow, so that the AR-HUD displays the AR navigation arrow of this type at the corrected display position.

[0096] The warning prompt module 707 is used to warn about risky scenes, suspected error scenes, etc. according to the information output by the field of view image element recognition module 702 or the information inference module 703. Exemplarily, the risky scenes include road construction ahead, obstacles on the road ahead, the vehicle driving in the wrong direction, etc. The suspected error scene includes the detected road not matching the road prompted by the navigation, such as the fork in the road prompted by the navigation is not detected, etc.

[0097] AR-HUD 708 is used to present the vehicle's instrument information, driving reminder information, navigation information, and AR images integrated with actual road conditions in the vehicle's front field of view, such as AR navigation arrows, so that the driver can easily obtain information. See above for details.

[0098] It should be noted that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the vehicle. In other embodiments, the vehicle may include more components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0099] In some embodiments, the vehicle may further include a radar, such as a laser radar, a millimeter wave radar, etc. Exemplarily, the radar may work in conjunction with the above-mentioned field of view image acquisition module 701 and field of view image element recognition module 702. For example, the radar may be used to first detect whether there are obstacles near the vehicle. If there are obstacles near the vehicle, the field of view image acquisition module 701 may be used to acquire the field of view image, and the field of view image element recognition module 702 may be used to recognize the field of view image, so as to reduce the power consumption of the image acquisition and recognition process.

[0100] The following is a detailed description of the navigation arrow display method based on the vehicle provided in this embodiment.

[0101] Fig. 9 is a schematic flow chart of a navigation arrow display method provided in an embodiment of the present application. Fig. 9 As shown, the method is executed by a corresponding module in the vehicle, and specifically includes the following steps S901 to S903.

[0102] S901, acquiring a visual field image in front of the vehicle.

[0103] In this embodiment, the vehicle obtains the front view image through the view image acquisition module 701. The view image includes the view image in front of the driver's windshield, and may also include the view images outside the left and right windows, which is not limited in this embodiment.

[0104] S902: Detect, based on the field of view image, that an obstacle exists in the first lane where the vehicle is currently located.

[0105] In this embodiment, the vehicle passes through the visual field image element recognition module 702 using an image recognition algorithm to detect whether there is an obstacle in the first lane where the vehicle is currently located. The obstacle includes a series of static objects that hinder traffic, such as fences, water barriers, bifurcations at forks, construction fences, sand piles, earth piles, large rocks, goods scattered by other vehicles, and various parked vehicles (such as bicycles, motorcycles, and electric vehicles). Alternatively, the obstacle can be a dynamic object such as a vehicle traveling outside the vehicle, a walking pedestrian, or a running animal. This embodiment does not limit the specific type of obstacles.

[0106] S903, using the AR-HUD to display an AR navigation arrow according to the location of the obstacle.

[0107] This embodiment does not limit the display style of the AR navigation arrow, which can be, for example, Fig.10 The solid arrow shown in (a) may also be in the form of Fig.10 The fishbone-style arrows shown in (b) may of course also be of other styles.

[0108] In S903, the obstacles in the first lane have different effects on the traffic conditions of each lane depending on their size, shape and position. For example, when the obstacle is located in the middle of the first lane and is large in size, it usually affects the traffic of vehicles in the first lane. When the obstacle is located at the edge of the first lane and occupies a small area of ​​the first lane, it usually does not affect the traffic of vehicles in the first lane. It can be understood that when the traffic conditions of each road are different, the display timing and position of the AR navigation arrow are different.

[0109] To this end, the vehicle can first perform image recognition on the field of view to determine the traffic conditions of each lane, and then display the AR navigation arrow according to the traffic conditions of the lane. The details are as follows.

[0110] Case 1, for example Fig.11A As shown, there is an obstacle in the first lane where the vehicle is currently located, but it is passable. In this case, the vehicle combines the position of the obstacle and the position of the first lane to display an AR navigation arrow indicating the passage of the first lane.

[0111] Case 2, for example Fig. 11B As shown, the first lane where the vehicle is currently located has an obstacle and is impassable, but the second lane in the same direction is passable. In this case, the vehicle combines the location of the obstacle, the location of the first lane, and the location of the second lane to display an AR navigation arrow instructing the vehicle to drive from the first lane to the second lane. Alternatively, in this case, the vehicle displays a prompt message, which is used to prompt: the current lane is impassable, please drive to another lane.

[0112] Case 3, for example Fig. 11C As shown, the first lane where the vehicle is currently located and other lanes in the same direction have obstacles and are impassable. In this case, the vehicle does not display the AR navigation arrow, but uses the AR-HUD to display a prompt message, which is used to prompt that the current road is impassable.

[0113] It can be seen that this embodiment can intelligently display the AR navigation arrow in combination with the location of the obstacle, so that the display effect of the AR navigation arrow is more in line with the actual road conditions and has a better look and feel.

[0114] The following describes in detail the display of the AR navigation arrow for the above situations 1 to 3 respectively.

[0115] Case 1: There is an obstacle in the first lane but it is passable.

[0116] The vehicle may encounter various driving scenarios during driving, such as moving forward, turning left at an intersection, turning right at an intersection, turning around, driving to a fork in the road, etc. In different driving scenarios, the AR-HUD can display an AR navigation arrow that matches the actual road conditions in the driving scenario in combination with the actual field of view image. The following uses the vehicle moving forward, turning left at an intersection, and driving to a fork in the road as examples to illustrate the display method of the AR navigation arrow provided in this embodiment.

[0117] (1) The vehicle is driving in the first lane

[0118] In this embodiment, the vehicle is currently traveling in the first lane, and according to the navigation information, the vehicle should continue to move forward in the first lane. In this embodiment, the vehicle moving forward in the first lane means that the vehicle continues to move forward along the direction of the first lane. Since the first lane may be a straight line or a curve, the actual path of the vehicle moving forward may be a straight line or a curve. In other words, the vehicle moving forward cannot be narrowly understood as moving forward in a straight line.

[0119] In this scenario, if the vehicle detects an obstacle in the first lane that does not affect traffic, such as Fig.11AIf there is a barrier in the first lane, an AR navigation arrow indicating forward movement, such as an AR forward arrow, is displayed in the first lane according to the position of the obstacle.

[0120] Fig.12 It is a schematic flow chart of the AR navigation arrow display method in the forward driving scenario provided by the embodiment of the present application. The method is executed by the corresponding module in the vehicle, and specifically includes the following steps S1201 to S1203.

[0121] S1201: Determine a traffic guide line for the first lane according to the position of the first lane and the position of an obstacle in the first lane.

[0122] In some embodiments, when the first lane is a straight line, the vehicle can determine a passable area of ​​the first lane, and determine the center line of the passable area as the passable guide line of the first lane. The passable area is a passable area in the first lane that does not include obstacles.

[0123] For example Fig.13 As shown, the first lane includes a first lane line and a second lane line, and the obstacle fence occupies the first lane and is located near the second lane line. The curve AB in the figure is the baseline of the fence, A′B′ is the projection line of the fence on the second lane line, and the point of the fence baseline AB closest to the first lane line is the feature point M. Among them, the baseline of the obstacle is the intersection line of the obstacle and the road surface in the field of view image. It can be understood that the area to the left of the feature point shown in the figure is the passable area of ​​the first lane. The vehicle can determine the midpoint N of the perpendicular line between the feature point M and the first lane line, and the straight line passing through the midpoint N and parallel to the first lane is the center line of the passable area. The vehicle can determine the center line of the area as the pass guide line.

[0124] In other embodiments, when the first lane is a curve, the vehicle can determine the center line of the boundary lines on both sides of the passable area of ​​the first lane (i.e., the boundary center line), and determine the boundary center line as the pass guide line. Wherein, each point on the boundary center line is equidistant from the boundary lines on both sides; the boundary line includes the lane line and the baseline of the obstacle.

[0125] For example Fig.14 As shown in the figure, assuming that there are no obstacles in the first lane, the boundary lines of the passable area of ​​the first lane are the lane lines on the left and right sides. The lane line on the left side of the vehicle in the field of view image is curve A. 1 B 1 , the lane line on the right side of the vehicle is curve A 2 B 2 For example, point A 1 , B 1 Projected onto curve A 2 B 2 The projection points are denoted as A′1 , B′ 1 Similarly, point A 2 , B 2 Projected onto curve A 1 B 1 The projection points are denoted as A′ 2 , B′ 2 . Intercept curve A 1 B 1 A′ 2 B′ 2 Segment and Curve A 2 B 2 A′ 1 B′ 1 Segments, traverse the points on one of the curves, draw a perpendicular line to the tangent of this curve and intersect it with the other curve. Connect the midpoints of each perpendicular line segment to get the center line of this segment of the curve. Among them, the projection point of a point on the curve is the point on the curve closest to the point.

[0126] It can be understood that if there is an obstacle in the first lane, the boundary line on one or both sides of the first lane is replaced with the baseline of the obstacle on that side, and the traffic guide line of the first lane can be determined by the above method. For example, if there is no obstacle on the left side of the first lane, but there is an obstacle on the right side, then according to the left lane line A of the first lane 1 B 1 and the baseline A of the obstacle on the right side of the first lane 2 B 2 Determine the traffic guide line of the first lane. Alternatively, if there is an obstacle on the left side of the first lane but no obstacle on the right side, then use the baseline A of the obstacle on the left side of the first lane. 1 B 1 and the right lane marking A of the first lane 2 B 2 Determine the traffic guidance line for the first lane.

[0127] It should be noted that, due to the presence of an obstacle in the first lane, the obstacle causes the passable area of ​​the vehicle in the first lane to become smaller. Therefore, the passable guide line determined based on the passable area has a certain deviation from the lane centerline of the first lane. Moreover, the passable guide line is more suitable for guiding the vehicle to pass the obstacle safely.

[0128] S1202: Determine the display position of the AR forward arrow on the traffic guide line.

[0129] For example, the arrow determination module 706 of the vehicle may determine a point on the traffic guide line at a preset distance from the vehicle as the display position of the AR forward arrow. The preset distance may be 20 meters, 30 meters, etc., which is not limited in this embodiment.

[0130] S1203: Display an AR forward arrow along the traffic guide line of the first lane according to the display position of the AR forward arrow.

[0131] For example, the AR-HUD of the vehicle displays the following information along the traffic guide line of the first lane at a distance of 20 meters from the vehicle: Fig.13 In addition, this embodiment does not limit the display style of the AR forward arrow.

[0132] It should be noted that traditional technology usually does not consider the impact of obstacles in the first lane, and directly displays the AR navigation arrow in the center of the first lane. The AR navigation arrow may cause the driver to drive in the direction of the arrow and hit an obstacle, posing a certain safety hazard. In this embodiment, when the vehicle is moving forward in the first lane, if there are obstacles in the first lane that do not affect the passage, the vehicle can display the AR forward arrow in the center of the obstacle-free area in the first lane to guide the vehicle to pass safely. In contrast, the AR navigation arrow displayed in this application is not only highly integrated with the actual road conditions, but also helps to drive the vehicle safely.

[0133] (2) The vehicle is turning left at the intersection in the first lane.

[0134] In this embodiment, the vehicle is currently driving in the first lane, and according to the navigation information, the vehicle should turn left at the intersection in the first lane. In this scenario, if the vehicle detects that there is an obstacle in the first lane that does not affect the normal Fig.11A If there is a barrier in the middle, an AR navigation arrow indicating a left turn is displayed in the first lane based on the location of the obstacle.

[0135] Fig.15 1 is a schematic flow chart of an AR navigation arrow display method in a left turn scenario provided by an embodiment of the present application. The method is executed by a corresponding module in the vehicle, and specifically includes the following steps S1501 to S1505.

[0136] S1501, identifying whether there is an intersection on the road ahead based on the field of view image.

[0137] Since the navigation module 704 may have inaccurate road information, therefore, if the navigation module 704 indicates that the vehicle is about to turn left at the intersection, the field of view image element recognition module 702 can first identify whether there is an intersection in the road ahead to avoid errors in the display of the AR navigation arrow. Exemplarily, the field of view image element recognition module 702 can use image recognition algorithms such as BiSeNet, YOLO and OCR to recognize the field of view image. If it is recognized that there is a traffic light on the road ahead according to the field of view image, and / or, it is recognized that a signpost indicates that there is an intersection ahead, and / or, it is recognized that there are many vehicles traveling sideways ahead, it is determined that there is an intersection ahead.

[0138] When there is an intersection on the road ahead, step S1502 is executed. Of course, the vehicle may not execute S1501, assume the correctness of the navigation information, and directly execute S1502, which is not limited in this embodiment.

[0139] S1502: Determine a traffic guide line for the first lane according to the position of the first lane and the position of an obstacle in the first lane.

[0140] The method for determining the traffic guide line of the first lane is specifically shown in S1201, and will not be repeated here.

[0141] S1503, determining the navigation point after the vehicle turns left from the first lane.

[0142] In this embodiment, the navigation point is the future driving route planned by the navigation module 704 according to the destination information and the position of the vehicle, for example Fig.16 As shown, the navigation point after the vehicle turns left is the future driving route of the vehicle after turning left. After turning left, the vehicle usually keeps driving straight in a certain lane, so the navigation point after the vehicle turns left is basically a straight line. Therefore, the vehicle can directly obtain the navigation point after the vehicle turns left from the first lane from the navigation module 704.

[0143] S1504: Determine a display position of an AR left-turn arrow according to the navigation point after the vehicle turns left and the traffic guide line of the first lane.

[0144] In this embodiment, the display position of the AR navigation arrow in the turning scene is: the intersection of the tangent line of the navigation line point after the turn and the traffic guide line of the first lane. The turning scene includes left turn and right turn. Therefore, see Fig.16 As shown in the figure, in the left turn scenario, the display position of the AR left turn arrow is: the intersection of the tangent line of the navigation point after the left turn and the traffic guide line of the first lane, that is, Fig.16 The M point in .

[0145] S1505 , following the direction of the navigation line point after the left turn in the first lane, and according to the display position of the AR left turn arrow, using the AR-HUD to display the AR left turn arrow.

[0146] For example Fig.16 As shown, the vehicle's AR-HUD displays an AR left-turn arrow centered on point M along the direction of the navigation line point after the left turn in the first lane. Of course, the AR-HUD module can also use point M as the starting point of the AR left-turn arrow, and this embodiment does not limit the position of point M on the arrow.

[0147] (3) The vehicle is driving in the first lane towards the fork.

[0148] In this embodiment, the vehicle is currently traveling in the first lane, and according to the navigation information, the vehicle should travel in the first lane to the right fork in the road. In this scenario, the display method of the AR navigation arrow is as follows.

[0149] Fig.17 is a schematic flow chart of an AR navigation arrow display method in a fork road driving scenario provided by an embodiment of the present application. Fig.17 As shown, the method includes S1701 to S1702.

[0150] S1701, identifying a road bifurcation point in the first lane in the field of view.

[0151] In this embodiment, the visual field image element recognition module 702 can identify the bifurcation point of the fork in the road by using the BiSeNet algorithm to obtain the pixel coordinates of the bifurcation point.

[0152] S1702: Use the AR-HUD to display an AR navigation arrow at the fork point along the direction of the fork road.

[0153] For example Fig.18 As shown in the figure, the vehicle's AR-HUD takes the bifurcation point as the starting point, and displays the AR navigation arrow on the road wall on the left side of the vehicle facing the driver along the direction of the bifurcation road wall. It should be noted that compared with displaying the AR navigation arrow in the middle of the bifurcation road, displaying the AR navigation arrow along the direction of the bifurcation road wall facing the user is easier for the driver to see and can play a better guiding role.

[0154] In the second situation, there is an obstacle in the first lane and it is impassable, but the second lane is passable.

[0155] In some embodiments, see Fig.19A As shown, when there is an obstacle in the first lane and it is impassable, but the second lane is passable, the vehicle uses AR-HUD to display an AR navigation arrow to prompt the vehicle to travel from the first lane to the second lane.

[0156] In other embodiments, see Fig.19B As shown, when there is an obstacle in the first lane and it is impassable, but the second lane is passable, the vehicle can also use AR-HUD or vehicle headlights (i.e., the vehicle's headlights) to display and project a strip area on the ground. The strip area extends from the first lane to the second lane and maintains a preset distance (e.g., 1 meter, etc.) from the obstacle, so as to guide the vehicle to pass through the area and avoid the obstacle.

[0157] Optional, see Fig.19A and Fig.19BAs shown, the vehicle can also use AR-HUD to display a prompt message indicating that the first lane is impassable. For example, the prompt message is: The lane ahead is abnormal, please detour!

[0158] Case three: There are obstacles in the first lane and other lanes, making them impassable.

[0159] In this embodiment, see Fig. 20 As shown in the figure, when both the first lane and the second vehicle are blocked by obstacles, the vehicle does not display the AR navigation arrow, and uses the AR-HUD to display a prompt message to indicate that the road ahead is blocked. For example, the prompt message may be: The road ahead is blocked, please pay attention! Or, the prompt message may be: There is an obstacle on the road ahead, and it is blocked!

[0160] It should be noted that, in some embodiments, in multiple lanes in the same direction, for example Fig. 20 When both the first lane and the second lane shown are impassable, even if the opposite lane is passable, the vehicle does not use the AR-HUD to display information indicating reverse driving to bypass the obstacle to ensure driving safety.

[0161] In the above-mentioned display process of the AR navigation arrow, the vehicle directly determines and displays the AR navigation arrow based on the information output by each module. In other embodiments, the vehicle may also first determine the initial display position of the AR navigation arrow using a traditional method, and in the case of an obstacle in the first lane, correct the initial display position of the AR navigation arrow, and display the AR navigation arrow according to the corrected position. The details are as follows.

[0162] Fig.21 FIG. 1 is a flow chart of a method for displaying an AR navigation arrow provided by another embodiment of the present application. Fig.21 As shown, the method specifically includes the following steps S2101 to S2103.

[0163] S2101, determining an initial display position of the AR navigation arrow based on the vehicle's position information, navigation data, and lane line information of the first lane.

[0164] In this embodiment, this step is specifically performed by the arrow determination unit 706A in the arrow determination module 706. The specific process of determining the initial display position is referred to in the related art, and will not be described in detail in this embodiment of the application.

[0165] S2102: Correct the initial position of the AR navigation arrow according to the position of the obstacle in the first lane to obtain a corrected display position.

[0166] In this embodiment, this step is specifically performed by the optimization correction unit 706B in the arrow determination module 706. Exemplarily, the optimization correction unit 706B may first determine the target display position of the AR navigation arrow based on the information output by other modules, and use the target display position to replace the initial display position. The target display position is the corrected display position, and the specific determination process of the target display position can be found in the previous text, which will not be repeated here.

[0167] S2103, using the AR-HUD to display an AR navigation arrow according to the corrected position.

[0168] For S2101 to S2103, see Fig. 22 As shown, taking the scenario where the vehicle is driving in the first lane toward the right fork as an example, the vehicle can first use traditional technology to determine the initial display position of the AR navigation arrow as the intersection node of the fork based on the vehicle's position information, navigation data, and lane line information of the first lane. Subsequently, an image recognition algorithm is used to identify the intersection node of the fork, and the display position of the AR navigation arrow is corrected from the intersection node to the bifurcation point. Finally, the AR-HUD is used to display the AR navigation arrow at the bifurcation point along the direction of the fork in the first lane.

[0169] It should be noted that in the above-mentioned AR navigation arrow calibration process, S2101 to S2102 are algorithms executed inside the vehicle, and the user is unaware of them. In other words, in the scenario where there is an obstacle in the first lane, the vehicle does not first display the AR navigation arrow at the initial display position, and then move the AR navigation arrow to the calibrated display position, but directly calibrates the display position internally, and then directly displays the AR navigation arrow at the calibrated position.

[0170] To sum up, through the method provided in the embodiments of the present application, the electronic device can intelligently identify obstacles in the lane where the vehicle is currently located, and intelligently display traffic prompt information based on the obstacle, such as displaying an AR navigation arrow based on the location of the obstacle, or displaying traffic prompt information based on the traffic conditions of the current lane and other lanes. The display effect of the AR navigation arrow is highly integrated with the actual road conditions, and has a better user experience.

[0171] The vehicle may encounter some unreliable scenarios during driving, such as road construction ahead, navigation information that does not match the actual road conditions, or the vehicle going in the wrong direction. Based on this, the vehicle can identify the collected field of view images and issue warnings for these unreliable scenarios. The following is a detailed description of each.

[0172] (1) Road construction ahead

[0173] The vehicle can use the image recognition algorithm to identify the field of view image collected by the vehicle through the field of view image element recognition module 702 to determine whether the road ahead is under construction. If the road ahead is under construction, the AR-HUD is used to display the corresponding prompt information. For example, the prompt information can be: The road ahead is under construction, please be careful when passing.

[0174] (2) Navigation information does not match actual road conditions

[0175] Due to incomplete navigation information or untimely updates, the navigation information may not match the actual road conditions during vehicle driving. For example, the navigation application instructs the vehicle to turn right, but there is no intersection on the right. Or, the navigation application instructs the vehicle to go forward, but there is no road in the forward direction.

[0176] In this case, the vehicle can identify information such as traffic lights, road signs, lane lines, and actual road images in the field of view image through the field of view image element recognition module 702 to determine the actual road conditions ahead. When the navigation information does not match the actual road conditions, the AR-HUD is used to display the corresponding warning prompt information. For example, the warning prompt information can be: The road ahead is abnormal, please be careful.

[0177] (3) Vehicles driving in the wrong direction

[0178] During the driving process, the vehicle may drive into the opposite lane due to reasons such as the road ahead being impassable or the driver's illegal operation, which may cause certain safety hazards. In order to ensure the driving safety of the vehicle, the vehicle can identify the driving position of the vehicle based on the collected field of view images. If the vehicle is detected to be driving in the opposite lane, the corresponding prompt information will be displayed. The details are as follows.

[0179] Fig.23A is a schematic flow chart of the lane recognition method provided in the embodiment of the present application. Fig.23A As shown, the method specifically includes the following steps S2301 to S2303.

[0180] S2301, the vehicle obtains the field of view image in real time.

[0181] Generally, the vehicle collects the visual field image at a preset frequency during driving, which can be 30 Hz, 60 Hz, etc., and is not limited in this embodiment.

[0182] S2302: Determine whether the vehicle is traveling in the wrong direction based on the visual field image within a preset time period.

[0183] In this embodiment, the relative driving direction between two vehicles includes: driving in the same direction and driving in opposite directions. Driving in the same direction means that the two vehicles are driving in the same direction, and driving in opposite directions means that the two vehicles are driving in opposite directions. The vehicle can determine the relative driving direction between the vehicle and other vehicles in the same lane according to the field of view image within a preset time period through an image recognition algorithm. This embodiment does not limit the method for determining whether a vehicle is driving in the wrong direction.

[0184] In some embodiments, the vehicle uses an image recognition algorithm to identify the field of view image within a preset time period, and determines the proportion of other vehicles with their heads facing the vehicle in the same lane. When the proportion is greater than a threshold, it is determined that the vehicle may have reversed. Exemplarily, the first threshold may be 80%, 90%, etc. In addition, the preset time period may be 5S, 10S, etc., which is not limited in this embodiment.

[0185] In other embodiments, see Fig. 23B As shown, the vehicle can determine the driving direction of each vehicle in the lane in the world coordinate system based on the position information of the vehicle in the world coordinate system and the image recognition algorithm, and then determine whether the vehicle is going in the wrong direction. The method specifically includes the following steps S2302a to S2302f.

[0186] S2302a, determining the three-dimensional coordinates of the lane line in the field of view image in the vehicle coordinate system.

[0187] Based on the foregoing description, the vehicle monitoring module 705 can obtain the three-dimensional coordinates of the actual lane line in the vehicle coordinate system. It can be understood that after the vehicle projects the three-dimensional coordinates into the field of view image, each pixel point of the lane line in the field of view image can be determined.

[0188] S2302b, determining the longitudinal distance between the target vehicle and the vehicle according to the relative position relationship between the target vehicle and the lane line in the field of view image and the three-dimensional coordinates of the lane line in the vehicle coordinate system.

[0189] Taking the kth frame of the visual field image as an example, the vehicle can first segment the visual field image to find the sub-areas occupied by each target vehicle in each lane. Fig.24 As shown, the projection point of each target vehicle's wheel on the lane line is determined. Finally, the longitudinal distance Z of the target vehicle relative to the vehicle is predicted through the three-dimensional coordinates of the corresponding point on the lane line relative to the vehicle. In this embodiment, the longitudinal distance between two vehicles refers to the distance of the other vehicle in front of the vehicle along the driving direction of the vehicle.

[0190] S2302 c , according to the longitudinal distance between the ego vehicle and the target vehicle, the position coordinates of the target vehicle in the vehicle coordinate system are determined.

[0191] In some embodiments, the vehicle may determine the position coordinates of the target vehicle in the vehicle coordinate system according to the longitudinal distance between the vehicle and the target vehicle by means of back-projection transformation, as shown below.

[0192] by Figure 5 As an example, the position coordinates of the target vehicle in the vehicle coordinate system are assumed to be P = (X, Y, Z), where Z is the longitudinal distance Z of the target vehicle relative to the vehicle, which has been determined by step S2303b. Assume that the pixel point of the center of the car in the field of view image is p = (u, v), and the intrinsic parameter matrix of the camera is known and is K, then the parameters satisfy Transforming this formula yields Thus, the position coordinates of the target vehicle in the vehicle coordinate system are determined to be P = (x, Y, Z).

[0193] S2302d, determining the position coordinates of the target vehicle in the world coordinate system according to the position coordinates of the target vehicle in the vehicle coordinate system.

[0194] It can be understood that since the position of the ego vehicle in the world coordinate system and the heading information of the ego vehicle can be detected, the position coordinates of the target vehicle in the vehicle coordinate system can be converted into the position coordinates in the world coordinate system based on the information.

[0195] S2302e, determining the driving direction of the target vehicle in the world coordinate system according to the position coordinates of the target vehicle in the world coordinate system in the multiple frames of field of view images within a preset time period.

[0196] It can be understood that for each frame of field of view image, the vehicle can not only track and identify the target vehicle through an image recognition algorithm (such as an optical flow method), but also perform the method shown in S2302a to S2302d for each frame of video image to determine the position information of the target vehicle in the world coordinate system in each frame of field of view image. Therefore, according to the position information of the same target vehicle in the world coordinate system in multiple frames of images within a preset time period (such as within 5S), the driving direction of the target vehicle can be determined.

[0197] S2302f, determining whether the vehicle is traveling in the wrong direction based on the vehicle's traveling direction and the driving directions of most vehicles on the same-direction road within a preset time period.

[0198] For ease of description, in this embodiment, the direction of travel of the vehicle is referred to as the first direction, and the direction opposite to the first direction is referred to as the second direction. Based on this, in some embodiments, the vehicle can count the proportion of vehicles traveling in the first direction in the same lane to all vehicles. If the proportion is higher than a first threshold, such as 80%, it is determined that the vehicle is not traveling in the wrong direction; and if the proportion is lower than a second threshold (such as 10%), it is determined that the vehicle is traveling in the wrong direction. This embodiment does not limit the method of determining whether the vehicle is traveling in the wrong direction.

[0199] S2303: If the vehicle is driving in the wrong direction, the AR-HUD is used to display corresponding prompt information.

[0200] In one example, the prompt information may be: Attention, the vehicle may be driving in the opposite direction. Alternatively, the prompt information may be: Entered the opposite lane. This embodiment does not limit the specific content of the prompt information.

[0201] Through the above steps S2301 to S2303, the vehicle can identify the field of view images within a period of time to determine whether the vehicle is driving in the wrong direction, and give corresponding prompts when driving in the wrong direction to ensure driving safety.

[0202] To sum up, when displaying AR navigation arrows and prompt information, this embodiment comprehensively utilizes key elements in the field of view image, such as vehicles, fences, water barriers, traffic lights, signposts, and forks in the road, so that the display timing and display position of the vehicle AR navigation arrows and instruction information are more in line with the actual road conditions, and is compatible with various complex intersection scenes, with a higher user experience.

[0203] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0204] An embodiment of the present application also provides a vehicle-mounted device, which is configured to execute the navigation arrow display method shown in the above-mentioned embodiments.

[0205] An embodiment of the present application also provides a vehicle, which is provided with an AR-HUD and is configured to execute the display method of the navigation arrow shown in the above-mentioned embodiments.

[0206] It should be noted that the vehicles in this embodiment (including the vehicle itself and other vehicles around it) can be internal combustion engine vehicles that use an engine as a power source, hybrid vehicles that use an engine and an electric motor as power sources, electric vehicles that use an electric motor as a power source, and other vehicles with driving functions. This embodiment does not impose specific restrictions on the type of vehicle.

[0207] The present application also provides a chip, see Fig.25 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the display method of the navigation arrow in the above-mentioned embodiments is implemented.

[0208] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for displaying a navigation arrow provided in the above embodiments is implemented.

[0209] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the method for displaying a navigation arrow provided in the above embodiments.

[0210] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0211] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (doubledatarate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus RAM (DR RAM).

[0212] In the embodiments provided in the present application, the division of each framework or module is only a logical function division. There may be other division methods in actual implementation. For example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0213] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0214] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0216] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0217] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A navigation arrow display method, It is characterized in that An on-board device applied to a vehicle, wherein the vehicle is provided with a head-up display HUD, and the method comprises: Detecting that there is an obstacle in the first lane where the vehicle is currently located; Determining the position of the obstacle in the field of view image; The HUD is used to display a navigation arrow according to the position of the obstacle in the field of view image.

2. The method according to claim 1, It is characterized in that The detecting that there is an obstacle in the first lane where the vehicle is currently located includes: Acquiring a field of view image of the vehicle; It is detected according to the field of view image that there is an obstacle in the first lane where the vehicle is currently located.

3. The method according to claim 1 or 2, It is characterized in that The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: When the obstacle in the first lane does not affect the passage, the HUD is used to display a navigation arrow indicating the passage of the first lane according to the position of the obstacle in the field of view image.

4. The method according to any one of claims 1 to 3, It is characterized in that The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: When the obstacle in the first lane affects the passage, but the second lane in the same direction as the first lane is passable, the HUD is used to display a navigation arrow indicating the passage from the first lane into the second lane according to the position of the obstacle in the field of view image.

5. The method according to any one of claims 1 to 4, It is characterized in that The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: Determining a display position of a navigation arrow according to a position of the obstacle in the field of view image; The navigation arrow is displayed using the HUD according to the display position of the navigation arrow.

6. The method according to claim 5, It is characterized in that When the navigation arrow is a forward arrow, determining a display position of the navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow include: Determining a passage guide line of the navigation arrow according to a position of the obstacle in the field of view image; Determine a position on the traffic guide line that is a preset distance from the vehicle as a display position of the navigation arrow; Using the HUD, the navigation arrow is displayed along the passage guide line according to the display position of the navigation arrow.

7. The method according to claim 6, It is characterized in that The step of determining the passage guide line of the navigation arrow according to the position of the obstacle in the field of view image includes: When the first lane in the field of view image is a straight lane, determining a passable area of ​​the first lane according to the position of the obstacle, wherein the passable area does not include the obstacle; The area center line of the passable area is determined as the passing guide line of the first lane.

8. The method according to claim 6, It is characterized in that The step of determining the passage guide line of the navigation arrow according to the position of the obstacle in the field of view image includes: When the first lane in the field of view image is a curved lane, the center line of the boundary lines on both sides of the passable area of ​​the first lane is determined according to the position of the obstacle in the field of view image, and each point on the center line is at an equal distance from the boundary lines on both sides of the passable area; The center line is determined as a passing guide line for the navigation arrow.

9. The method according to claim 5, It is characterized in that When the navigation arrow is a left-turn arrow or a right-turn arrow, determining a display position of the navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow include: Determine the passage guide line of the navigation arrow according to the position of the obstacle; Determine, based on the navigation information and the driving information of the vehicle, a tangent of a trajectory of the vehicle when it is driving stably after turning left or right; Determine the intersection of the passage guide line and the track tangent as the display position of the navigation arrow; Using the HUD, the navigation arrow is displayed along the direction of the trajectory tangent after the vehicle turns left or right, according to the display position of the navigation arrow.

10. The method according to claim 5, It is characterized in that When the obstacle is a fork in a fork in the road, determining a display position of a navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow, comprises: Determine the location of the bifurcation point as the display position of the navigation arrow; Using the HUD, the navigation arrow is displayed at the location of the fork point along the direction of the fork road.

11. The method according to claim 5, It is characterized in that Determining a display position of a navigation arrow according to a position of the obstacle in the field of view image includes: Determining an initial display position of a navigation arrow according to the navigation information and position information of the vehicle; According to the position of the obstacle in the field of view image, the initial display position is corrected to obtain a corrected display position; the corrected display position is the display position of the navigation arrow.

12. The method according to any one of claims 1 to 11, It is characterized in that The obstacles include at least one of fences, water barriers, road signs, fork points at fork intersections, road construction fences, pedestrians, and other vehicles.

13. The method according to any one of claims 1 to 12, It is characterized in that The method further comprises: Identify whether the vehicle is traveling in the wrong direction based on the visual field image within a preset time; If the vehicle is traveling in the wrong direction, a prompt message is displayed to remind the vehicle that it is traveling in the wrong direction.

14. A vehicle-mounted device, It is characterized in that The vehicle-mounted device is configured to execute the method according to any one of claims 1 to 13.

15. A vehicle, It is characterized in that The vehicle comprises the in-vehicle device as claimed in claim 14, and a head-up display HUD.

16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A chip, It is characterized in that The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images