Navigation arrow display method, vehicle-mounted device, readable storage medium and chip

Abstract

The present application relates to the technical field of vehicle navigation. Provided are a navigation arrow display method, a vehicle-mounted device, a readable storage medium and a chip. The method is applied to a vehicle-mounted device in a vehicle, which is provided with a head-up display (HUD). The method comprises: detecting that there is an obstacle on a first lane where a vehicle is currently located; determining the position of the obstacle in a field-of-view image; determining the display position of a navigation arrow on the basis of the position of the obstacle in the field-of-view image; and on the basis of the display position of the navigation arrow, using an HUD to display the navigation arrow. In the technical solution provided by the present application, a navigation arrow has a higher degree of integration with actual road conditions, thereby providing a better user experience.

Description

Navigation arrow display method, vehicle-mounted device, readable storage medium and chip

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 22, 2023, with application number 202311573540.4 and application name “Navigation arrow display method, vehicle-mounted equipment, readable storage medium and chip”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of vehicle navigation technology, and in particular to a navigation arrow display method, a vehicle-mounted device, a readable storage medium, and a chip. Background Art

[0003] With the development of computer technology, people have gradually combined augmented reality (AR) technology with head-up displays (HUDs) in vehicles to form a more powerful augmented reality head-up display (AR-HUD). The vehicle can display the AR navigation arrow through the AR-HUD while driving. At present, vehicles usually use navigation data and lane-related information to determine the position of the AR navigation arrow, and then use the AR-HUD to display the AR navigation arrow at that position. However, under this display mode, the display effect of the AR navigation arrow is less robust, that is, the degree of integration between the AR navigation arrow and the actual road conditions is not high, and the display effect is not good. Summary of the Invention

[0004] The present application provides a navigation arrow display method, a vehicle-mounted device, a readable storage medium and a chip, which are used to solve the problem in the prior art that the AR navigation arrow is not highly integrated with the actual road conditions and the display effect is poor.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a navigation arrow display method, which is applied to an on-board device in a vehicle, wherein the vehicle is provided with a head-up display HUD, and the method includes: detecting the presence of an obstacle in a first lane where the vehicle is currently located; determining the position of the obstacle in a field of view image; and displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image.

[0007] The head-up display HUD may be a traditional head-up display or a virtual reality head-up display AR-HUD, which is not limited in this embodiment.

[0008] Through the method provided in this embodiment, the vehicle-mounted equipment can determine a suitable display position for the navigation arrow based on the location of obstacles such as fences and water barriers in the actual road conditions, and use the HUD to display the navigation arrow at that position, so that the navigation arrow has a high degree of integration with the actual road conditions, thereby improving the user experience.

[0009] In some embodiments, detecting the presence of an obstacle in a first lane currently occupied by a vehicle includes: acquiring a field of view image of the vehicle; and detecting the presence of an obstacle in the first lane currently occupied by the vehicle based on the field of view image. In other words, in this embodiment, the onboard device determines whether there is an obstacle in the first lane through image recognition.

[0010] Of course, in some other embodiments, the vehicle-mounted device may also use a combination of radar and field of view images to determine whether there is an obstacle in the first lane. For example, radar may be used to first detect whether there is an obstacle near the vehicle. If there is an obstacle near the vehicle, field of view images may be collected and recognized to determine whether there is an obstacle in the first lane. The radar may be a lidar, millimeter-wave radar, etc., and this embodiment does not limit this.

[0011] In some embodiments, a HUD is used to display navigation arrows based on the location of obstacles in the field of view image, including:

[0012] When the obstacle in the first lane does not affect passage, the HUD is used to display a navigation arrow indicating passage through the first lane based on the position of the obstacle in the field of view image.

[0013] When the obstacle in the first lane affects the passage, but the second lane in the same direction as the first lane is passable, the HUD is used to display a navigation arrow indicating the passage from the first lane to the second lane based on the position of the obstacle in the field of view image.

[0014] Through the method provided in this embodiment, the vehicle-mounted equipment can further determine the impact of the obstacle on the traffic conditions of the first lane based on the identification of the existence of the obstacle in the first lane, and display different navigation arrows according to different impact results. The display process is relatively intelligent.

[0015] In addition, when the first lane and other lanes in the same direction as the first lane are blocked due to obstacles, the on-board equipment can also display a warning message indicating that the road is blocked to remind the user.

[0016] In some embodiments, the navigation arrow is displayed using the HUD according to the position of the obstacle in the field of view image, including: determining the display position of the navigation arrow according to the position of the obstacle in the field of view image; and displaying the navigation arrow using the HUD according to the display position of the navigation arrow.

[0017] In some embodiments, when the navigation arrow is a forward arrow, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD according to the display position of the navigation arrow, including: determining the passage guide line of the navigation arrow according to the position of the obstacle in the field of view image; determining the position on the passage guide line at a preset distance from the vehicle as the display position of the navigation arrow; using the HUD to display the navigation arrow along the passage guide line according to the display position of the navigation arrow.

[0018] In this embodiment, the vehicle can determine a suitable display position and a passage guide line for the navigation arrow according to the position of the obstacle, and guide the vehicle to pass safely along the passage guide line. This method helps to ensure safe driving of the vehicle.

[0019] In some embodiments, the navigation arrow's passage guide line is determined based on the position of the obstacle in the field of view image, including: when the first lane in the field of view image is a straight lane, the passable area of ​​the first lane is determined based on the position of the obstacle, and the obstacle is not included in the passable area; the center line of the passable area is determined as the passage guide line of the first lane.

[0020] Through the method provided in this embodiment, when the first lane is a straight lane, the on-board equipment determines the center line of the passable area of ​​the first lane as the pass guide line, so that the navigation arrow can be displayed in the center position of the passable area, guiding the vehicle to safely pass through obstacles and avoid hitting obstacles.

[0021] In some embodiments, the passage guide line of the navigation arrow is determined according to the position of the obstacle in the field of view image, including: when the first lane in the field of view image is a curved lane, according to the position of the obstacle in the field of view image, the center line of the boundary lines on both sides of the passable area of ​​the first lane is determined, and the distance between each point on the center line and the boundary lines on both sides of the passable area is equal; and the center line is determined as the passage guide line of the navigation arrow.

[0022] Through the method provided in this embodiment, when the first lane is a curved lane, the traffic guide line determined by the on-board equipment can not only guide the vehicle to avoid obstacles, but also is close to the curvature of the curved lane, which not only helps to improve driving safety, but also helps to improve the display effect of the navigation arrow.

[0023] In some embodiments, when the navigation arrow is a left-turn arrow or a right-turn arrow, the display position of the navigation arrow is determined according to the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD according to the display position of the navigation arrow, including: determining the passage guide line of the navigation arrow according to the position of the obstacle; determining the trajectory tangent of the vehicle when it turns left or right and is driving stably according to the navigation information and the driving information of the vehicle; determining the intersection of the passage guide line and the trajectory tangent as the display position of the navigation arrow; using the HUD to display the navigation arrow along the direction of the vehicle's trajectory tangent according to the display position of the navigation arrow.

[0024] Through the method provided in this embodiment, under the action of the traffic guide line, the vehicle-mounted equipment can display the left turn arrow and the right turn arrow in front of the actual traffic position of the vehicle, which has a better visual experience.

[0025] In some embodiments, when the obstacle is a fork in the road, the display position of the navigation arrow is determined based on the position of the obstacle in the field of view image, and the navigation arrow is displayed using the HUD based on the display position of the navigation arrow, including: determining the location of the fork in the road as the display position of the navigation arrow; and using the HUD to display the navigation arrow at the location of the fork in the road along the direction of the fork in the road.

[0026] Compared with displaying the navigation arrow in the middle of the fork in the road, displaying the navigation arrow starting from the fork point and along the direction of the fork in the road is easier for the driver to see and can provide better guidance.

[0027] In some embodiments, the display position of the navigation arrow is determined based on the position of the obstacle in the field of view image, including: determining the initial display position of the navigation arrow based on the navigation information and position information of the vehicle; correcting the initial display position based on the position of the obstacle in the field of view image to obtain a corrected display position; wherein the corrected display position is the final display position of the navigation arrow.

[0028] In some embodiments, the obstacle includes at least one of a fence, a water barrier, a signpost, a fork in the road, a road construction fence, a pedestrian, and other vehicles.

[0029] In some embodiments, the method further includes: identifying whether the vehicle is traveling in the opposite direction based on the field of view image within a preset time; if the vehicle is traveling in the opposite direction, displaying a prompt message indicating that the vehicle is traveling in the opposite direction to ensure driving safety.

[0030] In a second aspect, an embodiment of the present application provides a vehicle-mounted device, which is configured to execute the method shown in the first aspect above.

[0031] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method shown in the first aspect above.

[0032] In a fourth aspect, an embodiment of the present application provides a chip, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method shown in the first aspect above is implemented.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, which stores a computer program. When the computer program is executed by a processor, it can implement the method shown in the first aspect above.

[0034] In a sixth aspect, an embodiment of the present application further provides a vehicle, comprising the vehicle-mounted device as shown in the second aspect above, and a head-up display HUD, wherein the vehicle-mounted device uses the HUD to display navigation arrows.

[0035] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of the working principle of the HUD provided in an embodiment of the present application;

[0037] FIG2 is a schematic diagram showing an AR navigation arrow display according to an embodiment of the present application;

[0038] FIG3 is a schematic diagram showing a display of a danger warning message provided by an embodiment of the present application;

[0039] FIG4 is a schematic diagram of a lane and lane markings provided in an embodiment of the present application;

[0040] FIG5 is a schematic diagram of a vehicle coordinate system provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of a pixel coordinate system provided in an embodiment of the present application;

[0042] FIG7A is a schematic diagram of a portion of the structure of a vehicle to which the navigation arrow display method provided in an embodiment of the present application is applicable;

[0043] FIG7B is a schematic diagram of the structure of an arrow determination module provided in an embodiment of the present application;

[0044] FIG8 is a schematic diagram of a steering angle provided in an embodiment of the present application;

[0045] FIG9 is a schematic flow chart of a navigation arrow display method provided in an embodiment of the present application;

[0046] FIG10 is a schematic diagram of a display style of an AR navigation arrow provided in an embodiment of the present application;

[0047] 11A to 11C are schematic diagrams of different road traffic conditions provided by embodiments of the present application;

[0048] FIG12 is a schematic flowchart of a method for displaying an AR navigation arrow in a forward-moving scenario provided by an embodiment of the present application;

[0049] FIG13 is a schematic diagram of determining the display position of an AR forward arrow according to an embodiment of the present application;

[0050] FIG14 is a schematic diagram of determining the display position of an AR forward arrow according to another embodiment of the present application;

[0051] FIG15 is a schematic flowchart of a method for displaying AR navigation arrows in a turning scenario provided by an embodiment of the present application;

[0052] FIG16 is a schematic diagram showing the display effect of an AR navigation arrow in a left turn scenario provided by an embodiment of the present application;

[0053] FIG17 is a schematic flowchart of a method for displaying an AR navigation arrow in a fork-road driving scenario provided by an embodiment of the present application;

[0054] FIG18 is a schematic diagram showing the display effect of an AR navigation arrow display in a fork-road driving scenario provided by an embodiment of the present application;

[0055] 19A and 19B are schematic diagrams showing prompt information provided by an embodiment of the present application;

[0056] FIG20 is a schematic diagram showing prompt information provided by another embodiment of the present application;

[0057] FIG21 is a flowchart of a method for displaying an AR navigation arrow provided by another embodiment of the present application;

[0058] FIG22 is a schematic diagram of correcting the display position of an AR navigation arrow provided in an embodiment of the present application;

[0059] FIG23A is a schematic flowchart of a lane recognition method provided in an embodiment of the present application;

[0060] FIG23B is a flowchart of a method for determining whether a vehicle is traveling in the wrong direction according to an embodiment of the present application;

[0061] FIG24 is a schematic diagram of the longitudinal distance between the target vehicle and the vehicle according to an embodiment of the present application;

[0062] Figure 25 is a schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0064] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0065] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0066] With the development of computer technology, augmented reality (AR) technology has gradually been combined with head-up displays (HUDs) in vehicles, forming more powerful AR-HUDs. AR-HUDs can enhance the user's driving experience and have a wide range of applications in vehicles.

[0067] The following first provides an exemplary explanation of the display principle of the HUD.

[0068] FIG1 is a schematic diagram of the working principle of the HUD provided in an embodiment of the present application. The HUD mainly works based on the principle of optical reflection. For example, as shown in FIG1 , the HUD is arranged below the windshield in the vehicle cockpit, and is capable of emitting projection light to the windshield through a light source, and the projection light carries the relevant information to be displayed. After the projection light is reflected by the windshield, the reflected light just enters the driver's eyes along the direction from the front field of view to the human eye. Since the human eye believes that light propagates in a straight line, the driver will believe that the reflected light is emitted from the front field of view, and thus see the relevant information projected by the HUD in the front field of view.

[0069] Typically, the relevant information projected by the HUD includes the vehicle's instrument information (such as speed, fuel level, gear information, etc.), driving reminder information (such as vehicle collision warning information, lane departure warning information, etc.), navigation information (navigation arrows, route congestion, remaining driving time, arrival time, etc.), etc. It can be understood that compared to the traditional technology of displaying information through the car screen, the HUD can directly display the information in the front field of view. It can be understood that during the vehicle's driving process, the driver can quickly obtain the information by directly observing the front field of view, avoiding the situation of frequently looking down at the car screen to obtain information, which can improve driving safety.

[0070] AR-HUD further enhances HUD with AR display capabilities. Specifically, AR-HUD displays an AR image integrated with actual road conditions in the forward field of view, enhancing the driver's understanding of the field of view and providing a better user experience. Consequently, an increasing number of cars are adopting AR-HUD.

[0071] In some implementations, the vehicle can use AR-HUD to implement AR navigation, AR warning and other functions by combining navigation information, advanced driver assistance system (ADAS) information and driving field of view information (such as field of view images). For example, as shown in Figure 2, the vehicle can use AR-HUD to display AR navigation arrows that are integrated with the actual road conditions in the front field of view based on the navigation information. Alternatively, as shown in Figure 3, after detecting pedestrians near the lane, the vehicle can use AR-HUD to display danger warning information at the location of the pedestrian.

[0072] Currently, when a vehicle equipped with an AR-HUD displays an AR navigation arrow, it usually uses navigation data, lane recognition information, and lane line detection information to determine the type and position of the AR navigation arrow, and then displays the AR navigation arrow at that position. Ideally, the AR navigation arrow should appear at the appropriate time, look and feel consistent with the actual road, and be compatible with various complex actual road conditions. However, the current display method does not take into account obstacles such as fences, water barriers, and fork points at intersections in the driving field of view. Therefore, the display effect of the AR navigation arrow is less robust, the display effect is not well integrated with the actual road conditions, and the display effect is not good.

[0073] To this end, an embodiment of the present application provides a navigation arrow display method, through which a vehicle equipped with AR-HUD can display AR navigation arrows in combination with the position of obstacles in actual road conditions, so that the AR navigation arrows have a high degree of integration with the actual road conditions, thereby improving the user experience.

[0074] First, some terms involved in the embodiments of this application are explained below.

[0075] (1) Lanes and lane lines

[0076] A lane is an area where vehicles travel, typically between two lane markings. For ease of description, as shown in Figure 4, this embodiment refers to the lane the vehicle is currently in as the first lane, and lanes traveling in the same direction as the first lane as the second lane, third lane, and so on. Lanes traveling in the opposite direction of the first lane are referred to as opposite lanes. It should be noted that in this embodiment, the first, second, and third defined lanes are used to distinguish different lanes; this embodiment does not restrict the relative positions of the first, second, and third lanes.

[0077] In this embodiment, lane marking attributes typically include line type and color. Lane marking line types typically include dashed and solid lines, lane marking colors typically include yellow and white, and each lane marking typically includes one or two lines. For example, as shown in FIG4 , the two yellow lines in the middle of the road are double yellow center lines, used to separate lanes in different directions. A dashed white line in the road separates lanes in the same direction.

[0078] (2) Vehicle coordinate system

[0079] The vehicle coordinate system is a special coordinate system used to describe the movement of a vehicle. For example, as shown in Figure 5, the coordinate origin of the vehicle coordinate system is usually the center of mass of the vehicle, the direction of vehicle travel is the Z-axis, the vertical direction is the Y-axis, and the direction from the left side of the vehicle to the right side is the X-axis. During vehicle travel, for various objects around the vehicle, we are more concerned about the relative position of these objects to the vehicle, rather than their absolute position. Therefore, the position information of various objects in the field of view image (such as lanes, lane lines, obstacles, etc.) is usually determined based on the vehicle coordinate system.

[0080] (3) World coordinate system

[0081] The world coordinate system (WCS), also known as the measurement coordinate system, is a three-dimensional rectangular coordinate system used to describe the spatial positions of the camera and the object being measured. The position of the world coordinate system can be freely determined based on the actual situation.

[0082] (4) Pixel coordinate system

[0083] The pixel coordinate system is an image coordinate system with pixels as units, which is used to describe the position of each pixel in the image. For example, as shown in Figure 6, the coordinate origin of the pixel coordinate system is usually the upper left corner vertex of the image plane, and the X-axis and Y-axis are parallel to the horizontal and vertical edges of the image respectively. The position of the pixel point in the image pixel coordinate system can be expressed as (u, v). It should be noted that the vehicle coordinate system, the world coordinate system and the pixel coordinate system can be converted into each other.

[0084] The navigation arrow display method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0085] Figure 7A is a schematic diagram of a portion of the structure of a vehicle applicable to the navigation arrow display method provided in an embodiment of the present application. As shown in Figure 7A , the vehicle includes a field of view image acquisition module 701, a field of view image element recognition module 702, an information inference module 703, a navigation module 704, an onboard monitoring module 705, an arrow determination module 706, a warning prompt module 707, and an AR-HUD 708.

[0086] The visual field image acquisition module 701 may be a camera for acquiring a visual field image. Exemplarily, the visual field image acquisition module 701 includes at least a camera disposed in front of the vehicle, and the visual field image includes at least a visual field image in front of the vehicle.

[0087] It should be noted that to capture images of the vehicle's exterior, the camera can be located at an appropriate location on the vehicle's exterior. For example, in embodiments of the present application, the camera can be located near the vehicle's windshield, or positioned around the front bumper or radiator grille to capture images of the area in front of the vehicle. Alternatively, to capture images of the side of the vehicle, the camera can be located near the interior side windows. This embodiment does not limit the specific location of the camera.

[0088] The field of view image element recognition module 702 is used to identify the acquired field of view image through an image recognition algorithm and determine the key information in the field of view image. For example, the field of view image is segmented by a bilateral segmentation network (BiSeNet) algorithm to obtain information on obstacles such as vehicles, pedestrians, sidewalks, fences, water barriers, and road bifurcations. Alternatively, the position and range of traffic lights and road signs in the field of view image are identified through a "you only look once" (YOLO) algorithm. Alternatively, the text on the road sign is further identified through an optical character recognition (OCR) algorithm. This embodiment does not limit the image recognition algorithm used by the field of view image element recognition module 702.

[0089] The information inference module 703 is used to further determine actual vehicle driving information, road condition information, and other information based on the key information determined by the field of view image element recognition module 702. For example, based on the position of the vehicle in each lane at different times, it is used to infer whether the vehicle is driving in the wrong direction. Alternatively, it is used to determine whether there are obstacles in each lane based on the field of view image, and if there is an obstacle in the first lane, it is used to determine whether the obstacle affects the vehicle's passage. Alternatively, based on information such as the text on the signpost, traffic lights, zebra crossings, and the direction of the vehicle ahead, it is used to infer whether there is an intersection within the current field of view. Alternatively, based on the text on the signpost, it is used to infer whether there is construction ahead.

[0090] Navigation module 704 is used to provide navigation information for the vehicle, including but not limited to navigation points, intersection nodes, and intersection types. A navigation point refers to the future driving trajectory planned by the navigation application based on the vehicle's location and destination information. Intersection nodes are used to separate navigation points and indicate and distinguish different intersections. Intersection types include but are not limited to forward intersections, left-turn intersections, right-turn intersections, U-turn intersections, and forks in the road.

[0091] The onboard monitoring module 705 is used to determine lane information and monitor the vehicle's position and posture information. In this embodiment, lane information includes lane attributes and the three-dimensional coordinates of the lane in the vehicle coordinate system. Lane attributes may include color, line type, etc. The vehicle's position and posture information includes position information and steering angle. Position information may include longitude and latitude information. The steering angle refers to the angle between the direction of the vehicle's tires after turning left or right and the direction of the vehicle without turning. See Figure 8 for details.

[0092] The arrow determination module 706 is used to determine the type and position of the AR navigation arrow based on the information output by the above modules, so that the AR-HUD displays the AR navigation arrow of the type at the position.

[0093] Among them, the type of arrow can be determined according to the type of intersection, and different types of arrows have different indication functions. For example, the arrow type of a left-turn intersection is determined to be a left-turn arrow, and the left-turn arrow is used to indicate that the vehicle is turning left at the intersection ahead. Alternatively, the arrow type of a right-turn intersection is determined to be a right-turn arrow, and the right-turn arrow is used to indicate that the vehicle is turning right at the intersection ahead. Alternatively, the arrow type of a U-turn intersection is determined to be a U-turn arrow, and the U-turn arrow is used to indicate that the vehicle is turning around at the intersection ahead. Alternatively, the arrow type of a fork in the road is determined to be a fork-road driving arrow, and is used to indicate that the vehicle is driving towards the fork in the road. In addition, the arrow determination module 706 can also determine a lane change arrow to indicate that the vehicle is changing lanes based on actual road conditions. This embodiment does not specifically limit the type of arrow.

[0094] In addition, the position of the arrow can be determined based on the position of the vehicle, the position of the first lane where the vehicle is currently located, and the positions of obstacles in the first lane such as water barriers, fences, and fork points at intersections.

[0095] In some implementations, the arrow determination module 706 comprehensively analyzes the information output by modules such as the field of view image element recognition module 702, the information inference module 703, the navigation module 704, and the vehicle monitoring module 705 to determine the type and position of the AR navigation arrow so that the AR-HUD displays the AR navigation arrow of this type at that position.

[0096] In some other implementations, as shown in FIG7B , the arrow determination module 706 includes an arrow determination unit 706A and an optimization and correction unit 706B. The arrow determination unit 706A determines the type and initial display position of the AR navigation arrow based on the information output by the navigation module 704 and the vehicle monitoring module 705. The optimization and correction unit 706B first corrects the initial display position of the navigation arrow based on the information output by the field of view image element recognition module 702 and the information inference module 703, and determines the corrected display position of the arrow so that the AR-HUD displays the AR navigation arrow of that type at the corrected display position.

[0097] The warning module 707 is configured to issue warnings for risky scenarios or suspected error scenarios based on the information output by the field of view image element recognition module 702 or the information inference module 703. For example, risky scenarios include road construction ahead, obstacles on the road ahead, or the vehicle driving the wrong way. Suspected error scenarios include scenarios where the detected road does not match the road indicated by the navigation, such as when a fork in the road indicated by the navigation is not detected.

[0098] AR-HUD 708 is used to present vehicle instrument information, driving reminders, navigation information, and AR images integrated with actual road conditions in the vehicle's forward field of view, such as AR navigation arrows, so that the driver can easily obtain information. See above for details.

[0099] It should be noted that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the vehicle. In other embodiments, the vehicle may include more components than shown, or some components may be combined, separated, or arranged differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] In some embodiments, the vehicle may also include a radar, such as a laser radar or millimeter-wave radar. Exemplarily, the radar may work in conjunction with the aforementioned field of view image acquisition module 701 and field of view image element recognition module 702. For example, the radar may first detect whether there are obstacles near the vehicle. If there are obstacles near the vehicle, the field of view image acquisition module 701 may capture a field of view image, and the field of view image element recognition module 702 may recognize the field of view image, thereby reducing power consumption during the image acquisition and recognition process.

[0101] The following is a detailed description of the method for displaying navigation arrows on the vehicle provided in this embodiment.

[0102] FIG9 is a schematic flow chart of a navigation arrow display method provided in an embodiment of the present application. Referring to FIG9 , the method is executed by a corresponding module in the vehicle and specifically includes the following steps S901 to S903.

[0103] S901, acquiring a visual field image in front of the vehicle.

[0104] In this embodiment, the vehicle obtains the front view image through the view image acquisition module 701. The view image includes the view image in front of the driver's windshield, and may also include the view images outside the left and right windows, which is not limited in this embodiment.

[0105] S902: Detecting, based on the field of view image, that an obstacle exists in the first lane where the vehicle is currently located.

[0106] In this embodiment, the vehicle's field of view image element recognition module 702 uses an image recognition algorithm to detect whether there are any obstacles in the first lane the vehicle is currently in. Obstacles include a range of static objects that obstruct traffic, such as fences, water barriers, forks at intersections, construction fences, sand piles, earth piles, large rocks, cargo spilled by other vehicles, and parked vehicles (such as bicycles, motorcycles, and electric vehicles). Alternatively, obstacles can be dynamic objects, such as vehicles traveling outside the vehicle, pedestrians, and running animals. This embodiment does not limit the specific types of obstacles.

[0107] S903: Display an AR navigation arrow using the AR-HUD according to the location of the obstacle.

[0108] This embodiment does not limit the display style of the AR navigation arrow. It can be a solid line arrow as shown in (a) in Figure 10, or a fishbone arrow as shown in (b) in Figure 10. Of course, it can also be other styles.

[0109] In S903, the size, shape, and location of obstacles within the first lane will affect the traffic conditions in each lane differently. For example, if an obstacle is located in the middle of the first lane and is large, it will generally affect the traffic in the first lane. On the other hand, if an obstacle is located at the edge of the first lane and occupies a small area of ​​the first lane, it will generally not affect the traffic in the first lane. It is understood that the timing and location of the AR navigation arrow display will vary depending on the traffic conditions of each road.

[0110] To this end, the vehicle can first perform image recognition on the field of view to determine the traffic conditions of each lane, and then display AR navigation arrows based on the traffic conditions of the lanes. The details are shown below.

[0111] In case 1, as shown in FIG11A , the first lane where the vehicle is currently located has an obstacle but is passable. In this case, the vehicle combines the location of the obstacle and the location of the first lane to display an AR navigation arrow indicating that the vehicle is passing through the first lane.

[0112] In the second scenario, as shown in Figure 11B, the vehicle's current lane 1 is blocked by an obstacle, but the second lane in the same direction is passable. In this scenario, the vehicle displays an AR navigation arrow indicating the vehicle should move from the first lane to the second lane, taking into account the location of the obstacle, the first lane, and the second lane. Alternatively, the vehicle displays a prompt indicating that the current lane is blocked and that the vehicle should move to another lane.

[0113] In case three, as shown in Figure 11C, both the first lane and other lanes in the same direction are blocked and impassable. In this case, the vehicle does not display the AR navigation arrow, but instead uses the AR-HUD to display a prompt indicating that the current road is impassable.

[0114] It can be seen that this embodiment can intelligently display the AR navigation arrow in combination with the location of the obstacle, so that the display effect of the AR navigation arrow is more in line with the actual road conditions and has a better visual experience.

[0115] The following describes the display of the AR navigation arrow in detail for the above situations 1 to 3.

[0116] Case 1: There is an obstacle in the first lane but it is passable.

[0117] A vehicle may encounter various driving scenarios during its journey, such as moving forward, turning left at an intersection, turning right at an intersection, making a U-turn, or driving toward a fork in the road. In different driving scenarios, the AR-HUD can combine the actual field of view image to display AR navigation arrows that match the actual road conditions in the driving scenario. The following examples illustrate the display method of AR navigation arrows provided in this embodiment, using moving forward, turning left at an intersection, and driving toward a fork in the road as examples.

[0118] (1) The vehicle is moving forward in the first lane

[0119] In this embodiment, the vehicle is currently traveling in the first lane, and according to navigation information, the vehicle should continue to move forward in the first lane. In this embodiment, the vehicle's movement forward in the first lane means that the vehicle continues to move forward along the direction of the first lane. Since the first lane can be straight or curved, the actual path of the vehicle's movement forward can be straight or curved. In other words, the vehicle's movement forward cannot be narrowly understood as moving forward in a straight line.

[0120] In this scenario, if the vehicle detects an obstacle in the first lane that does not affect passage, such as the fence in Figure 11A, an AR navigation arrow indicating forward movement, such as an AR forward arrow, is displayed in the first lane based on the location of the obstacle.

[0121] Figure 12 is a schematic flow chart of a method for displaying an AR navigation arrow in a forward driving scenario provided by an embodiment of the present application. The method is executed by a corresponding module in the vehicle and specifically includes the following steps S1201 to S1203.

[0122] S1201: Determine a traffic guide line for the first lane based on the position of the first lane and the position of an obstacle in the first lane.

[0123] In some embodiments, when the first lane is a straight line, the vehicle may determine a passable area of ​​the first lane and determine the centerline of the passable area as the passable guide line of the first lane. The passable area is a passable area in the first lane that does not include obstacles.

[0124] For example, as shown in Figure 13, the first lane includes a first lane line and a second lane line, and the obstacle fence occupies the first lane and is located near the second lane line. The curve AB in the figure is the baseline of the fence, A'B' is the projection line of the fence on the second lane line, and the point of the fence baseline AB closest to the first lane line is the feature point M. Among them, the baseline of the obstacle is the intersection line of the obstacle and the road surface in the field of view image. It can be understood that the area to the left of the feature point shown in the figure is the passable area of ​​the first lane. The vehicle can determine the midpoint N of the perpendicular line between the feature point M and the first lane line. The straight line passing through the midpoint N and parallel to the first lane is the center line of the passable area. The vehicle can determine the center line of the area as the pass guide line.

[0125] In other embodiments, when the first lane is curved, the vehicle may determine the centerline of the boundary lines on both sides of the traversable area of ​​the first lane (i.e., the boundary centerline) and determine this boundary centerline as the traversable guide line. Each point on this boundary centerline is equidistant from the boundary lines on both sides; this boundary line includes the lane markings and the baseline of the obstacle.

[0126] For example, as shown in Figure 14, assuming there are no obstacles in the first lane, the boundaries of the first lane's traversable area are the lane lines on its left and right sides. For example, let's assume the lane line on the left of the ego vehicle in the field of view is curve A1B1, and the lane line on the right of the ego vehicle is curve A2B2. Project points A1 and B1 onto curve A2B2, denoting the projection points A'1 and B'1, respectively. Similarly, project points A2 and B2 onto curve A1B1, denoting the projection points A'2 and B'2, respectively. Intercept the A'2B'2 segment of curve A1B1 and the A'1B'1 segment of curve A2B2. Traverse the points on one curve and draw a perpendicular line to the tangent of that curve, intersecting it with the other curve. Connect the midpoints of each perpendicular line segment to obtain the centerline of that curve segment. The projection point of a point on a curve is the point on that curve closest to that point.

[0127] It will be appreciated that if there is an obstacle in the first lane, the boundary line on one or both sides of the first lane is replaced with the baseline of the obstacle on that side, and the traffic guiding line for the first lane can be determined using the above method. For example, if there is no obstacle on the left side of the first lane but an obstacle on the right side, the traffic guiding line for the first lane is determined based on the left lane line A1B1 of the first lane and the baseline A2B2 of the obstacle on the right side of the first lane. Alternatively, if there is an obstacle on the left side of the first lane but no obstacle on the right side, the traffic guiding line for the first lane is determined based on the baseline A1B1 of the obstacle on the left side of the first lane and the right lane line A2B2 of the first lane.

[0128] It should be noted that the obstacle in the first lane reduces the traversable area within the first lane. Therefore, the guide line determined based on this traversable area deviates from the centerline of the first lane to a certain extent. Furthermore, this guide line is more suitable for guiding vehicles safely through obstacles.

[0129] S1202: Determine the display position of the AR forward arrow on the traffic guide line.

[0130] For example, the vehicle arrow determination module 706 may determine a point on the traffic guide line at a preset distance from the vehicle as the display position of the AR forward arrow. The preset distance may be 20 meters, 30 meters, etc., and this embodiment is not limited to this.

[0131] S1203: Display an AR forward arrow along the traffic guide line of the first lane according to the display position of the AR forward arrow.

[0132] For example, the vehicle's AR-HUD displays an AR forward arrow as shown in FIG13 along the traffic guide line of the first lane at a distance of 20 meters from the vehicle. In addition, this embodiment does not limit the display style of the AR forward arrow.

[0133] It should be noted that traditional technologies usually do not take into account the impact of obstacles in the first lane and directly display the AR navigation arrow in the center of the first lane. The AR navigation arrow may cause the driver to drive in the direction of the arrow and hit an obstacle, posing a certain safety hazard. In this embodiment, when the vehicle is moving forward in the first lane, if there is an obstacle in the first lane that does not affect the passage, the vehicle can display the AR forward arrow in the center of the obstacle-free area in the first lane to guide the vehicle to pass safely. In contrast, the AR navigation arrow displayed in this application is not only highly integrated with the actual road conditions, but also contributes to safe driving of the vehicle.

[0134] (2) The vehicle is turning left at the intersection in the first lane

[0135] In this embodiment, the vehicle is currently traveling in lane 1, and according to navigation information, the vehicle should turn left at the intersection in lane 1. In this scenario, if the vehicle detects an obstacle in lane 1 that does not affect passage, such as the fence in Figure 11A, an AR navigation arrow indicating a left turn is displayed in lane 1 based on the location of the obstacle.

[0136] Figure 15 is a schematic flow chart of a method for displaying an AR navigation arrow in a left-turn scenario provided by an embodiment of the present application. The method is executed by a corresponding module in the vehicle and specifically includes the following steps S1501 to S1505.

[0137] S1501: Identify whether there is an intersection on the road ahead based on the field of view image.

[0138] Since the navigation module 704 may have inaccurate road information, if the navigation module 704 indicates that the vehicle is about to turn left at an intersection, the field of view image element recognition module 702 can first identify whether there is an intersection on the road ahead to avoid errors in the display of the AR navigation arrow. Exemplarily, the field of view image element recognition module 702 can use image recognition algorithms such as BiSeNet, YOLO and OCR to recognize the field of view image. If a traffic light is identified on the road ahead based on the field of view image, and / or a signpost is identified indicating that there is an intersection ahead, and / or a large number of vehicles traveling laterally ahead are identified, it is determined that there is an intersection ahead.

[0139] When there is an intersection on the road ahead, step S1502 is executed. Of course, the vehicle may not execute S1501, assume the correctness of the navigation information, and directly execute S1502. This embodiment does not limit this.

[0140] S1502: Determine a traffic guide line for the first lane based on the position of the first lane and the position of an obstacle in the first lane.

[0141] The method for determining the traffic guide line of the first lane is specifically shown in S1201 and will not be repeated here.

[0142] S1503: Determine the navigation point after the vehicle turns left from the first lane.

[0143] In this embodiment, a navigation point represents the future route planned by the navigation module 704 based on the destination information and the vehicle's location. For example, as shown in FIG16 , the navigation point after the vehicle turns left represents the vehicle's future route after the turn. After turning left, the vehicle typically maintains a stable straight line within a lane. Therefore, the navigation point after the vehicle turns left is essentially a straight line. Therefore, the vehicle can directly obtain the navigation point after the vehicle turns left from the first lane from the navigation module 704.

[0144] S1504: Determine the display position of the AR left-turn arrow based on the navigation point after the vehicle turns left and the traffic guide line of the first lane.

[0145] In this embodiment, the AR navigation arrow is displayed at the intersection of the tangent line of the navigation point after the turn and the traffic guide line of the first lane in the turning scenario. This turning scenario includes left and right turns. Therefore, as shown in Figure 16, in the left turn scenario, the AR left turn arrow is displayed at the intersection of the tangent line of the navigation point after the left turn and the traffic guide line of the first lane, which is point M in Figure 16.

[0146] S1505 , displaying an AR left-turn arrow using the AR-HUD according to the display position of the AR left-turn arrow in the direction of the navigation line point after the left turn in the first lane.

[0147] For example, as shown in Figure 16, the vehicle's AR-HUD displays an AR left-turn arrow centered on point M, in the direction of the navigation line after turning left in the first lane. Of course, the AR-HUD module can also use point M as the starting point of the AR left-turn arrow. This embodiment does not restrict the position of point M on the arrow.

[0148] (3) The vehicle is driving in the first lane towards the intersection

[0149] In this embodiment, the vehicle is currently traveling in the first lane, and according to the navigation information, the vehicle should travel in the first lane toward the right fork in the road. In this scenario, the AR navigation arrow is displayed as follows.

[0150] FIG17 is a schematic flow chart of a method for displaying AR navigation arrows in a fork-road driving scenario provided by an embodiment of the present application. Referring to FIG17 , the method includes S1701 to S1702.

[0151] S1701: Identify a road bifurcation point in the first lane in the field of view.

[0152] In this embodiment, the visual field image element recognition module 702 may identify the bifurcation point of the intersection by using the BiSeNet algorithm to obtain the pixel coordinates of the bifurcation point.

[0153] S1702: Use the AR-HUD to display an AR navigation arrow at the fork point along the direction of the fork road.

[0154] For example, as shown in Figure 18, the vehicle's AR-HUD displays an AR navigation arrow on the road wall on the left side of the vehicle, starting from the fork in the road and following the direction of the fork. It should be noted that compared to displaying the AR navigation arrow in the middle of the fork, displaying the AR navigation arrow along the direction of the fork in the road, facing the user, is easier for the driver to see and can provide better guidance.

[0155] In the second case, there is an obstacle in the first lane and it is impassable, but the second lane is passable.

[0156] In some embodiments, as shown in FIG19A , when there is an obstacle in the first lane and the vehicle cannot pass through, but the second lane is passable, the vehicle uses the AR-HUD to display an AR navigation arrow to prompt the vehicle to travel from the first lane to the second lane.

[0157] In other embodiments, as shown in FIG19B , when there is an obstacle in the first lane and the vehicle cannot pass, but the second lane is passable, the vehicle can also use AR-HUD or vehicle headlights (i.e., the vehicle's headlights) to display and project a strip area on the ground. The strip area extends from the first lane to the second lane and maintains a preset distance (e.g., 1 meter, etc.) from the obstacle to guide the vehicle to pass through the area and avoid the obstacle.

[0158] Optionally, as shown in Figures 19A and 19B , the vehicle may also use AR-HUD to display a prompt message indicating that the first lane is impassable. For example, the prompt message is: The lane ahead is abnormal, please detour!

[0159] Case three: there are obstacles in the first lane and other lanes, making them impassable.

[0160] In this embodiment, as shown in Figure 20, when both the first lane and the second vehicle are impassable due to obstacles, the vehicle does not display the AR navigation arrow, and instead uses the AR-HUD to display a prompt indicating that the road ahead is impassable. For example, the prompt could be: "Road ahead is impassable, please be aware!" or: "Road ahead is impassable due to obstacles!"

[0161] It should be noted that, in some embodiments, when multiple lanes in the same direction, such as the first lane and the second lane shown in Figure 20, are impassable, even if the opposite lane is passable, the vehicle does not use the AR-HUD to display information indicating reverse driving to bypass the obstacle, in order to ensure driving safety.

[0162] During the display of the AR navigation arrow, the vehicle directly determines and displays the AR navigation arrow based on the information output by each module. In other embodiments, the vehicle may first use traditional methods to determine the initial display position of the AR navigation arrow, and if there is an obstacle in the first lane, the initial display position of the AR navigation arrow is corrected and the AR navigation arrow is displayed according to the corrected position. The details are as follows.

[0163] FIG21 is a flow chart of a method for displaying an AR navigation arrow provided by another embodiment of the present application. Referring to FIG21 , the method specifically includes the following steps S2101 to S2103.

[0164] S2101: Determine the initial display position of the AR navigation arrow based on the vehicle's position information, navigation data, and lane line information of the first lane.

[0165] In this embodiment, this step is specifically performed by the arrow determination unit 706A in the arrow determination module 706. The specific process of determining the initial display position can be found in related art and will not be described in detail in this embodiment.

[0166] S2102: Correct the initial position of the AR navigation arrow according to the position of the obstacle in the first lane to obtain a corrected display position.

[0167] In this embodiment, this step is specifically performed by the optimization and correction unit 706B in the arrow determination module 706. For example, the optimization and correction unit 706B can first determine the target display position of the AR navigation arrow based on the information output by the other modules, and use the target display position to replace the initial display position. The target display position is the corrected display position. The specific process of determining the target display position can be found in the previous section and will not be repeated here.

[0168] S2103: Display an AR navigation arrow using the AR-HUD according to the corrected position.

[0169] For S2101 to S2103, see Figure 22. Taking the scenario of a vehicle traveling in the first lane toward a right fork in the road as an example, the vehicle can first use traditional technology to determine the initial display position of the AR navigation arrow as the intersection node of the fork in the road based on the vehicle's position information, navigation data, and lane line information of the first lane. Subsequently, an image recognition algorithm is used to identify the intersection node of the fork in the road, and the display position of the AR navigation arrow is corrected from the intersection node to the bifurcation point. Finally, the AR-HUD is used to display the AR navigation arrow at the bifurcation point along the direction of the bifurcation in the first lane.

[0170] It should be noted that during the AR navigation arrow calibration process described above, S2101-S2102 are executed internally by the vehicle, and the user is unaware of the process. In other words, if there is an obstacle in the first lane, the vehicle does not first display the AR navigation arrow at its initial position and then move it to the corrected position. Instead, the vehicle directly calibrates the display position internally and then displays the AR navigation arrow at the corrected position.

[0171] To sum up, through the method provided in the embodiments of the present application, the electronic device can intelligently identify obstacles in the lane where the vehicle is currently located, and intelligently display traffic prompt information based on the obstacle, such as displaying AR navigation arrows based on the location of the obstacle, or displaying traffic prompt information based on the traffic conditions of the current lane and other lanes. The display effect of the AR navigation arrow is highly integrated with the actual road conditions, and provides a better user experience.

[0172] While driving, a vehicle may encounter unreliable scenarios, such as road construction ahead, navigation information that doesn't match actual road conditions, or a vehicle traveling the wrong way. Based on this, the vehicle can identify the captured field of view imagery and issue warnings for these unreliable scenarios. Each of these scenarios is described in detail below.

[0173] (1) Road construction ahead

[0174] The vehicle can use an image recognition algorithm to identify the field of view image captured by the vehicle through the field of view image element recognition module 702 to determine whether the road ahead is under construction. If construction is detected, the AR-HUD displays a corresponding prompt message. For example, the prompt message may be: "Road ahead under construction, please proceed with caution."

[0175] (2) Navigation information does not match the actual road conditions

[0176] Due to incomplete navigation information or untimely updates, the navigation information may not match the actual road conditions while the vehicle is driving. For example, the navigation app may instruct the vehicle to turn right, but there is no intersection on the right. Or, the navigation app may instruct the vehicle to go forward, but there is no road in the forward direction.

[0177] In this case, the vehicle can use the field of view image element recognition module 702 to identify information such as traffic lights, road signs, lane markings, and actual road images in the field of view image to determine the actual road conditions ahead. If the navigation information does not match the actual road conditions, the AR-HUD will display a corresponding warning message. For example, the warning message may be: "The road ahead is abnormal. Please proceed with caution."

[0178] (3) Vehicles driving in the wrong direction

[0179] During traffic flow, a vehicle may drift into the oncoming lane due to reasons such as the road ahead being blocked or the driver's illegal operation, posing a safety hazard. To ensure safe driving, the vehicle can identify its location based on captured field of view images. If a vehicle is detected in the oncoming lane, a corresponding prompt message will be displayed. The details are as follows.

[0180] FIG23A is a schematic flow chart of a lane recognition method provided in an embodiment of the present application. Referring to FIG23A , the method specifically includes the following steps S2301 to S2303.

[0181] S2301, the vehicle obtains a field of view image in real time.

[0182] Typically, a vehicle collects field of view images at a preset frequency while driving, for example, 30 Hz, 60 Hz, etc., which is not limited in this embodiment.

[0183] S2302: Determine whether the vehicle is traveling in the wrong direction based on the visual field image within a preset time period.

[0184] In this embodiment, the relative travel direction between two vehicles includes: traveling in the same direction and traveling in opposite directions. Traveling in the same direction means the two vehicles are traveling in the same direction, while traveling in opposite directions means the two vehicles are traveling in opposite directions. A vehicle can use an image recognition algorithm to determine the relative travel direction between itself and other vehicles in the same lane based on field of view images within a preset time period. This embodiment does not limit the method for determining whether a vehicle is traveling in the wrong direction.

[0185] In some embodiments, the vehicle uses an image recognition algorithm to identify field of view images within a preset time period and determine the proportion of other vehicles in the same lane with their heads facing the vehicle. When this proportion exceeds a threshold, the vehicle is determined to be likely traveling the wrong way. For example, the first threshold may be 80%, 90%, etc. Furthermore, the preset time period may be 5 seconds, 10 seconds, etc., and this embodiment is not limited thereto.

[0186] In other embodiments, as shown in FIG23B , the vehicle can determine the direction of travel of each vehicle in the lane in the world coordinate system based on the vehicle's position information in the world coordinate system and an image recognition algorithm, thereby determining whether the vehicle is traveling in the wrong direction. The method specifically includes the following steps S2302a to S2302f.

[0187] S2302a: Determine the three-dimensional coordinates of the lane line in the field of view image in the vehicle coordinate system.

[0188] Based on the above description, the vehicle monitoring module 705 can obtain the three-dimensional coordinates of the actual lane line in the vehicle coordinate system. It can be understood that after the vehicle projects the three-dimensional coordinates into the field of view image, the various pixel points of the lane line in the field of view image can be determined.

[0189] S2302b: Determine the longitudinal distance between the target vehicle and the vehicle based on the relative positional relationship between the target vehicle and the lane line in the field of view image and the three-dimensional coordinates of the lane line in the vehicle coordinate system.

[0190] Taking the kth frame of the visual field image as an example, the vehicle can first segment the visual field image to find the subregions occupied by each target vehicle in each lane. Subsequently, as shown in Figure 24, the projection points of each target vehicle's wheels on the lane lines are determined. Finally, the longitudinal distance Z of the target vehicle relative to the ego vehicle is predicted using the 3D coordinates of the corresponding points on the lane lines relative to the ego vehicle. In this embodiment, the longitudinal distance between two vehicles refers to the distance between the other vehicle and the ego vehicle in the direction of travel.

[0191] S2302c: Determine the position coordinates of the target vehicle in the vehicle coordinate system based on the longitudinal distance between the ego vehicle and the target vehicle.

[0192] In some embodiments, the vehicle may determine the position coordinates of the target vehicle in the vehicle coordinate system based on the longitudinal distance between the vehicle and the target vehicle by back-projection transformation, as shown below.

[0193] Taking the vehicle coordinate system shown in Figure 5 as an example, let the position coordinates of the target vehicle in the vehicle coordinate system be P = (X, Y, Z), where Z is the longitudinal distance Z of the target vehicle relative to the ego vehicle, which has been determined in step S2303b. Let the pixel point of the vehicle center in the field of view image be p = (u, v), and the intrinsic parameter matrix of the camera is known and is K. Then the parameters satisfy By transforming this formula, we can get The position coordinates of the target vehicle in the vehicle coordinate system are thereby determined to be P = (X, Y, Z).

[0194] S2302d: Determine the position coordinates of the target vehicle in the world coordinate system based on the position coordinates of the target vehicle in the vehicle coordinate system.

[0195] It can be understood that since the position of the ego vehicle in the world coordinate system and the heading information of the ego vehicle can be detected, the position coordinates of the target vehicle in the vehicle coordinate system can be converted into the position coordinates in the world coordinate system based on this information.

[0196] S2302e: Determine the driving direction of the target vehicle in the world coordinate system based on the position coordinates of the target vehicle in the multi-frame field of view image within a preset time period.

[0197] It is understood that for each frame of the field of view image, the vehicle can not only track and identify the target vehicle using an image recognition algorithm (e.g., optical flow), but can also execute the method shown in S2302a-S2302d for each frame of the video image to determine the position information of the target vehicle in the world coordinate system for each frame of the field of view image. Therefore, based on the position information of the same target vehicle in the world coordinate system in multiple frames of images within a preset time period (e.g., within 5 seconds), the driving direction of the target vehicle can be determined.

[0198] S2302f: Determine whether the vehicle is traveling in the wrong direction based on the vehicle's driving direction and the driving directions of most vehicles on the same-direction road within a preset time period.

[0199] For ease of description, this embodiment refers to the vehicle's direction of travel as the first direction, and the direction opposite the first direction as the second direction. Based on this, in some embodiments, the vehicle may count the proportion of vehicles traveling in the first direction among all vehicles in the same lane. If this proportion is above a first threshold, such as 80%, the vehicle is determined not to be traveling in the wrong direction. If this proportion is below a second threshold (such as 10%), the vehicle is determined to be traveling in the wrong direction. This embodiment does not limit the method for determining whether the vehicle is traveling in the wrong direction.

[0200] S2303: If the vehicle is driving in the wrong direction, the AR-HUD is used to display corresponding prompt information.

[0201] In one example, the prompt message may be: Attention, the vehicle may be driving in the wrong direction. Alternatively, the prompt message may be: Entered the opposite lane. This embodiment does not limit the specific content of the prompt message.

[0202] Through the above steps S2301 to S2303, the vehicle can identify the field of view image within a period of time to determine whether the vehicle is driving in the wrong direction, and provide corresponding prompts when driving in the wrong direction to ensure driving safety.

[0203] To sum up, when displaying AR navigation arrows and prompt information, this embodiment comprehensively utilizes key elements in the field of view image such as vehicles, fences, water barriers, traffic lights, signposts, and forks in the road, so that the display timing and display location of vehicle AR navigation arrows and instruction information are more in line with actual road conditions, and is compatible with various complex intersection scenes, with a higher user experience.

[0204] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0205] An embodiment of the present application also provides a vehicle-mounted device, which is configured to execute the navigation arrow display method shown in the above-mentioned embodiments.

[0206] An embodiment of the present application also provides a vehicle, which is provided with an AR-HUD and is configured to execute the navigation arrow display method shown in the above-mentioned embodiments.

[0207] It should be noted that the vehicles in this embodiment (including the vehicle itself and other vehicles around it) can be internal combustion engine vehicles using an engine as a power source, hybrid vehicles using an engine and an electric motor as power sources, electric vehicles using an electric motor as a power source, and other vehicles with driving functions. This embodiment does not impose specific restrictions on the type of vehicle.

[0208] An embodiment of the present application also provides a chip, as shown in Figure 25, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the display method of the navigation arrow in the above embodiments.

[0209] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for displaying the navigation arrow provided in the above embodiments is implemented.

[0210] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device implements the method for displaying a navigation arrow provided in the above embodiments.

[0211] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0212] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0213] In the embodiments provided in this application, the division of each framework or module is merely a logical function division. In actual implementation, there may be other division methods, for example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0214] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0215] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0216] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0217] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0218] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A navigation arrow display method, characterized in that: An on-board device applied to a vehicle, wherein the vehicle is provided with a head-up display HUD, and the method comprises: Detecting that there is an obstacle in the first lane where the vehicle is currently located; Determining the position of the obstacle in the field of view image; The HUD is used to display a navigation arrow according to the position of the obstacle in the field of view image.

2. The method according to claim 1, characterized in that The detecting that there is an obstacle in the first lane where the vehicle is currently located includes: Acquiring a field of view image of the vehicle; It is detected according to the field of view image that there is an obstacle in the first lane where the vehicle is currently located.

3. The method according to claim 1 or 2, characterized in that: The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: When the obstacle in the first lane does not affect the passage, the HUD is used to display a navigation arrow indicating the passage of the first lane according to the position of the obstacle in the field of view image.

4. The method according to any one of claims 1 to 3, characterized in that: The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: When the obstacle in the first lane affects the passage, but the second lane in the same direction as the first lane is passable, the HUD is used to display a navigation arrow indicating the passage from the first lane into the second lane according to the position of the obstacle in the field of view image.

5. The method according to any one of claims 1 to 4, characterized in that: The method of displaying a navigation arrow using the HUD according to the position of the obstacle in the field of view image includes: Determining a display position of a navigation arrow according to a position of the obstacle in the field of view image; The navigation arrow is displayed using the HUD according to the display position of the navigation arrow.

6. The method according to claim 5, characterized in that When the navigation arrow is a forward arrow, determining a display position of the navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow include: Determining a passage guide line of the navigation arrow according to a position of the obstacle in the field of view image; Determine a position on the traffic guide line that is a preset distance from the vehicle as a display position of the navigation arrow; Using the HUD, the navigation arrow is displayed along the passage guide line according to the display position of the navigation arrow.

7. The method according to claim 6, characterized in that The step of determining the passage guide line of the navigation arrow according to the position of the obstacle in the field of view image includes: When the first lane in the field of view image is a straight lane, determining a passable area of ​​the first lane according to the position of the obstacle, wherein the passable area does not include the obstacle; The area center line of the passable area is determined as the passing guide line of the first lane.

8. The method according to claim 6, characterized in that The step of determining the passage guide line of the navigation arrow according to the position of the obstacle in the field of view image includes: When the first lane in the field of view image is a curved lane, the center line of the boundary lines on both sides of the passable area of ​​the first lane is determined according to the position of the obstacle in the field of view image, and the distances between each point on the center line and the boundary lines on both sides of the passable area are equal; The center line is determined as a passing guide line for the navigation arrow.

9. The method according to claim 5, characterized in that When the navigation arrow is a left-turn arrow or a right-turn arrow, determining a display position of the navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow include: Determine the passage guide line of the navigation arrow according to the position of the obstacle; Determine, based on the navigation information and the driving information of the vehicle, a tangent of a trajectory of the vehicle when it is driving stably after turning left or right; Determine the intersection of the passage guide line and the track tangent as the display position of the navigation arrow; Using the HUD, the navigation arrow is displayed along the direction of the trajectory tangent after the vehicle turns left or right, according to the display position of the navigation arrow.

10. The method according to claim 5, characterized in that When the obstacle is a fork in a fork in the road, determining a display position of a navigation arrow according to a position of the obstacle in the field of view image, and displaying the navigation arrow using the HUD according to the display position of the navigation arrow, comprises: Determine the location of the bifurcation point as the display position of the navigation arrow; Using the HUD, the navigation arrow is displayed at the location of the fork point along the direction of the fork road.

11. The method according to claim 5, characterized in that Determining a display position of a navigation arrow according to a position of the obstacle in the field of view image includes: Determining an initial display position of a navigation arrow according to the navigation information and position information of the vehicle; According to the position of the obstacle in the field of view image, the initial display position is corrected to obtain a corrected display position; the corrected display position is the display position of the navigation arrow.

12. The method according to any one of claims 1 to 11, characterized in that: The obstacles include at least one of fences, water barriers, road signs, fork points at fork intersections, road construction fences, pedestrians, and other vehicles.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: Identify whether the vehicle is traveling in the wrong direction based on the visual field image within a preset time; If the vehicle is traveling in the wrong direction, a prompt message is displayed to remind the vehicle that it is traveling in the wrong direction.

14. A vehicle-mounted device, characterized in that: The vehicle-mounted device is configured to execute the method according to any one of claims 1 to 13.

15. A vehicle, characterized in that: The vehicle comprises the in-vehicle device as claimed in claim 14, and a head-up display HUD.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A chip, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images