Navigation information display method, lane line tracking method, device and storage medium

Abstract

The application relates to a navigation information display method, a lane line tracking method, a device and a storage medium, and relates to the technical field of intelligent transportation. The method comprises the following steps: displaying a navigation interface; displaying a first road image in the navigation interface, the first road image being an image collected by an image collection device arranged on a vehicle in real time; superimposing and displaying navigation information on the first road image; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of a lane line on the first road image. Through the above method, the display of the navigation information is closer to the actual observed road surface information of a user, the route information can be more intuitively indicated compared with a voice prompt, and the display effect of the navigation information is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a navigation information display method, lane line tracking method, device and storage medium. Background Technology

[0002] In the field of intelligent transportation, intelligent navigation is essential to facilitate users' access to real-time road information.

[0003] In related technologies, in order to help users better obtain road information, voice instructions are usually used to remind users of road conditions or to instruct users to choose a driving route.

[0004] However, in the above methods, when providing road conditions via voice, it is impossible to adaptively adjust the broadcast information based on the user's current lane. For example, when the user says "Please move to the right two lanes," if the user is already in the right lane, the current voice prompt will cause the user to have an unnecessary information judgment process, resulting in poor road route guidance. Summary of the Invention

[0005] This application provides a navigation information display method, lane line tracking method, device, and storage medium, which can improve the accuracy of navigation information prompts by combining the content displayed on the navigation interface. The technical solution is as follows:

[0006] On the one hand, a method for displaying navigation information is provided, the method comprising:

[0007] Display the navigation interface;

[0008] The navigation interface displays a first road image, which is an image captured in real time by the image acquisition device installed in the corresponding vehicle.

[0009] Navigation information is overlaid on the first road image; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of the lane lines on the first road image.

[0010] On the other hand, a lane line tracking method is provided, the method comprising:

[0011] Acquire a first road image, which is an image captured in real time by an image acquisition device installed on the corresponding vehicle;

[0012] Lane line recognition is performed on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle;

[0013] At least one predicted lane line is obtained from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system.

[0014] Based on at least one projection point string, the at least one predicted lane line is updated to obtain the lane line in the ground coordinate system corresponding to the first road image.

[0015] On the other hand, a navigation information display device is provided, the device comprising:

[0016] The interface display module is used to display the navigation interface;

[0017] An image display module is used to display a first road image in the navigation interface. The first road image is an image captured in real time by an image acquisition device set in the corresponding vehicle.

[0018] The navigation information display module is used to overlay navigation information on the first road image; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of the lane lines on the first road image.

[0019] In one possible implementation, the device further includes:

[0020] The lane line recognition module is used to perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle;

[0021] The lane line prediction module is used to obtain at least one predicted lane line from the first road image based on a reference lane line; the reference lane line is the lane line in the ground coordinate system corresponding to the second road image; the second road image is a road image displayed in the navigation interface before the first road image.

[0022] The lane line update module is used to update the at least one predicted lane line based on at least one projection point string to obtain the lane line of the first road image in the ground coordinate system.

[0023] The navigation information display module is used to overlay and display the navigation information on the first road image based on the lane lines corresponding to the first road image in the ground coordinate system.

[0024] In one possible implementation, the navigation information display module includes:

[0025] The ground display location determination submodule is used to determine the ground display location of the navigation information in the ground coordinate system based on the lane lines corresponding to the first road image in the ground coordinate system.

[0026] The image display location determination submodule is used to project the ground display location onto the first road image based on the installation pose of the image acquisition device, so as to obtain the image display location of the navigation information in the first road image;

[0027] The navigation information display submodule is used to overlay and display the navigation information on the first road image based on the image display location.

[0028] In one possible implementation, the lane line update module includes:

[0029] A distance calculation submodule is used to calculate the lateral distance between at least one of the projected point strings and at least one of the predicted lane lines;

[0030] The matching relationship determination submodule is used to determine the matching relationship between the projection point string and the predicted lane line based on the lateral distance;

[0031] The lane line update submodule is used to update the at least one predicted lane line based on the matching relationship to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0032] In one possible implementation, the lane line update submodule includes:

[0033] The first lane line acquisition unit is used to update the first predicted lane line based on the first projection point string and the first predicted lane line that the matching relationship indicates a successful match, so as to obtain a lane line in the ground coordinate system corresponding to the first road image.

[0034] The second lane line acquisition unit is used to fit the second projection point string that is not matched according to the matching relationship, and to obtain the fitted lane line as a lane line in the ground coordinate system corresponding to the first road image.

[0035] The third lane line acquisition unit is used to acquire the second predicted lane line, which is not matched according to the matching relationship, as a lane line in the ground coordinate system corresponding to the first road image.

[0036] In one possible implementation, the third lane line acquisition unit is used to acquire the number of consecutive unsuccessful matches of the second predicted lane line;

[0037] In response to the number of consecutive unmatched attempts being less than a threshold for the number of unmatched attempts, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

[0038] In one possible implementation, the second lane line acquisition unit is used to fit the second projection point string based on a quadratic polynomial, and acquire the fitted lane line as a lane line in the ground coordinate system corresponding to the first road image.

[0039] In one possible implementation, the first lane line acquisition unit is used to establish lane line distance constraints based on the first predicted lane line and the first projection point string.

[0040] Based on the lane line distance constraint, the first predicted lane line is updated to obtain a lane line in the ground coordinate system corresponding to the first road image.

[0041] In one possible implementation, the lane line distance constraint is used to constrain the distance between the first predicted lane line and the first projection point string, such that the lateral distance between each point in the first projection point string and the first predicted lane line is less than a specified threshold.

[0042] In one possible implementation, the lane prediction module is used to obtain at least one predicted lane line of the first road image based on the reference lane line and the local motion data of the vehicle.

[0043] The local motion data refers to the motion data of the vehicle within a motion time period, which is the time period between a first time point and a second time point; the first time point is the time when the second road image was acquired, and the second time point is the time when the first road image was acquired; the first time point is before the second time point.

[0044] In one possible implementation, the lane line recognition module includes:

[0045] The lane line segmentation submodule is used to segment the first road image into lane lines and obtain a mask image of the first road image.

[0046] The pixel clustering submodule is used to cluster the pixels on the mask image to obtain the pixels corresponding to at least one lane line.

[0047] The projection submodule is used to project the clustered pixels onto the ground coordinate system to obtain at least one projection point string of the first road image.

[0048] On the other hand, a lane line tracking device is provided, the device comprising:

[0049] The image acquisition module is used to acquire a first road image, which is an image acquired in real time by the image acquisition device set up for the corresponding vehicle.

[0050] The lane line recognition module is used to perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle;

[0051] The lane line prediction module is used to obtain at least one predicted lane line from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system.

[0052] The lane line update module updates the at least one predicted lane line based on at least one projection point string to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0053] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described navigation information display method and / or lane line tracking method.

[0054] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the navigation information display method and / or lane line tracking method provided in the various optional implementations described above.

[0055] The technical solution provided in this application may include the following beneficial effects:

[0056] By overlaying navigation information onto the ground in the first road image, based on the lane line positions obtained in real time by the terminal, in the navigation interface, the navigation information is displayed more closely to the road surface information actually observed by the user. Compared with voice prompts, it can more intuitively indicate route information and improve the display effect of navigation information. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] Figure 1 This is a schematic diagram illustrating the structure of an electronic map application system according to an exemplary embodiment;

[0059] Figure 2 This is a flowchart illustrating a lane tracking method according to an exemplary embodiment of this application;

[0060] Figure 3 A schematic diagram illustrating a ground coordinate system as shown in an exemplary embodiment of this application is provided.

[0061] Figure 4 This is a flowchart illustrating a lane tracking method according to an exemplary embodiment of this application;

[0062] Figure 5 A schematic diagram illustrating lane line recognition is shown in an exemplary embodiment of this application;

[0063] Figure 6 This illustration shows a schematic diagram of the matching relationship between the projection point string and the predicted lane line provided in an exemplary embodiment of this application;

[0064] Figure 7 This invention illustrates a framework diagram of a lane tracking system corresponding to a lane tracking method provided in an exemplary embodiment of this application.

[0065] Figure 8 A flowchart illustrating a navigation information display method provided in an exemplary embodiment of this application is shown;

[0066] Figures 9-12 A schematic diagram of a navigation interface shown in an example embodiment of this application is illustrated;

[0067] Figure 13 This illustration shows a block diagram of a navigation information display device provided in an exemplary embodiment of this application;

[0068] Figure 14 A block diagram of a lane tracking device provided in an exemplary embodiment of this application is shown;

[0069] Figure 15 A structural block diagram of a computer device illustrated in an exemplary embodiment of this application is shown;

[0070] Figure 16 A structural block diagram of a computer device illustrated in an exemplary embodiment of this application is shown. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0073] This application provides a lane tracking method that can improve the classification accuracy of the obtained image classification model. For ease of understanding, several terms used in this application are explained below.

[0074] 1) Artificial Intelligence (AI)

[0075] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0076] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. The display device with image acquisition components shown in this application mainly involves computer vision and machine learning / deep learning technologies.

[0077] 2) Autonomous driving technology

[0078] Autonomous driving technology typically includes high-precision maps, environmental perception, behavior decision-making, path planning, motion control, and other technologies, and autonomous driving technology has broad application prospects.

[0079] 3) Kalman filtering

[0080] When an object is in motion, its position at the next moment can be predicted based on its current position and the equation of motion. When it reaches the next position, a sensor can observe a new position, and the two positions are fused to obtain the final position.

[0081] Kalman filtering prediction refers to predicting the next state based on the previous state. The content to be predicted includes: position and the corresponding covariance matrix.

[0082] Figure 1 This is a schematic diagram illustrating the structure of an electronic map application system according to an exemplary embodiment. The system includes a server 120 and a terminal 140.

[0083] Server 120 is a single server, or a combination of several servers, or a virtualization platform, or a cloud computing service center.

[0084] Terminal 140 can be a terminal device with an interface display function. For example, terminal 140 can be a car navigation device, mobile phone, tablet computer, e-book reader, smart glasses, smartwatch, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, and desktop computer, etc.

[0085] Terminal 140 and server 120 are connected via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0086] Optionally, the system may also include a management device (not shown in the figure) connected to the server 120 via a communication network. Optionally, the communication network may be a wired network or a wireless network.

[0087] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0088] The aforementioned terminal 140 includes an electronic map application that can provide various applications based on electronic maps, such as navigation, route query, location search, etc.

[0089] In one possible implementation, terminal 140 includes an image acquisition device, or terminal 140 is externally connected to an image acquisition device.

[0090] In one possible implementation, terminal 140 includes a motion sensor, such as an inertial measurement unit (IMU) sensor.

[0091] Server 120 can be a server corresponding to an electronic map application. Server 120 can provide online map services, such as online navigation, online route query, and online location search, to the electronic map application in terminal 140 based on road network data; alternatively, server 120 can also provide road network data to terminal 140, allowing the electronic map application in terminal 140 to provide local map services based on the road network data. Simultaneously, server 140 can also perform lane tracking based on terminal data, providing navigation instructions to the electronic map application in terminal 140.

[0092] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of a lane tracking method, which is executed by a computer device, wherein the computer device can be implemented as described above. Figure 1 The system shown is represented by server 120 or terminal 140. For example... Figure 2 As shown, the lane tracking method can be implemented through the following steps:

[0093] Step 210: Obtain the first road image; the first road image is an image captured in real time by the image acquisition device set up for the corresponding vehicle.

[0094] In one possible implementation, the first road image can be an image containing lane lines acquired in real time by a computer device through an image acquisition device corresponding to the vehicle. The image acquisition device can be installed on the vehicle to collect image information on the road surface where the vehicle is traveling or parked. Alternatively, the image acquisition device can be installed on any terminal capable of collecting image information on the road surface.

[0095] Step 220: Perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on vehicles.

[0096] Unlike related technologies that establish a coordinate system on an image plane to obtain lane line information, this embodiment obtains lane line information using a ground coordinate system based on the vehicle. In one possible implementation, a ground coordinate system corresponding to the vehicle is established with a designated location of the vehicle as the origin. Schematic, this designated location can refer to the center of the vehicle, which can be the center of multiple wheels. For example, in a four-wheeled vehicle, the center can be the intersection of the diagonal lines connecting the four wheels, i.e., the center of each wheel. When establishing the ground coordinate system, the designated location of the vehicle is used as the center, with the X-axis pointing in the direction of vehicle movement and the Y-axis pointing to the left or right of that direction. The XY plane is parallel to or coincides with the ground. Figure 3 A schematic diagram illustrating a ground coordinate system as shown in an exemplary embodiment of this application is provided, as follows: Figure 3As shown, taking a four-wheeled vehicle with the specified position as the center as an example, the vehicle's center position 310 is obtained as the origin, the X-axis points in the direction of vehicle movement (i.e., the front of the vehicle), and the Y-axis points to the left of the direction of vehicle movement, thus making the XY plane parallel or coincident with the ground. In this ground coordinate system, the unit length on each coordinate axis is meters (m). The lane line information obtained through the ground coordinate system carries scale information. Compared to related technologies that establish a coordinate system on the image plane and obtain lane lines in pixels, the lane line information obtained based on the ground coordinate system is more consistent with reality. It allows for direct acquisition of lane lines on the ground, facilitating subsequent application and processing. For example, it can be directly applied to in-vehicle AR (Augmented Reality) navigation, lane keeping assist, and autonomous driving positioning. For instance, it can display distance information in the navigation interface or render virtual objects that match the actual physical world on the navigation interface.

[0097] Step 230: Obtain at least one predicted lane line from the first road image based on the reference lane line; the reference lane line is the lane line in the ground coordinate system corresponding to the second road image; the second road image is the road image displayed in the navigation interface before the first road image.

[0098] In the process of lane line tracking, the lane lines in subsequent image frames are often tracked and predicted based on the lane line detection results in the existing image frames. In other words, the lane lines in the first road image are tracked and predicted based on the lane line detection results in the second road image.

[0099] The second road image corresponds to at least one reference lane line in the ground coordinate system. It should be noted that the number of reference lane lines may be the same as or different from the number of predicted lane lines.

[0100] Step 240: Based on at least one projection point string, update at least one predicted lane line to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0101] The acquisition of predicted lane lines can be represented by acquiring predicted lane line parameters, which are used to represent the predicted lane line. In other words, given the predicted lane line parameters, the y-axis coordinates can be obtained based on the x-axis coordinates acquired during vehicle movement, thereby constructing the predicted lane line. Correspondingly, updating the predicted lane line can be achieved by updating the predicted lane line parameters.

[0102] In summary, the lane tracking method provided in this application, during the lane tracking process, obtains a string of projection points corresponding to each lane line in the first road image based on the projection of lane line pixels in the first road image onto the ground coordinate system. Subsequently, after obtaining the predicted lane line based on the lane line parameters of the second road image, the predicted lane line is updated based on the string of projection points to obtain the lane line of the first road image in the ground coordinate system, thereby achieving lane line tracking in the ground coordinate system. Through the above method, the comprehensiveness and effectiveness of utilizing information identified from the image in the lane tracking scenario are improved, thereby enabling more continuous and stable lane line tracking, and thus improving the accuracy of lane line tracking based on images.

[0103] Figure 4 This is a flowchart illustrating an exemplary embodiment of the present application of a lane tracking method, which is executed by a computer device, wherein the computer device can be implemented as described above. Figure 1 The system shown is represented by server 120 or terminal 140. For example... Figure 4 As shown, the lane tracking method can be implemented through the following steps:

[0104] Step 410: Obtain the first road image, which is an image acquired by the real-time image acquisition device set up for the corresponding vehicle.

[0105] In one possible implementation, the first road image can be an image captured by a computer device through an image acquisition device installed on its corresponding vehicle; alternatively, the first road image can also be an image captured by the image acquisition device of the corresponding vehicle and then transmitted to the computer device. This application does not limit the source of the first road image.

[0106] In one possible implementation, the image acquisition device can be implemented as a monocular camera.

[0107] Step 420: Perform lane line recognition on the first road image to obtain at least one projection point string of the first road image.

[0108] In one possible implementation, lane line recognition is performed on the first road image to obtain at least one projection point string of the first road image, which can be achieved as follows:

[0109] Lane line segmentation is performed on the first road image to obtain a mask image of the first road image;

[0110] Cluster the pixels on the mask image to obtain the pixels corresponding to at least one lane line;

[0111] The clustered pixels are projected onto the ground coordinate system to obtain at least one projection point string of the first road image.

[0112] Optionally, lane line segmentation algorithms based on deep learning can be used to segment the lane lines in the first road image. For example, the LaneNet network can be used to process the first road image to obtain a mask image for image segmentation.

[0113] Optionally, the pixels on the mask image can be clustered using image clustering algorithms. For example, methods such as DB-SCAN can be used to cluster pixels on the same lane line on the mask image together.

[0114] In one possible implementation, the process of segmenting lane lines and clustering pixels on the Mask image can be accomplished using an instance segmentation deep learning method to obtain well-clustered lane lines.

[0115] Schematic, the points on the image are When projecting the clustered pixels in an image onto a ground coordinate system, we can have:

[0116]

[0117] in, This represents the homography matrix from the image to the ground coordinate system. The setting of this homography matrix can be determined by the position of the vehicle corresponding to the image acquisition device. The parameter represents the normalization parameter, and x and y represent the horizontal and vertical coordinates of the pixels in the image in the ground coordinate system, respectively.

[0118] The clustered pixels can be projected onto the ground coordinate system using IPM (Inverse Perspective Mapping) projection. Figure 5 A schematic diagram illustrating lane line recognition in an exemplary embodiment of this application is shown, such as... Figure 5 As shown, after lane line segmentation, the first road image 510 is used to obtain a mask image 520 corresponding to the first road image 510; after clustering the pixels on the mask image 520, the pixels on the mask image corresponding to different lane lines are obtained, such as... Figure 5 As shown, the Mask image 520 has three lane lines, each corresponding to a pixel. IPM projection is performed on the clustered pixels to obtain N IPM projection point strings 530, where N represents the number of lane lines identified in the first road image.

[0119] Step 430: Obtain at least one predicted lane line from the first road image based on the reference lane line; the reference lane line is the lane line in the ground coordinate system corresponding to the second road image; the second road image is the road image displayed in the navigation interface before the first road image.

[0120] In one possible implementation, lane tracking can be performed based on a reference lane line using various filters such as Kalman filters or particle filters to obtain at least one predicted lane line from the target image frame.

[0121] To fully utilize other information during vehicle movement and achieve a more stable lane tracking effect, in this embodiment, at least one predicted lane line is obtained from the first road image by combining the vehicle's local motion data. In other words, obtaining at least one predicted lane line from the first road image can be achieved as follows:

[0122] Based on the reference lane lines and the local motion data of the vehicle, at least one predicted lane line is obtained from the first road image;

[0123] The local motion data refers to the vehicle's motion data within a specific time period, which is the time between a first time point and a second time point. The first time point is the time when the second road image was acquired, and the second time point is the time when the first road image was acquired. The first time point is prior to the second time point.

[0124] In one possible implementation, the second road image is an image that is spaced a specified number of frames apart from the first road image, and the acquisition time of the second road image is earlier than the acquisition time of the first road image.

[0125] To improve the continuity of lane line tracking, the specified frame number can be set to 1, meaning that the second road image is the previous frame of the first road image in the image frame sequence captured by the image acquisition device.

[0126] Optionally, the local motion data is acquired based on the inertial measurement unit (IMU) and wheel speed sensors installed on the corresponding vehicle; before acquiring at least one predicted lane line of the first road image based on the reference lane line and the vehicle's local motion data, the method further includes:

[0127] Acquire inertial measurement unit parameters and wheel speed sensor parameters during the motion period;

[0128] Local motion data of the vehicle is acquired based on the parameters of the inertial measurement unit and the wheel speed sensor.

[0129] Among them, the inertial measurement unit parameters are the data collected by the inertial measurement unit during the motion period, and the wheel speed sensor parameters are the parameters collected by the wheel speed sensor during the motion period.

[0130] In one possible implementation, the local motion parameters include a relative translation distance and a relative rotation angle, wherein the relative translation distance refers to the translation distance between the vehicle's position at the second time point and the vehicle's position at the first time point; and the relative rotation angle refers to the rotation angle between the vehicle's direction of travel at the second time point and the vehicle's direction of travel at the first time point.

[0131] This method acquires local motion data of the vehicle based on inertial measurement unit parameters and wheel speed sensor parameters, including:

[0132] The relative rotation angle is obtained based on the parameters of the inertial measurement unit during the motion period;

[0133] The relative translational distance is obtained based on the wheel speed sensor parameters during the movement period.

[0134] The inertial measurement unit parameter is represented by an angular velocity measurement value. By integrating the angular velocity measurement value, the relative rotation angle can be obtained. The wheel speed sensor parameter is represented by a speed measurement value. By integrating the speed measurement value, the relative translation distance can be obtained.

[0135] Taking the second road image as the frame preceding the first road image as an example, when obtaining at least one predicted lane line from the first road image based on the reference lane line and the local motion data of the vehicle, a lane line prediction equation can be established based on the reference lane line and the local motion data of the vehicle. This lane line prediction equation can be expressed as:

[0136]

[0137] Among them, the This represents the lane line parameters of the second road image (frame k). This represents the lane line parameters of the first road image (frame k+1). This represents the relative translation distance of the vehicle's position at frame k+1 compared to its position at frame k. This represents the relative rotation angle of the vehicle's driving direction at frame k+1 relative to the vehicle's driving direction at frame k.

[0138] The prediction equation for the covariance of the lane line parameters in the (k+1)th frame is:

[0139]

[0140] in, Relative to Jacobi, F is relative to Jacobi, This represents the covariance of the lane line parameters in the previous frame. This represents the covariance of the lane line parameters in the current frame, and R is the noise of the local motion, which can be set by relevant personnel according to the motion parameters.

[0141] Covariance changes according to the confidence level of the current prediction result and is used to represent the uncertainty of the prediction result. When the prediction result fluctuates greatly, the lane lines are unclear, or there is severe occlusion, the confidence level of the prediction result decreases. Therefore, covariance can play a role in stabilizing lane line prediction.

[0142] In one possible implementation, this application uses the Extended Kalman Filter (EKF) algorithm to predict lane lines. Therefore, the reference lane line and local motion data are input into the Extended Kalman Filter algorithm to obtain at least one predicted lane line of the first road image.

[0143] Step 440: Calculate the lateral distance between at least one projected point string and at least one predicted lane line.

[0144] The above process can be implemented by calculating the distance between each point in the projected point string and the predicted value corresponding to that point in the predicted lane line. For illustration, the coordinates of a point in the projected point string are (x, y), and the corresponding coordinates of that point in the predicted lane line are (x, y). ), with y The distance between points is obtained as the lateral distance between the point and the predicted lane line. The lateral distance between a projection point string and a predicted lane line can be the average of the lateral distances between each projection point in the projection point string and the predicted lane line.

[0145] Step 450: Based on the lateral distance, determine the matching relationship between the projected point string and the predicted lane line.

[0146] In one possible implementation, the computer device may set a lateral distance threshold, and the matching relationship between the projected point string and the predicted lane line is determined based on the lateral distance threshold; illustratively, when the lateral distance between the projected point string and the predicted lane line is less than the lateral distance threshold, it is determined that the projected point string matches the predicted lane line; when the lateral distance between the projected point string and the predicted lane line is greater than the lateral distance threshold, it is determined that the projected point string does not match the predicted lane line.

[0147] The matching relationship can indicate three situations: the projected point string matches the predicted lane line, there is a projected point string that does not match any predicted lane line, and there is a predicted lane line that does not match any projected point string.

[0148] Step 460: Based on the matching relationship, at least one predicted lane line is updated to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0149] Figure 6 This illustration shows a schematic diagram of the matching relationship between the projected point string and the predicted lane line provided in an exemplary embodiment of this application, such as... Figure 6 As shown, IPM point strings 1-3 match the predicted lane lines 1-3 respectively, while IPM point string 4 does not match the predicted lane lines, and the predicted lane lines 4 do not match the projected point strings.

[0150] In this embodiment, for the first projection point string and the first predicted lane line indicating a successful match, the first predicted lane line is updated based on the first projection point string to obtain a lane line in the ground coordinate system corresponding to the first road image. That is, as follows... Figure 6 The IPM point strings 1-3 shown are matched with any of the predicted lane lines 1-3. The IPM point strings and predicted lane lines that are successfully matched are the first projected point strings and the first predicted lane lines mentioned above. For example, IPM point string 1 and predicted lane line 1.

[0151] The process of updating the first predicted lane line is implemented as follows:

[0152] Establish lane line distance constraints based on the first predicted lane line and the first projection point string;

[0153] Based on lane line distance constraints, the first predicted lane line is updated to obtain a lane line in the ground coordinate system corresponding to the first road image.

[0154] In one possible implementation, the lane line distance constraint is used to constrain the distance between the first predicted lane line and the first projected point string, such that the lateral distance between each point in the first projected point string and the first predicted lane line is less than a specified threshold.

[0155] The lateral distance is used to indicate the distance between each point in the first projection point string and the first predicted lane line in a direction perpendicular to the vehicle's travel direction.

[0156] In one possible implementation, based on the x-coordinate of each point in the first projection point string, the point corresponding to each point in the first predicted lane line is obtained, and the distance between each point and the point corresponding to each point in the first predicted lane line is obtained as the lateral distance.

[0157] The lane distance constraint can be implemented as a lane distance constraint equation. This equation constrains the distance between the first predicted lane line and the first projected point string, ensuring that the distance between a point in the first projected point string and its corresponding point in the first predicted lane line is less than a specified threshold. For example, if the lane distance constraint indicates that the distance from a point in the first projected point string to the first tracking lane line is 0, then the lane distance constraint equation can be expressed as follows:

[0158]

[0159] in, This represents the x-coordinate of a point in the projection point string. The table lists the ordinates of the points in the projection point string, where a, b, and c represent the lane line parameters corresponding to the predicted lane lines. This indicates observation noise.

[0160] Based on the above lane distance constraint equation, the lane parameters corresponding to the predicted lane lines are corrected so that the above lane distance constraint equation holds.

[0161] The lane distance constraint equation can be linearized, and the linearized lane distance constraint equation is expressed as:

[0162]

[0163] in, , This represents the homography matrix from the image to the ground. For small increments, stacking multiple points to form linearized lane distance constraint equations together results in linearized lane distance constraint equations corresponding to the projected point string.

[0164]

[0165] The lane line parameters of the first predicted lane line are updated based on the linearized lane line distance constraint equation corresponding to the projection point string.

[0166] In one possible implementation, this application uses the extended card EKF algorithm to predict lane lines. The above-mentioned update of the lane line parameters of the first predicted lane line based on the linearized lane line distance constraint equation corresponding to the projection point string can be implemented as EKF update of the lane line parameters of the first predicted lane line based on the linearized lane line distance constraint equation corresponding to the projection point string.

[0167] In this embodiment of the application, for the second projection point string indicating a mismatch, the second projection point string is fitted, and the fitted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image; that is, as shown below. Figure 6 The IPM point string shown is the unmatched second projection point string.

[0168] For unmatched second projection point strings, they can be considered as newly added lane lines in the target image frame. By fitting the second projection point strings, the new lane lines corresponding to the target image frame can be obtained.

[0169] In order to improve the fitting effect by ensuring that the predicted lane line obtained by fitting passes through as many projection points as possible, the second projection point string is fitted based on a quadratic polynomial in this embodiment, and the fitted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

[0170] For a given sequence of projected points, a linear equation is established to solve for the parameters of the lane lines corresponding to that sequence. This linear equation can be expressed as:

[0171]

[0172] The parameters of the lane lines corresponding to this projection point string are L=[a,b,c],[x] i ,y i The symbol represents the projection point in the projection point string. At the same time, an initial covariance value can be set for the newly added lane line to detect the newly added lane line.

[0173] In this embodiment of the application, for a second predicted lane line that does not match the matching relationship, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

[0174] The second predicted lane line may fail to match due to factors such as occlusion or missing ground lane lines causing missing projection point strings, or due to interruption or merging of ground lane lines, resulting in the absence of a projection point string corresponding to the second predicted lane line. In order to avoid accidental deletion or misdisplay of the predicted lane line, in one possible implementation, the number of consecutive unmatched second predicted lane lines is obtained.

[0175] In response to the number of consecutive unmatched attempts being less than a threshold, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

[0176] In one possible implementation, the second predicted lane line is deleted in response to the number of consecutive unmatched attempts reaching a threshold.

[0177] Optionally, when the number of consecutive unmatched attempts for the second predicted lane line is less than the unmatched attempt threshold, the lane line parameters of the currently predicted second lane line are maintained. When tracking lane lines in the next frame of the first road image, the lane line parameters of the second lane line predicted from the first road image are used as the basis for tracking the lane lines in the next frame.

[0178] In one possible scenario, the vehicle is in the initial startup phase, or the electronic map application in the vehicle has just been turned on, and there are no detected lane lines in the computer device, and the first road image does not contain the second road image. In the above case: in response to the lack of a reference lane line, at least one projection point string is fitted to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0179] In other words, the lane lines obtained by fitting the projection point string obtained based on the target image frame are obtained as the lane lines corresponding to the first road image in the ground coordinate system.

[0180] Optionally, an initial covariance value is set for the lane lines obtained by fitting the projection point string, so as to predict the lane lines of subsequent image frames based on the initial covariance value and the lane lines corresponding to the target image frame.

[0181] In summary, the lane tracking method provided in this application, during the lane tracking process, obtains a string of projection points corresponding to each lane line in the first road image based on the projection of lane line pixels in the first road image onto the ground coordinate system. Subsequently, after obtaining the predicted lane line based on the lane line parameters of the second road image, the predicted lane line is updated based on the string of projection points to obtain the lane line of the first road image in the ground coordinate system, thereby achieving lane line tracking in the ground coordinate system. Through the above method, the comprehensiveness and effectiveness of utilizing information identified from the image in the lane tracking scenario are improved, thereby enabling more continuous and stable lane line tracking, and thus improving the accuracy of lane line tracking based on images.

[0182] Meanwhile, this application improves the lane line prediction effect by predicting lane lines based on reference lane lines and local motion data of vehicles, and enhances the continuity and stability of lane line output during lane line tracking in different image frames.

[0183] Taking the lane lines in the ground coordinate system corresponding to the first road image as an example, which are obtained based on reference lane lines and local vehicle motion data, Figure 7 This application shows a framework diagram of a lane tracking system corresponding to a lane tracking method provided in an exemplary embodiment of the present application, as follows: Figure 7As shown, the lane tracking system may include an image data preprocessing module 710, a local motion estimation module 720, a system initialization module 730, an EKF prediction module 740, a lane matching module 750, an EKF update module 760, and a lane addition and deletion module 770. The image data preprocessing module performs lane segmentation, lane pixel clustering, and IPM projection on the acquired first road image to obtain at least one projection point string corresponding to the target image frame. The local motion estimation module 720 processes the inertial measurement unit parameters and wheel speed sensor parameters acquired by the IMU sensor and wheel speed sensor to obtain local motion data. Before performing EKF prediction, it determines whether the system needs to be initialized, i.e., whether there are lanes to be tracked in the system. If no lane line to be tracked exists, the system initialization module 730 fits at least one projection point string obtained by the image data preprocessing module to obtain the lane line corresponding to the target image frame in the ground coordinate system and outputs it. If a lane line to be tracked exists in the system, the EKF prediction module 740 predicts the lane line based on the reference lane line and the local motion data of the vehicle to obtain at least one predicted lane line of the first road image. The lane line matching module 750 is used to match at least one predicted lane line based on at least one projection point string to determine the matching relationship between each projection point string and each predicted lane line. The EKF update module 760 is used to update at least one predicted lane line based on the matching relationship. The process of updating at least one predicted lane line can be referred to Figure 4 The relevant content of the illustrated embodiment will not be repeated here; the lane line addition and deletion module 770 is used to add or delete lane lines corresponding to the target image based on the matching relationship of the lane line matching module 750. This process can be referred to Figure 4 The relevant content of the illustrated embodiment will not be repeated here; finally, the lane lines in the ground coordinate system corresponding to the determined target image frame are output.

[0184] It should be noted that the lane tracking method provided in this application can be applied to driving assistance functions implemented through lane lines, such as AR navigation functions, autonomous driving positioning functions, ADAS (Advanced Driving Assistance System) assisted driving functions (such as lane keeping function, lane departure warning function, autonomous driving function), etc. Based on Figure 2 or Figure 4 The lane tracking method shown in the embodiment is compared with the navigation information display method provided in this application. Figure 2 Alternatively, based on the embodiment shown in the figure, taking the lane tracking method for implementing AR navigation function, with the ground coordinate system coinciding with the ground as an example, Figure 8The flowchart illustrates a navigation information display method provided in an exemplary embodiment of this application. This navigation information display method can be executed by a terminal, which can be implemented as follows: Figure 1 The terminal 140 shown is as follows: Figure 8 As shown, the method may include the following steps:

[0185] Step 810: Display the navigation interface.

[0186] Step 820: Display the first road image in the navigation interface. The first road image is an image captured in real time by the image acquisition device set in the corresponding vehicle.

[0187] Step 830: Display navigation information overlaid on the first road image; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of the lane lines on the first road image.

[0188] In one possible implementation, before step 830, the following is performed: Figure 2 or Figure 4 The lane line tracking method shown obtains the lane lines corresponding to the first road image in the ground coordinate system, and displays the navigation information on top of the first road image based on the lane lines corresponding to the first road image in the ground coordinate system.

[0189] To improve the display effect of navigation information by displaying it close to and parallel to the ground in the first road image, one possible implementation is to overlay the navigation information on top of the first road image based on the lane lines corresponding to the first road image in the ground coordinate system.

[0190] Based on the lane lines in the ground coordinate system corresponding to the first road image, determine the ground display position of the navigation information in the ground coordinate system;

[0191] Based on the installation pose of the image acquisition device, the ground display position is projected onto the first road image to obtain the image display position of navigation information in the first road image;

[0192] The navigation information is overlaid on top of the first road image based on the image display location.

[0193] In one possible implementation, based on the installation location of the image acquisition device, the angular relationship between the image acquisition device and the ground is obtained, and the ground navigation information is projected onto the upper layer of the first road image based on the angular relationship to obtain the navigation information.

[0194] The installation information for the image acquisition device includes its installation location and orientation.

[0195] Specifically, when projecting the ground display position onto the first road image, the display positions of each pixel of the navigation information in the ground display position are projected onto the first road image to obtain the image display position of each pixel. Based on the image display positions of each pixel, the navigation information is rendered on the upper layer of the first road image.

[0196] In one possible implementation, the process of projecting each pixel from its ground display position to its image display position can be implemented as projecting the coordinates of each pixel in the ground coordinate system to its coordinates in the image coordinate system.

[0197] Since the ground navigation information is added based on the lane lines in the ground coordinate system corresponding to the first road image (i.e., the lane lines based on the ground), when the image is projected onto the upper layer of the first road image based on the correspondence between the image acquisition device and the ground, the obtained navigation information is also parallel to the lane lines in the first road image, thus achieving the effect of displaying the navigation information close to the ground.

[0198] The navigation information may include road sign direction information, lane change instructions, and distance information for executing driving commands. Figures 9-12 A schematic diagram of a navigation interface shown in an example embodiment of this application is illustrated below. Figure 9 As shown, the road sign direction information 920 can be displayed as an arrow in the first road image of the current navigation interface to indicate that the first lane 910 (the lane between lane lines 911 and 912) in the first road image is a straight lane. This road sign direction information 920 is displayed close to the ground based on the lane line positions in the first road image of the current navigation interface, i.e., the positions of lane lines 911 and 912, and the lane line positions in the second road image previously displayed on the current navigation interface (not shown in the figure); or, as... Figure 10 As shown, the road sign direction information 1020 can be displayed in the first road image of the current navigation interface in the form of text instructions, to indicate that the first lane 1010 (the lane between lane lines 1011 and 1012) in the first road image is a straight lane. The road sign direction information 1020 is displayed close to the ground based on the lane line positions on the first road image in the current navigation interface, i.e., the positions of lane lines 1011 and 1012, and the lane line positions in the second road image previously displayed on the current navigation interface (not shown in the figure); alternatively, the road sign direction information can also be a combination of arrow instructions and text instructions. This application does not limit the form in which the road sign direction information is presented. Figure 11As shown, the lane change instruction information can be an arrow indicator 1130 displayed on the ground based on the lane lines. The image display position of the arrow indicator 1130 is related to the lane line position in the current image frame and the lane line position in the image frames before the current image frame, used to indicate that the vehicle can change lanes from the first lane 1110 (the lane between lane lines 1111 and 1112) in the first road image to the second lane 1120 (the lane between lane lines 1112 and 1113). Figure 12 As shown, the driving instruction execution distance information 1220 can be a combination of arrow indication information and text indication information displayed on the lane line. The image display position of the driving instruction execution distance information 1220 is related to the lane line position in the current image frame and the lane line position in the image frames before the current image frame, such as... Figure 12 As shown, this is used to instruct vehicles to travel 50m along the first lane 1210 (the lane between lane lines 1211 and 1212) and then turn right, to assist users in driving. It should be noted that this application does not limit the type or display method of navigation information.

[0199] In summary, the navigation information display method provided in this application displays navigation information by overlaying it close to and parallel to the ground in the first road image based on the lane line position of the first road image obtained in real time by the terminal in the navigation interface. This makes the display of navigation information closer to the road information actually observed by the user. Compared with voice prompts, it can more intuitively indicate route information and improve the display effect of navigation information.

[0200] Figure 13 This application shows a block diagram of a navigation information display device provided in an exemplary embodiment, such as... Figure 13 As shown, the device includes:

[0201] Interface display module 1310 is used to display the navigation interface;

[0202] Image display module 1320 is used to display a first road image in the navigation interface. The first road image is an image captured in real time by an image acquisition device set in the corresponding vehicle.

[0203] The navigation information display module 1330 is used to overlay and display navigation information on the top layer of the first road image; the navigation information is close to and parallel to the ground in the first road image; the display position of the navigation information is related to the position of the lane line on the first road image.

[0204] In one possible implementation, the device further includes:

[0205] The lane line recognition module is used to perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle;

[0206] The lane line prediction module is used to obtain at least one predicted lane line from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system.

[0207] The lane line update module is used to update the at least one predicted lane line based on at least one projection point string to obtain the lane line of the first road image in the ground coordinate system.

[0208] The navigation information display module 1330 is used to overlay and display the navigation information on the first road image based on the lane lines corresponding to the first road image in the ground coordinate system.

[0209] In one possible implementation, the navigation information display module includes:

[0210] The ground display location determination submodule is used to determine the ground display location of the navigation information in the ground coordinate system based on the lane lines corresponding to the first road image in the ground coordinate system.

[0211] The image display location determination submodule is used to project the ground display location onto the first road image based on the installation pose of the image acquisition device, so as to obtain the image display location of the navigation information in the first road image;

[0212] The navigation information display submodule is used to overlay and display the navigation information on the first road image based on the image display location.

[0213] In one possible implementation, the lane line update module includes:

[0214] A distance calculation submodule is used to calculate the lateral distance between at least one of the projected point strings and at least one of the predicted lane lines;

[0215] The matching relationship determination submodule is used to determine the matching relationship between the projection point string and the predicted lane line based on the lateral distance;

[0216] The lane line update submodule is used to update the at least one predicted lane line based on the matching relationship to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0217] In one possible implementation, the lane line update submodule includes:

[0218] The first lane line acquisition unit is used to update the first predicted lane line based on the first projection point string and the first predicted lane line that the matching relationship indicates a successful match, so as to obtain a lane line in the ground coordinate system corresponding to the first road image.

[0219] The second lane line acquisition unit is used to fit the second projection point string that is not matched according to the matching relationship, and to obtain the fitted lane line as a lane line in the ground coordinate system corresponding to the first road image.

[0220] The third lane line acquisition unit is used to acquire the second predicted lane line, which is not matched according to the matching relationship, as a lane line in the ground coordinate system corresponding to the first road image.

[0221] In one possible implementation, the third lane line acquisition unit is used to acquire the number of consecutive unsuccessful matches of the second predicted lane line;

[0222] In response to the number of consecutive unmatched attempts being less than a threshold for the number of unmatched attempts, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

[0223] In one possible implementation, the second lane line acquisition unit is used to fit the second projection point string based on a quadratic polynomial, and acquire the fitted lane line as a lane line in the ground coordinate system corresponding to the first road image.

[0224] In one possible implementation, the first lane line acquisition unit is used to establish lane line distance constraints based on the first predicted lane line and the first projection point string.

[0225] Based on the lane line distance constraint, the first predicted lane line is updated to obtain a lane line in the ground coordinate system corresponding to the first road image.

[0226] In one possible implementation, the lane line distance constraint is used to constrain the distance between the first predicted lane line and the first projection point string, such that the lateral distance between each point in the first projection point string and the first predicted lane line is less than a specified threshold.

[0227] In one possible implementation, the lane prediction module is used to obtain at least one predicted lane line of the first road image based on the reference lane line and the local motion data of the vehicle.

[0228] The local motion data refers to the motion data of the vehicle within a motion time period, which is the time period between a first time point and a second time point; the first time point is the time when the second road image was acquired, and the second time point is the time when the first road image was acquired; the first time point is before the second time point.

[0229] In one possible implementation, the lane line recognition module includes:

[0230] The lane line segmentation submodule is used to segment the first road image into lane lines and obtain a mask image of the first road image.

[0231] The pixel clustering submodule is used to cluster the pixels on the mask image to obtain the pixels corresponding to at least one lane line.

[0232] The projection submodule is used to project the clustered pixels onto the ground coordinate system to obtain at least one projection point string of the first road image.

[0233] In summary, the navigation information display device provided in this application displays navigation information by overlaying it close to and parallel to the ground in the first road image based on the lane line position of the first road image obtained in real time by the terminal in the navigation interface. This makes the display of navigation information closer to the road information actually observed by the user. Compared with voice prompts, it can more intuitively indicate the route information and improve the display effect of navigation information.

[0234] Figure 14 A block diagram of a lane tracking device provided in an exemplary embodiment of this application is shown, as follows: Figure 14 As shown, the device includes:

[0235] Image acquisition module 1410 is used to acquire a first road image, which is an image acquired in real time by an image acquisition device set up for the corresponding vehicle;

[0236] The lane line recognition module 1420 is used to perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle.

[0237] The lane line prediction module 1430 is used to obtain at least one predicted lane line from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system.

[0238] The lane line update module 1440 updates the at least one predicted lane line based on at least one projection point string to obtain the lane line corresponding to the first road image in the ground coordinate system.

[0239] In summary, the navigation information display device provided in this application, during lane line tracking, obtains a string of projection points corresponding to each lane line in the first road image based on the projection of lane line pixels in the first road image onto the ground coordinate system. Subsequently, after obtaining the predicted lane line based on the lane line parameters of the second road image, the predicted lane line is updated based on the string of projection points to obtain the lane line of the first road image in the ground coordinate system, thereby achieving lane line tracking in the ground coordinate system. Through the above method, the comprehensiveness and effectiveness of utilizing information identified from the image in the lane line tracking scenario are improved, thereby enabling more continuous and stable lane line tracking, and thus improving the accuracy of lane line tracking based on images.

[0240] Figure 15 A structural block diagram of a computer device 1500 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device 1500 includes a Central Processing Unit (CPU) 1501, a system memory 1504 including Random Access Memory (RAM) 1502 and Read-Only Memory (ROM) 1503, and a system bus 1505 connecting the system memory 1504 and the CPU 1501. The computer device 1500 also includes a mass storage device 1506 for storing an operating system 1509, application programs 1510, and other program modules 1511.

[0241] The mass storage device 1506 is connected to the central processing unit 1501 via a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1506 and its associated computer-readable media provide non-volatile storage for the computer device 1500. That is, the mass storage device 1506 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0242] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1504 and mass storage device 1506 described above can be collectively referred to as memory.

[0243] According to various embodiments of this disclosure, the computer device 1500 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1500 can be connected to a network 1508 via a network interface unit 1507 connected to the system bus 1505, or the network interface unit 1507 can be used to connect to other types of networks or remote computer systems (not shown).

[0244] The memory also includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 1501 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the navigation information display method and / or lane line tracking method shown in the above embodiments.

[0245] Figure 16 A structural block diagram of a computer device 1600 provided in an exemplary embodiment of this application is shown. The computer device 1600 can be implemented as the aforementioned terminal, such as an in-vehicle navigation device, mobile phone, tablet computer, laptop computer, or desktop computer. The computer device 1600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0246] Typically, computer device 1600 includes a processor 1601 and a memory 1602.

[0247] Processor 1601 may include one or more processing cores, such as a quad-core processor or a 16-core processor. Processor 1601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0248] The memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1602 is used to store at least one instruction, which is executed by the processor 1601 to implement all or part of the steps in the navigation information display method and / or lane line tracking method provided in the method embodiments of this application.

[0249] In some embodiments, the computer device 1600 may optionally include a peripheral device interface 1603 and at least one peripheral device. The processor 1601, memory 1602, and peripheral device interface 1603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607, a positioning assembly 1608, and a power supply 1609.

[0250] Peripheral interface 1603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1601 and memory 1602. In some embodiments, processor 1601, memory 1602 and peripheral interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1601, memory 1602 and peripheral interface 1603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0251] In some embodiments, the computer device 1600 further includes one or more sensors 1610. The one or more sensors 1610 include, but are not limited to: an accelerometer 1611, a gyroscope 1612, a pressure sensor 1613, a fingerprint sensor 1614, an optical sensor 1615, and a proximity sensor 1616.

[0252] Those skilled in the art will understand that Figure 16 The structure shown does not constitute a limitation on the computer device 1600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0253] In one exemplary embodiment, a computer-readable storage medium is also provided for storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement all or part of the steps in the navigation information display method and / or lane line tracking method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0254] In one exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 2 , Figure 4 or Figure 8 All or part of the steps of the method shown in any embodiment.

[0255] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0256] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for displaying navigation information, characterized in that, The method includes: Display the navigation interface; The navigation interface displays a first road image, which is an image captured in real time by the image acquisition device installed in the corresponding vehicle. Lane line recognition is performed on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle; At least one predicted lane line is obtained from the first road image based on a reference lane line; the reference lane line is the lane line in the ground coordinate system corresponding to the second road image; the second road image is the road image displayed in the navigation interface before the first road image; Based on at least one projection point string, the at least one predicted lane line is updated to obtain the lane line in the ground coordinate system corresponding to the first road image; Based on the lane lines corresponding to the first road image in the ground coordinate system, navigation information is superimposed and displayed on the first road image; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of the lane lines on the first road image.

2. The method according to claim 1, characterized in that, The step of overlaying and displaying the navigation information on top of the first road image, based on the lane lines corresponding to the first road image in the ground coordinate system, includes: Based on the lane lines in the ground coordinate system corresponding to the first road image, the ground display position of the navigation information in the ground coordinate system is determined; Based on the installation orientation of the image acquisition device, the ground display position is projected onto the first road image to obtain the image display position of the navigation information in the first road image; Based on the image display location, the navigation information is overlaid and displayed on top of the first road image.

3. The method according to claim 1, characterized in that, The step of updating the at least one predicted lane line based on at least one projection point string to obtain the lane line corresponding to the first road image in the ground coordinate system includes: Calculate the lateral distance between at least one of the projected point strings and at least one of the predicted lane lines; Based on the lateral distance, the matching relationship between the projected point string and the predicted lane line is determined; Based on the matching relationship, the at least one predicted lane line is updated to obtain the lane line in the ground coordinate system corresponding to the first road image.

4. The method according to claim 3, characterized in that, The step of updating the at least one predicted lane line based on the matching relationship to obtain the lane line corresponding to the first road image in the ground coordinate system includes: For the first projection point string and the first predicted lane line that are successfully matched, the first predicted lane line is updated based on the first projection point string to obtain a lane line in the ground coordinate system corresponding to the first road image. For the second projection point string that is not matched as indicated by the matching relationship, the second projection point string is fitted, and the fitted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image. For the second predicted lane line that is not matched as indicated by the matching relationship, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

5. The method according to claim 4, characterized in that, The step of obtaining the second predicted lane line as a lane line in the ground coordinate system corresponding to the first road image for the second predicted lane line that does not match the matching relationship includes: Obtain the number of consecutive unsuccessful matches of the second predicted lane line; In response to the number of consecutive unmatched attempts being less than a threshold for the number of unmatched attempts, the second predicted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

6. The method according to claim 4, characterized in that, The step of fitting the second projection point string that does not match according to the matching relationship, and obtaining the fitted lane line as a lane line in the ground coordinate system corresponding to the first road image, includes: The second projection point string is fitted based on a quadratic polynomial, and the fitted lane line is obtained as a lane line in the ground coordinate system corresponding to the first road image.

7. The method according to claim 4, characterized in that, For the first projection point string and the first predicted lane line that indicate a successful match, based on the first projection point string, the first predicted lane line is updated to obtain a lane line in the ground coordinate system corresponding to the first road image, including: Establish lane line distance constraints based on the first predicted lane line and the first projection point string; Based on the lane line distance constraint, the first predicted lane line is updated to obtain a lane line in the ground coordinate system corresponding to the first road image.

8. The method according to claim 7, characterized in that, The lane line distance constraint is used to constrain the distance between the first predicted lane line and the first projection point string, such that the lateral distance between each point in the first projection point string and the first predicted lane line is less than a specified threshold.

9. The method according to claim 2, characterized in that, The step of obtaining at least one predicted lane line from the first road image based on the reference lane line includes: Based on the reference lane line and the local motion data of the vehicle, at least one predicted lane line of the first road image is obtained; The local motion data refers to the motion data of the vehicle within a motion time period, which is the time period between a first time point and a second time point; the first time point is the time when the second road image was acquired, and the second time point is the time when the first road image was acquired; the first time point is before the second time point.

10. The method according to claim 2, characterized in that, The step of performing lane line recognition on the first road image to obtain at least one projection point string of the first road image includes: Lane line segmentation is performed on the first road image to obtain a mask image of the first road image; Cluster the pixels on the mask image to obtain the pixels corresponding to at least one lane line; The clustered pixels are projected onto the ground coordinate system to obtain at least one projection point string of the first road image.

11. A lane line tracking method, characterized in that, The method includes: Acquire a first road image, which is an image captured in real time by an image acquisition device installed on the corresponding vehicle; Lane line recognition is performed on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle; At least one predicted lane line is obtained from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system; the second road image is a road image acquired before the first road image; Based on at least one projection point string, the at least one predicted lane line is updated to obtain the lane line in the ground coordinate system corresponding to the first road image.

12. A navigation information display device, characterized in that, The device includes: The interface display module is used to display the navigation interface; An image display module is used to display a first road image in the navigation interface. The first road image is an image captured in real time by an image acquisition device set in the corresponding vehicle. The lane line recognition module is used to perform lane line recognition on the first road image to obtain at least one projection point string of the first road image; the projection point string is obtained by projecting the lane lines in the first road image onto a ground coordinate system; the ground coordinate system is established based on the vehicle; The lane line prediction module is used to obtain at least one predicted lane line from the first road image based on a reference lane line; the reference lane line is the lane line in the second road image corresponding to the ground coordinate system. The lane line update module is used to update the at least one predicted lane line based on at least one projection point string to obtain the lane line of the first road image in the ground coordinate system. The navigation information display module is used to overlay navigation information on the first road image based on the lane lines corresponding to the first road image in the ground coordinate system; the navigation information is close to and parallel to the ground in the first road image; the image display position of the navigation information is related to the position of the lane lines on the first road image.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the navigation information display method as claimed in any one of claims 1 to 10 or the lane line tracking method as claimed in claim 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the navigation information display method as described in any one of claims 1 to 10 or the lane line tracking method as described in claim 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions that are loaded and executed by a processor to implement the navigation information display method as described in any one of claims 1 to 10 or the lane line tracking method as described in claim 11.

Images