Live navigation method, device and equipment, storage medium and program product

Abstract

The application provides a real scene navigation method and device, equipment, computer readable storage medium and computer program product; it is applied to the map field; the method comprises the following steps: real-time display of current frame road surface image in the display interface, a plurality of lane lines are identified from the current frame road surface image, the plurality of lane lines are projected into the vehicle coordinate system, the corresponding plurality of projection lane lines are obtained, and two current following lane lines are detected from the plurality of projection lane lines, and the current pose of the vehicle is detected from the current frame road surface image; in the display interface, the ground navigation guide is displayed between the two current following lane lines based on the current pose. Through the application, the accurate fitting of navigation guide and road surface can be realized in the real scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the Internet of Vehicles technology, and in particular to a real scene navigation method and device, equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] In a navigation system, a Global Positioning System (GPS) antenna built therein receives data information transmitted from at least three of 24 GPS satellites surrounding the earth, combines with an electronic map stored in the navigation device, determines the position coordinates of the vehicle through the GPS satellite signal, determines the accurate position of the vehicle in the electronic map based on the position coordinates of the vehicle, so as to realize the GPS positioning function. On the basis of positioning, the best driving route, the road conditions ahead, and the information of the nearest gas station, restaurant, hotel, etc. can be provided through a multifunctional display.

[0003] Taking vehicle navigation as an example, the navigation guidance needs to be displayed on the road surface in the provided driving route, and the driver is intuitively guided by the guidance target. However, the navigation guidance of the related art cannot be well matched with the ground, which affects the display effect of the navigation guidance in the navigation process. SUMMARY

[0004] The embodiments of the present application provide a real scene navigation method, device, computer readable storage medium and program product, which can realize accurate matching of navigation guidance and road surface in a real scene.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide a real scene navigation method, which comprises the following steps:

[0007] real-time display of a current frame road surface image in a display interface, wherein the current frame road surface image is obtained by image acquisition of a road on which a vehicle travels;

[0008] identification of a plurality of lane lines from the current frame road surface image, and projection of the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines;

[0009] detection of two current following lane lines from the plurality of projected lane lines, wherein the two current following lane lines are the two closest projected lane lines to the vehicle;

[0010] detection of a current pose of the vehicle from the current frame road surface image;

[0011] in the display interface, display of ground-adhesion navigation guidance between the two current following lane lines based on the current pose.

[0012] An embodiment of the present application provides a real scene navigation device, comprising:

[0013] A data acquisition module is configured to display a current frame road surface image in real time in a display interface, wherein the current frame road surface image is obtained by image acquisition on a road traveled by a vehicle;

[0014] A lane projection module is configured to identify a plurality of lane lines from the current frame road surface image, project the plurality of lane lines into a vehicle coordinate system, and obtain a plurality of corresponding projected lane lines;

[0015] A lane following module is configured to detect two current following lane lines from the plurality of projected lane lines, wherein the two current following lane lines are the two closest projected lane lines to the vehicle;

[0016] A pose detection module is configured to detect a current pose of the vehicle from the current frame road surface image;

[0017] A guidance generation module is configured to display ground-hugging navigation guidance between the two current following lane lines based on the current pose in the display interface.

[0018] An embodiment of the present application provides a real scene navigation device, comprising:

[0019] A memory is configured to store executable instructions;

[0020] A processor is configured to execute the executable instructions stored in the memory, and implement the real scene navigation method provided by an embodiment of the present application.

[0021] An embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and is configured to cause a processor to execute the real scene navigation method provided by an embodiment of the present application.

[0022] An embodiment of the present application provides a computer program product, which comprises a computer program or instructions, and the computer program or instructions are executed by a processor to implement the real scene navigation method provided by an embodiment of the present application.

[0023] An embodiment of the present application has the following beneficial effects:

[0024] By identifying a plurality of lane lines from a current frame road surface image, projecting the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines, detecting two current following lane lines from the plurality of projected lane lines, and displaying ground-hugging navigation guidance between the two current following lane lines based on a current pose of a vehicle detected from the current frame road surface image, accurate fitting of navigation guidance and a road surface in a real scene can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a structural schematic diagram of a real scene navigation system architecture provided by an embodiment of the present application;

[0026] Figure 2 is a structural schematic diagram of a real scene navigation device provided by an embodiment of the present application;

[0027] Figure 3A is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0028] Figure 3B is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0029] Figure 3C is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0030] Figure 3D is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0031] Figure 3E is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0032] Figure 3F is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0033] Figure 3G is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0034] Figure 3H is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0035] Figure 3I is a flow schematic diagram of a real scene navigation method provided by an embodiment of the present application;

[0036] Figure 3J is a lane line extraction network structure diagram provided by an embodiment of the present application;

[0037] Figure 3K is a U-shaped network structure diagram provided by an embodiment of the present application;

[0038] Figure 3L is a homographic matrix projection principle diagram provided by an embodiment of the present application;

[0039] Figure 4A is a terminal setting human-computer interaction interface diagram provided by an embodiment of the present application;

[0040] Figure 4B is a navigation scheme selection human-computer interaction interface diagram provided by an embodiment of the present application;

[0041] Figure 4C is a straight-ahead paving ground-guiding human-computer interaction interface diagram provided by an embodiment of the present application;

[0042] Figure 4D is a lane-changing reminding ground-guiding human-computer interaction interface diagram provided by an embodiment of the present application;

[0043] Figure 4E is a road intersection guiding human-computer interaction interface diagram provided by an embodiment of the present application;

[0044] Figure 5 is a lane line extraction schematic diagram provided by an embodiment of the present application;

[0045] Figure 6A is a vehicle coordinate system schematic diagram provided by an embodiment of the present application

[0046] Figure 6B is a vehicle coordinate system origin selection schematic diagram provided by an embodiment of the present application;

[0047] Figure 6C is a vehicle-mounted camera calibration schematic diagram provided by an embodiment of the present application;

[0048] Figure 6D is a vehicle-mounted camera internal parameter in an image coordinate system schematic diagram provided by an embodiment of the present application;

[0049] Figure 6E is a lane line projection schematic diagram provided by an embodiment of the present application;

[0050] Figure 7A is a current following lane line detection schematic diagram provided by an embodiment of the present application;

[0051] Figure 7B is a ground-guiding navigation indication display schematic diagram provided by an embodiment of the present application;

[0052] Figure 8 is a vehicle pose detection schematic diagram provided by an embodiment of the present application;

[0053] Figure 9 is a ground-guiding navigation indication life cycle schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0055] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets as each other and can be combined with each other, without conflict.

[0056] In the following description, the terms "first\second\third" are merely to distinguish similar objects, and do not represent a specific order of the objects. It is understood that the "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0058] Before the embodiments of the application are further described in detail, the terms and phrases involved in the embodiments of the application are explained, and the terms and phrases involved in the embodiments of the application are applicable to the following explanations.

[0059] 1) Navigation equipment, navigation equipment refers to navigation and guidance devices for navigation, astronomy, hydrology, land transportation, etc., and terminal products thereof include navigation, measurement, timekeeping, direction finding, etc.

[0060] 2) Positioning system, a positioning system is a collection or device (component) that aims to determine the spatial position, in addition to GPS, there are Beidou satellite navigation system, etc.

[0061] 3) Vehicle-mounted camera, which generally records the current road image at a frame rate of 30 frames per second.

[0062] 4) Deep neural network (DNN), a branch of machine learning, is an algorithm for data compression of high correlation, which can automatically and efficiently parameterize data features and avoid manual design of data features.

[0063] 5) Homography, homography is a reversible transformation from a real projective plane to a projective plane, and the embodiments of the application can realize the conversion of the camera coordinate system to the vehicle coordinate system based on homography. Straight lines are still mapped as straight lines under the transformation, and homography is used to describe the position mapping relationship between the object in the world coordinate system and the pixel coordinate system. In the field of computer vision, any two images of the same plane in space are related through homography.

[0064] 6) Augmented Reality (AR, Augmented Reality) navigation, in the real road picture shot, real-time intuitive navigation guide technology. Can greatly reduce the driver's reading cost of traditional 2D electronic map, assist users in turning, fork, lane change and other key scenes, faster and more accurate action decision.

[0065] In the implementation process of the embodiments of the present application, the applicant finds that the related art has the following problems: in the related art, the navigation device usually adds navigation guidance in the driving route to guide the driver intuitively in the direction of the navigation guidance, but the navigation guidance is usually drawn based on a two-dimensional image or a bird's eye view, and the related art lacks a scheme for constructing navigation guidance based on the real world.

[0066] The embodiments of the present application provide a real scene navigation method, device, equipment, computer readable storage medium and computer program product, which can construct ground navigation guidance based on the real world. The following describes an exemplary application of the real scene navigation device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), a vehicle-mounted terminal and various types of user terminal devices, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal device.

[0067] Referring to Figure 1 , Figure 1 is an architecture schematic diagram of the real scene navigation system 100 provided by the embodiments of the present application, to realize supporting one real scene navigation application, the terminal device 400 is connected to the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the two.

[0068] The terminal device 400 is configured to provide a real scene navigation function, and display real scene navigation information on the graphical interface 411.

[0069] In some embodiments, the real scene navigation method provided by the embodiments of the present application can be implemented by the terminal device 400. The client running in the terminal device 400 can be implanted with a real scene navigation plug-in to implement the real scene navigation method locally in the client. For example, the terminal device 400 calls the real scene navigation plug-in, displays a current frame road surface image in real time in a display interface, identifies a plurality of lane lines from the current frame road surface image, projects the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines, detects two current following lane lines from the plurality of projected lane lines, detects a current pose of the vehicle from the current frame road surface image, displays a ground-hugging navigation guide between the two current following lane lines based on the current pose, and displays real scene navigation information in the graphical interface 411 to assist the user to make action decisions more quickly and accurately in various navigation scenarios such as turning, fork, and lane changing.

[0070] In some embodiments, the real scene navigation method provided by the embodiments of the present application can be implemented by the terminal device and the server. For example, the user inputs a starting point and a destination of navigation in the terminal device 400, and the terminal device 400 sends the starting point and the destination input by the user to the server 200. The server 200 formulates a navigation route according to the received starting point and destination, identifies a plurality of lane lines from a current frame road surface image, projects the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines, detects two current following lane lines from the plurality of projected lane lines, detects a current pose of the vehicle from the current frame road surface image, determines a ground-hugging navigation guide between the two current following lane lines based on the current pose, and sends the navigation information (including the current following lane lines, the navigation guide, etc.) to the terminal device 400 to display real scene navigation information in the graphical interface 411 to assist the user to make action decisions more quickly and accurately in various navigation scenarios such as turning, fork, and lane changing.

[0071] In some embodiments, the server 200 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle navigation device, etc., but is not limited thereto. The terminal device 400 and the server 200 can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0072] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the terminal device 400 provided by the embodiments of the present application with a real scene navigation function,Figure 2 The terminal device 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the terminal device 400 are coupled together by a bus system 440, which is configured to facilitate the communication of information between each of the components. The bus system 440 can be implemented using a series of buses and / or a network of one or more interconnected or embedded buses. Figure 2 The various buses are collectively referred to herein as the bus system 440 for clarity, but can include data buses, power buses, control buses, and / or status buses, among others.

[0073] The processor 410 can be an integrated circuit chip with processing capability that can be a general purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The processor 410 can be a microprocessor, or any conventional processor, or the like.

[0074] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls, and the like.

[0075] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 450 optionally includes one or more storage devices remotely located from the processor 410 in physical location.

[0076] The memory 450 includes volatile memory or non-volatile memory, or both. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 450 described herein is intended to include any suitable type of memory.

[0077] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or superset thereof, which are described below.

[0078] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0079] a network communication module 452 for communicating to other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), etc.;

[0080] a presentation module 453 for enabling presentation of information via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, etc.) (e.g., a user interface for operating the peripheral device and displaying content and information);

[0081] an input processing module 454 for detecting and interpreting one or more user inputs or interactions from one or more input devices 432.

[0082] In some embodiments, the live navigation device provided by the embodiments of the present application can be implemented in software, Figure 2 A live navigation device 455 stored in the memory 450 is shown, which can be in the form of software such as programs and plug-ins, including the following software modules: a data acquisition module 4551, a lane projection module 4552, a lane following module 4553, a pose detection module 4554, and a guidance generation module 4555. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0083] In some other embodiments, the device provided by the embodiments of the present application can be implemented in hardware, as an example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the live navigation method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0084] In some embodiments, the terminal device or the server can implement the permission issuing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or a software module in an operating system; can be a native application program (APP), i.e., a program that needs to be installed in an operating system to run, such as a navigation APP; can also be a mini-program, i.e., a program that only needs to be downloaded into a browser environment to run; and can also be a mini-program that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in.

[0085] As described above, the real scene navigation method provided in the embodiments of the present application will be described in combination with the exemplary application and implementation of the real scene navigation device provided in the embodiments of the present application. Referring to Figure 3A , Figure 3A is a flowchart of the real scene navigation method provided in the embodiments of the present application, which will be described in combination with the steps shown in Figure 3A .

[0086] In step 101, a current frame road surface image is displayed in real time in a display interface, wherein the current frame road surface image is obtained by image acquisition on a road traveled by a vehicle.

[0087] For example, the current frame road surface image can be captured by a vehicle-mounted camera or a mobile phone, and the current frame road surface image includes road information traveled by the vehicle. Unlike traditional two-dimensional plane-based map navigation, the embodiments of the present application perform navigation in the actually captured current frame road surface image.

[0088] As an example, the vehicle-mounted camera captures road information traveled by the vehicle in real time. In general, the vehicle-mounted camera captures at a frame rate of 30 frames per second, and displays the current frame road surface image through a man-machine interaction device (for example, a terminal device) in the vehicle.

[0089] It should be noted that the display interface in the embodiments of the present application can be an interface with only a display function, or an interface with a man-machine interaction function (i.e., a man-machine interaction interface).

[0090] The above-mentioned manner displays the current frame road surface image in real time in the display interface, which provides an image basis for real scene navigation.

[0091] In some embodiments, before the projecting the plurality of lane lines into the vehicle coordinate system in step 102 to obtain a corresponding plurality of projected lane lines, the following step can also be performed: taking the projection point of the intersection of the diagonals of the four end points of the vehicle on the chassis as the origin, taking the positive direction or negative direction of the driving direction of the vehicle as the horizontal axis, taking the positive direction or negative direction of the top of the vehicle as the vertical axis, and taking the positive direction or negative direction of the left side of the vehicle as the longitudinal axis to construct the vehicle coordinate system.

[0092] For example, the origin of the vehicle coordinate system is taken as the projection point of the intersection of the diagonals of the four end points of the vehicle on the chassis, the four end points can be the four corners of the wheels or the vehicle shell, and the positive direction or negative direction of the driving direction of the vehicle is taken as the horizontal axis, the positive direction or negative direction of the top of the vehicle is taken as the vertical axis, and the positive direction or negative direction of the left side of the vehicle is taken as the longitudinal axis to construct the vehicle coordinate system.

[0093] As an example, see Figure 6B , Figure 6B The vehicle coordinate system origin selection schematic diagram provided by the embodiments of the present application. In order to make the constructed coordinate system better "stick to the ground", the origin of the vehicle coordinate system is defined as the projection point 601 of the intersection of the diagonals of the four end points of the vehicle on the chassis, so as to realize "sticking to the ground".

[0094] In step 102, a plurality of lane lines are identified from the current frame road surface image, and the plurality of lane lines are projected into the vehicle coordinate system to obtain a corresponding plurality of projected lane lines.

[0095] For example, at least two lane lines are identified from the current frame road surface image, and the at least two lane lines are projected into the coordinate system constructed with the vehicle as the origin, so as to obtain the position of the identified lane lines in the vehicle coordinate system.

[0096] As an example, see Figure 6E , Figure 6E The lane line projection principle diagram provided by the embodiments of the present application. Four lane lines are identified from the current frame road surface image, which are respectively a first lane line 611, a second lane line 612, a third lane line 613, and a fourth lane line 614, and the first lane line 611, the second lane line 612, the third lane line 613, and the fourth lane line 614 are projected into the coordinate system constructed with the vehicle as the origin to obtain a corresponding first projected lane line 621, a second projected lane line 622, a third projected lane line 623, and a fourth projected lane line 624.

[0097] The above-mentioned manner, by identifying the current frame road surface image, at least two lane lines are identified, and the at least two lane lines are projected into the coordinate system constructed with the vehicle as the origin, so as to obtain the position of the identified lane lines in the vehicle coordinate system, which provides a basis for the generation of the "sticking to the ground" navigation guide in the real scene navigation.

[0098] In some embodiments, referring to Figure 3B , the identification of the plurality of lane lines from the current frame road surface image in step 102 can be implemented by steps 10211-10215, which will be described in connection with the steps.

[0099] In step 10211, the current frame road surface image is down-sampled to obtain a compressed size of the down-sampled feature map.

[0100] For example, the current frame road surface image is down-sampled to compress the size of the image and obtain a feature map of the compressed image size.

[0101] As an example, referring to Figure 3J , Figure 3J a lane line extraction network structure diagram provided by the embodiments of the present application. First, the image features of the current frame road surface image are extracted to obtain a feature map of the current frame road surface image, and then the feature map is down-sampled by a down-sampling layer, the down-sampling layer includes a convolution layer and a pooling layer, for example, the feature map is input into a first convolution layer 3001 for convolution to obtain a convolution feature map, the convolution feature map is input into a first pooling layer 3011 for pooling (pooling) to compress the size of the feature map, and then multiple convolution and pooling operations are performed, for example, the second convolution layer 3002 is sequentially input for convolution, the second pooling layer 3012 is input for pooling, the third convolution layer 3003 is input for convolution, and the third pooling layer 3013 is input for pooling to obtain a compressed feature map output after the last pooling.

[0102] The above method performs multiple convolution and pooling operations on the feature map of the current frame road surface image, compresses the image size, and provides a basis for the up-sampling of the feature map.

[0103] In step 10212, the down-sampled feature map is up-sampled to obtain a restored size of the up-sampled feature map.

[0104] For example, the compressed feature map is up-sampled to restore the size of the image and obtain a feature map of the restored compressed image size.

[0105] As an example, referring to Figure 3JThe compressed feature map is up-sampled, the compressed feature map is input into an up-sampling layer for up-sampling, the up-sampling layer includes a pooling layer and a convolution layer, for example, the compressed feature map is input into the fourth pooling layer 3014 for inverse pooling (Unpooling), and the inverse pooled feature map is input into the fourth convolution layer 3004 for convolution to extract image features, and then the inverse pooling and convolution operations are performed multiple times, for example, the fifth pooling layer 3015 is sequentially performed for pooling, the fifth convolution layer 3005 is performed for convolution, the sixth pooling layer 3016 is performed for pooling, and the sixth convolution layer 3006 is performed for convolution to obtain the feature map of the original size output by the last convolution layer.

[0106] The above method restores the image size by performing multiple pooling and convolution operations on the compressed feature map, and provides a basis for feature classification.

[0107] In step 10213, classification processing is performed based on the up-sampled feature map, and lane line category classification results, lane line color classification results, lane line solid and dashed line classification results, and corresponding confidence levels are obtained respectively.

[0108] Referring to Figure 3J The feature map needs to be classified and processed to extract the features of the lane line, and the feature map is input into the classification layer 3100 for classification, for example, the lane line category, lane line color and lane line solid and dashed line are classified to obtain the lane line category classification result, lane line color classification result and lane line solid and dashed line classification result.

[0109] As an example, the lane line category classification result includes lane line and stop line, the lane line color classification result includes white and yellow, and the lane line solid and dashed line classification result includes solid line and dashed line. For each classification result, the confidence level of the corresponding classification is obtained.

[0110] The above method obtains multiple groups of features of the lane line from the feature map, which provides a basis for the combination of subsequent features.

[0111] In step 10214, the lane line category classification result exceeding the lane line category confidence threshold, the lane line color classification result exceeding the lane line color confidence threshold, and the lane line solid and dashed line classification result exceeding the lane line solid and dashed line confidence threshold are taken as the lane line recognition result.

[0112] For example, after obtaining the classification results of the lane line category, lane line color and lane line solid and dashed line, the confidence levels corresponding to the above results need to be retained, and the results are classified based on the confidence levels to obtain the lane line recognition result.

[0113] For example, if the confidence level of a lane line exceeds the lane line type confidence threshold for lane lines and stop lines, then the lane line is considered one of the lane line recognition results. If the confidence level of white exceeds the lane line color confidence threshold for lane lines and yellow, then white is considered one of the lane line recognition results. If the confidence level of a solid line exceeds the lane line real / dash confidence threshold for lane lines, then the solid line is considered one of the lane line recognition results. The final lane line recognition result is: a solid white lane line. It should be noted that a stop line typically refers to the solid white line on the side of the zebra crossing at an intersection, used to prompt vehicles to stop. Therefore, there is only one type of solid white stop line.

[0114] In the above-mentioned manner, the lane line recognition results obtained provide a basis for lane line projection.

[0115] In step 10215, a plurality of lane lines included in the current frame road surface image is determined based on the lane line recognition result.

[0116] See also Figure 3K , Figure 3K The U-net structure diagram provided in the embodiment of the present application shows that the lane line extraction operation can also use a U-net to identify multiple lane lines from the current frame road image. The U-net consists of two parts: a contracting path and an expanding path. The search path is mainly used to capture the context information in the current frame road image, while the symmetrical expanding path is used to accurately locate the part that needs to be segmented in the current frame road image. The contracting path is a convolutional network architecture. Each repeated structure (downsampling layer) has two convolution layers and one pooling layer. The convolution kernel size in the convolution layer is 33, and the activation function uses ReLU. The two convolution layers are followed by a maximum pooling layer with a stride of 2. After each downsampling, the number of feature channels is doubled, for example, from 64 channels to 128 channels, from 128 channels to 256 channels, from 256 channels to 512 channels, and from 512 channels to 1024 channels. Each repetitive structure (upsampling layer) in the expansion path first uses deconvolution. Each deconvolution halves the number of feature channels and doubles the size of the feature map. After deconvolution, the deconvolution result is concatenated with the feature map from the corresponding step in the contraction path. The larger feature map in the contraction path is pruned and then concatenated. Two convolutions with a factor of 33 are performed on the concatenated feature map. The final layer uses a kernel size of 11, converting the 64-channel feature map to a specific depth (the number of categories, 2 for binary classification). The network has a total of 23 layers.

[0117] In some embodiments, see Figure 3CThe projecting, in step 102, the plurality of lane lines into the vehicle coordinate system to obtain a plurality of corresponding projected lane lines can be implemented through steps 10221-10222, which will be described in combination with the steps.

[0118] In step 10221, the following processing is performed for each lane line: based on the homography matrix between the current frame road surface image and the vehicle coordinate system, the lane line is projected into the vehicle coordinate system to obtain a plurality of corresponding mapping points.

[0119] For example, the current frame road surface image is specifically the camera coordinate system represented by the current frame road surface image, that is, the coordinate system with the camera as the reference, and the homography matrix is the mapping relationship between the points of the current road surface image and the points of the vehicle coordinate system. Using the homography matrix can find the exact position of the corresponding point of any point of the current road surface image on the vehicle coordinate system.

[0120] As an example, see Figure 6E The mapping points of the lane lines in the vehicle coordinate system, that is, the positions of the lane lines in the vehicle coordinate system, can be obtained through the homography matrix. For the current road surface image, the positions of the first lane line 611, the second lane line 612, the third lane line 613, and the fourth lane line 614 are obtained, respectively.

[0121] The above-mentioned manner obtains the positions of the detected lane lines in the vehicle coordinate system through the homography matrix, which provides a basis for subsequent fitting of the positions using a quadratic polynomial.

[0122] In step 10222, a quadratic polynomial curve fitted with the plurality of mapping points is constructed, and the quadratic polynomial curve is taken as the projected lane line corresponding to the lane line.

[0123] For example, the mapping points represent the positions of the lane lines, and for the vehicle coordinate system, the mapping points are points below the origin in the vehicle coordinate system. Considering that the road surface is not necessarily smooth, the mapping points are approximately located on the same plane. According to this characteristic, the mapping points can be fitted through a quadratic polynomial to obtain the projected lane line corresponding to the lane line.

[0124] As an example, see Figure 6E The projected lane lines obtained after fitting are: the first projected lane line 621, the second projected lane line 622, the third projected lane line 623, and the fourth projected lane line 624. Since there is a deviation in the fitting process, the projected lane line obtained after fitting using the quadratic polynomial is not completely fitted with the detected lane line.

[0125] In some embodiments, see Figure 3D , Figure 3DThe illustrated step 10221 can be implemented by steps 102211-102212, which will be described in conjunction with the steps.

[0126] In step 102211, camera calibration processing is performed on the camera in the vehicle for image acquisition, to obtain the position of the camera in the vehicle coordinate system, wherein the position includes the coordinates of the camera in the vehicle coordinate system and the attitude angle of the camera relative to the vehicle coordinate system.

[0127] In step 102212, the intrinsic parameters of the camera are obtained, and based on the position of the camera in the vehicle coordinate system and the intrinsic parameters of the camera, a homography matrix from the current road surface image to the vehicle coordinate system is constructed, wherein different road surface images taken by the same camera share the homography matrix, and different positions of the camera correspond to different homography matrices.

[0128] For example, in order to obtain the homography matrix of the current road surface image in the vehicle coordinate system, the coordinates of the camera in the vehicle coordinate system, the attitude angle and the intrinsic parameters of the camera are needed, and based on the coordinates of the camera in the vehicle coordinate system, the attitude angle and the intrinsic parameters of the camera, the homography matrix of the current road surface image in the vehicle coordinate system is constructed.

[0129] As an example, see Figure 3L , Figure 3L The homography matrix projection principle provided by the embodiment of the application. Through the homography matrix H, the mapping relationship of the pixel points in the current road surface image to the vehicle coordinate system can be obtained, for example, the first pixel point 3201 in the current road surface image is mapped to the fourth pixel point 3211 in the vehicle coordinate system, the second pixel point 3202 in the current road surface image is mapped to the fifth pixel point 3212 in the vehicle coordinate system, and the third pixel point 3203 in the current road surface image is mapped to the sixth pixel point 3213 in the vehicle coordinate system.

[0130] The above-mentioned manner constructs the homography matrix of the current road surface image to the vehicle coordinate system, and provides a projection basis for the projection of the lane line.

[0131] In some embodiments, the construction of the homography matrix can be achieved by Figure 3E The illustrated steps 1022121-1022124 are implemented, which will be described in conjunction with the steps.

[0132] In step 1022121, the coordinates of the camera in the vehicle coordinate system are determined as the translation relationship of the camera to the origin of the vehicle coordinate system, and the attitude angle of the camera in the vehicle coordinate system is determined as the rotation relationship of the camera to the origin of the vehicle coordinate system.

[0133] As an example, see Figure 6C , Figure 6CA camera attitude angle schematic diagram is provided for the embodiment of the present application. Coordinate values of the vehicle-mounted camera in the vehicle coordinate system and attitude angles compared with the vehicle coordinate system, i.e., roll angle (Roll), pitch angle (Yaw), and yaw angle (Pitch), are obtained, wherein the roll angle represents the rotation value of the vehicle-mounted camera on the vertical axis, the pitch angle represents the rotation value of the vehicle-mounted camera on the longitudinal axis, and the yaw angle represents the rotation value of the vehicle-mounted camera on the lateral axis. The lateral axis, longitudinal axis, and vertical axis values of the camera to the origin of the vehicle coordinate system are calculated as a translation relationship. The roll angle, pitch angle, and yaw angle of the camera compared with the vehicle coordinate system are calculated as a rotation relationship.

[0134] In step 1022122, the sum of the translation relationship and the rotation relationship of the physical transformation quantity is determined as the physical transformation quantity of the camera in the vehicle coordinate system.

[0135] As an example, the translation relationship is generally represented as t, the rotation relationship is represented as R, and the physical transformation quantity is represented as W=[Rt], wherein W represents the sum of the translation relationship and the rotation relationship.

[0136] In step 1022123, an intrinsic parameter matrix is constructed based on the intrinsic parameters of the camera.

[0137] As an example, refer to Figure 6D , Figure 6D A schematic diagram of the intrinsic parameters of the vehicle-mounted camera in the image coordinate system is provided for the embodiment of the present application. The intrinsic parameters f x , f y , c x , and c y of the vehicle-mounted camera are obtained, wherein f x , f y represent the focal lengths of the images of the vehicle-mounted camera in the x and y directions of the image coordinate system, and c x , c y represent the offsets of the optical axes of the vehicle-mounted camera in the x and y directions of the image coordinate system, wherein the positive direction of f x , c x is horizontally to the right, and the positive direction of f y , c y is vertically downward. The intrinsic parameter matrix is

[0138] In step 1022124, the product of the intrinsic parameter matrix, the physical transformation quantity, and the scale ratio is taken as the homography matrix, wherein the scale ratio represents the conversion ratio of the current frame road surface image to the vehicle coordinate system.

[0139] As an example, the scale ratio is generally defined as s, the scale ratio represents the conversion ratio of the current frame road surface image to the vehicle coordinate system, and the finally constructed homography matrix is represented as H=sMW.

[0140] In step 103, two current following lane lines are detected from the plurality of projected lane lines, wherein the two current following lane lines are the two projected lane lines closest to the vehicle.

[0141] For example, the two projected lane lines closest to the vehicle are detected from the plurality of projected lane lines as the two current following lane lines, so as to determine which two lane lines the vehicle is located between.

[0142] For example, referring to Figure 7A , Figure 7A is a current following lane line detection principle provided by the embodiment of the present application, and the two projected lane lines closest to the vehicle are detected from the plurality of projected lane lines as the seventh lane line 711 and the eighth lane line 712.

[0143] The above-mentioned manner takes the two projected lane lines closest to the vehicle as the current following lane lines, thereby providing a position basis for generation of the ground-hugging navigation guide.

[0144] In some embodiments, step 103 can be implemented by the following steps: for the two historical following lane lines in the previous frame of the road surface image, the following processing is respectively performed: matching the plurality of projected lane lines with the historical following lane lines respectively, and taking the projected lane line closest to the historical following lane line and having a distance less than a distance threshold from the historical following lane line as a current following lane line.

[0145] For example, the historical following lane lines in the previous frame of the road surface image and the following lane lines in the current frame of the road surface image can not match, and smoothing processing is required, so the plurality of projected lane lines are matched with the historical following lane lines respectively, and the projected lane line closest to the historical following lane line and having a distance less than a distance threshold is taken as a current following lane line.

[0146] For example, referring to Figure 7A , the two historical following lane lines of the previous frame of the road surface image are the fifth lane line 701 and the sixth lane line 702, and the projected lane line closest to the historical following lane line and having a distance less than a distance threshold is taken as a current following lane line according to the nearest matching principle. For example, the projected lane lines in the current frame are the seventh lane line 711 and the eighth lane line 712, and the seventh lane line 711 and the eighth lane line 712 are determined as the following lane lines in the current frame according to the nearest matching principle, such as the fifth lane line 701 being closest to the seventh lane line 711 and the sixth lane line 702 being closest to the eighth lane line 712.

[0147] The above method realizes the smoothing processing of the following lane lines in the road surface images of the last frame and the current frame by the nearest matching principle, and the projection lane line obtained by matching and closest to the historical following lane line and having a distance less than the distance threshold is taken as a current following lane line.

[0148] It should be noted that when the two current following lane lines are distributed on both sides of the vehicle, the ground navigation guide is displayed between the two current following lane lines based on the current pose; when the two current following lane lines are distributed on one side of the vehicle, the starting generation position of the ground navigation guide is determined through the current pose, the starting generation position is moved to the current following lane line farthest from the vehicle, to obtain a target position, wherein the target position is between the two current following lane lines and on the plane where the two current following lane lines are located, and the ground navigation guide is displayed between the two current following lane lines starting from the target position.

[0149] As shown in Figure 7B , the two current following lane lines are the seventh lane line 711 and the eighth lane line 712, and the seventh lane line 711 and the eighth lane line 712 are both on the left side of the vehicle, that is, the vehicle travels on the eighth lane line 712, the starting generation position 713 of the ground navigation guide is determined through the current pose, the starting generation position 713 is moved to the seventh lane line 711 farthest from the vehicle, to obtain a target position 714, and the ground navigation guide is displayed between the seventh lane line 711 and the eighth lane line 712 starting from the target position 714, so as to indicate the user to adjust the state through the ground navigation guide and avoid the dangerous driving behavior of line pressing.

[0150] In step 104, the current pose of the vehicle is detected from the current frame road surface image.

[0151] For example, the ground navigation guide needs to be drawn in front of the vehicle, so the current pose of the vehicle also needs to be detected.

[0152] In some embodiments, referring to Figure 3F , Figure 3F Step 104 shown in the figure can be implemented by steps 1041-1043, which will be described in combination with the steps.

[0153] In step 1041, when the current frame road surface image is collected, the inertial sensor synchronous output in the vehicle is acquired, and the first estimated pose of the vehicle is determined based on the inertial sensor data.

[0154] In step 1042, the image features are extracted from the current frame road surface image, and the second estimated pose of the vehicle is determined based on the image features.

[0155] In step 1043, the first estimated pose is corrected based on the second estimated pose to obtain the current pose of the vehicle.

[0156] For example, compared with the inertial sensor, the data of the vehicle-mounted camera hardly produces drift, and if the vehicle-mounted camera is fixed, the second estimated pose is also fixed. Therefore, the data of the vehicle-mounted camera can effectively correct the drift value generated in the inertial sensor, and the disadvantage of the vehicle-mounted camera is that motion blur will occur when the motion is too fast, that is, the first estimated pose will produce a drift value, which is just compensated by the inertial sensor. For short-time fast motion, the inertial sensor can provide an estimate.

[0157] In some embodiments, step 1041 can be implemented by the following steps: the inertial sensor data includes acceleration and angular velocity of the vehicle, wherein the acceleration and angular velocity include a drift value; obtaining a time difference between the current time and the last time of the vehicle, determining a displacement value of the vehicle relative to the time difference and the acceleration, and integrating the angular velocity with respect to the time difference as an angular displacement, wherein the displacement value and the angular displacement are relative to the vehicle at the last time; and determining the displacement value and the angular displacement as the first estimated pose of the vehicle.

[0158] As an example, the six-axis gyroscope in the inertial sensor can sense the acceleration and angular velocity of the vehicle. By using the relationship formula between acceleration and displacement, the displacement value relative to the vehicle at the last time is calculated by using the acceleration, the time difference and the initial speed at the last time. For angular displacement, the angular displacement relative to the vehicle at the last time can be obtained by integrating the angular velocity with respect to the time difference, and the displacement value and the angular displacement are determined as the first estimated pose of the vehicle. Wherein, the displacement value and the angular displacement are parameters representing the first estimated pose.

[0159] In some embodiments, referring to Figure 3G , Figure 3G The step 1043 shown can be implemented by steps 10431-10433, which will be described in conjunction with the steps.

[0160] In step 10431, the proportion of the second pose estimate value in the current pose is obtained.

[0161] As an example, when the vehicle has not started moving, determining the second estimated posture of the vehicle through image features will hardly produce a drift value. At the initial moment, the proportion of the second estimated posture can be defined as 100%. At this time, the ratio of the first estimated posture to the second estimated posture is obtained by the inertial sensor, and the ratio is used as the proportion of the first estimated posture; at the next moment, the first estimated posture at the next moment is calculated through the proportion of the first estimated posture, and the ratio of the first estimated posture at the next moment to the second estimated posture at the next moment is used as the proportion of the second estimated posture at the next moment. Then, the proportion of the second estimated posture at the initial moment and the proportion of the second estimated posture at the next moment are averaged to obtain the proportion of the current posture. This iteration is performed to obtain the proportion of the second estimated posture at each moment in the current posture.

[0162] In step 10432, the product of the specific weight and the second estimated posture is determined as the correction amount.

[0163] In step 10433, the first estimated posture and the correction amount are added together to determine the current posture of the vehicle.

[0164] For example, when the first estimated posture includes a displacement value and an angular displacement, the proportion of the displacement value in the second posture estimation value in the current posture is obtained, and the product of the proportion and the displacement value in the second estimated posture is determined as the displacement correction amount, and the sum of the displacement value and the displacement correction amount in the first estimated posture is determined as the displacement value of the current posture of the vehicle; the proportion of the angular displacement in the second posture estimation value in the current posture is obtained, and the product of the proportion and the angular displacement in the second estimated posture is determined as the angular displacement correction amount, and the sum of the angular displacement in the first estimated posture and the angular displacement correction amount is determined as the angular displacement of the current posture of the vehicle.

[0165] The above method obtains the current posture of the vehicle by correcting the second posture estimate, which provides a basis for the subsequent generation of ground-based navigation guidance.

[0166] In step 105 , a ground-hugging navigation guide is displayed between two currently following lane lines based on the current posture in the display interface.

[0167] In some embodiments, see Figure 3H , Figure 3H The illustrated step 105 can be implemented through steps 10511 - 10512 , which will be described in conjunction with the steps.

[0168] In step 10511, the starting generation position of the ground-based navigation guidance is determined based on the current posture, wherein the starting generation position is located between the two currently following lane lines and on the plane where the two currently following lane lines are located.

[0169] In step 10512, a ground-hugging navigation guide is displayed between the two current following lane lines starting from the starting generation position; wherein the types of the ground-hugging navigation guide include: straight-ahead paving guide, lane-changing guide, intersection guide.

[0170] As an example, the starting generation position of the ground-hugging navigation guide can be determined by the current pose, for example, the current pose is added to the set pose to obtain the starting generation position of the ground-hugging navigation guide, that is, it is known that the ground-hugging navigation guide should be generated at a distance far in front of the vehicle, the set pose can be set in advance according to the needs, and the generated ground-hugging navigation guide is located between the two current following lane lines and on the plane where the two current following lane lines are located, so that the ground-hugging navigation guide realizes the true sense of ground-hugging.

[0171] In some embodiments, step 105 can also be implemented by the following steps: outputting a voice navigation guide, in response to the vehicle not driving according to the voice navigation guide, displaying a ground-hugging navigation guide between the two current following lane lines based on the current pose; wherein the types of the voice navigation guide include at least one of the following: straight-ahead voice guide, lane-changing voice guide, intersection voice guide.

[0172] For example, in order to increase the immersion of the ground-hugging navigation guide, it is necessary to display the ground-hugging navigation guide at a certain distance in front of the vehicle and between the two current following lane lines; or synchronize with the voice navigation guide, display the ground-hugging navigation guide consistent with the voice navigation at a certain distance in front of the vehicle and between the two current following lane lines. The types of the voice navigation guide include at least one of the following: straight-ahead voice guide, lane-changing voice guide, intersection voice guide.

[0173] As an example, when the voice navigation guide prompts the vehicle to currently need to go straight, a straight-ahead paving guide is displayed in front of the vehicle and between the two current following lane lines, when the voice navigation guide prompts the vehicle to currently need to change lanes, a lane-changing guide is displayed in front of the vehicle and between the two current following lane lines, and when the voice navigation guide prompts the vehicle to currently need to turn at the intersection, an intersection guide is displayed in front of the vehicle and between the two current following lane lines.

[0174] The above-mentioned manner improves the immersion of the ground-hugging navigation guide by integrating the ground-hugging navigation guide with the voice navigation guide.

[0175] For example, the starting generation position of the ground-hugging navigation guide is determined by the current pose, the ground-hugging navigation guide is displayed between the two current following lane lines starting from the starting generation position, and the corresponding type of ground-hugging navigation guide is generated according to the navigation information, if the current navigation information requires straight-ahead, a straight-ahead paving guide is generated; if the current navigation information requires lane-changing, a lane-changing guide is generated; and if the current navigation information requires intersection turning, an intersection guide is generated.

[0176] In some embodiments, referring to Figure 3I , Figure 3I The steps 201-202 shown can be executed after the step 105, which will be described in combination with the steps.

[0177] In the step 201, a display position detection process is performed on each generated ground-hugging guide.

[0178] In the step 202, a destruction process is performed on the ground-hugging guide when the display position detection result indicates that the display position of the ground-hugging guide between the two current following lane lines is behind the vehicle.

[0179] For example, in order to save memory space, the ground-hugging guide whose position is behind the vehicle needs to be destroyed, that is, a position detection process is performed on each generated ground-hugging guide, and when the position is behind the vehicle, the ground-hugging guide is destroyed.

[0180] In the above manner, the redundant ground-hugging guide is destroyed, the related resources are released, and the memory is saved.

[0181] Next, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0182] Before the vehicle travels, the user can use a navigation device (for example, a real scene navigation device) to set a starting point and a destination, query a navigation route through a navigation function, and render the navigation route in the form of a ground-hugging guide in a real shot image presented by a display interface of the navigation device, so that the user can more intuitively know the navigation route. The real shot image presented by the display interface of the navigation device is obtained by a vehicle-mounted camera and sent to the navigation device by the vehicle-mounted camera.

[0183] Referring to Figure 4A , Figure 4A A terminal setting human-computer interaction interface diagram provided by the embodiments of the present application is shown. The navigation device displays an input box 401 of a destination in the human-computer interaction interface, for prompting the user to set the destination, and the setting of the starting point can be automatically obtained by a positioning system of the navigation device on the position of the current vehicle. In response to a trigger operation for setting the starting point and setting the destination, the navigation device obtains navigation information from a server.

[0184] Referring to Figure 4B , Figure 4B A navigation scheme selection human-computer interaction interface diagram provided by the embodiments of the present application is shown. After the navigation device obtains the navigation information from the server, the recommended route 402 is displayed in the human-computer interaction interface, and selection buttons of alternative schemes and navigation modes, such as an AR navigation button 403, are provided. In response to a trigger operation of the AR navigation button 403, a current road surface image fused with a ground-hugging guide is displayed on the human-computer interaction interface.

[0185] In some embodiments, referring to Figure 4C , Figure 4C The straight-ahead paving ground-adhesion guidance human-computer interaction interface diagram provided for the embodiments of the present application is shown in FIG. 4. According to the navigation route obtained by the navigation device from the server, the navigation route is straight-ahead driving at this time, and the straight-ahead paving ground-adhesion guidance 404 is displayed on the human-computer interaction interface of the navigation device.

[0186] In some embodiments, referring to Figure 4D , Figure 4D The lane-changing reminding ground-adhesion guidance human-computer interaction interface diagram provided for the embodiments of the present application is shown in FIG. 5. The navigation device obtains the navigation route from the server, and the navigation route is to remind the vehicle to change lanes at this time. The lane-changing reminding ground-adhesion guidance 405 is displayed on the human-computer interaction interface of the navigation device to remind the vehicle to change lanes.

[0187] In some embodiments, referring to Figure 4E , Figure 4E The intersection guidance human-computer interaction interface diagram provided for the embodiments of the present application is shown in FIG. 6. The navigation device obtains the navigation route from the server, and the navigation route is to remind the intersection to turn at this time. The lane-changing reminding ground-adhesion guidance 406 is displayed on the human-computer interaction interface of the navigation device, and the lane-changing reminding floating guidance 407 is displayed in the air.

[0188] In some embodiments, after the navigation device obtains the navigation information from the server, the corresponding ground-adhesion guidance is generated according to the navigation route obtained by the navigation device from the server. The lane line needs to be extracted from the current frame image captured by the vehicle-mounted camera, and the lane line is projected into the vehicle coordinate system constructed with the vehicle as the origin. Next, the lane line extraction operation and the lane line projection operation will be described respectively.

[0189] For the lane line extraction operation, referring to Figure 5 , first, the navigation device obtains the current frame image captured by the vehicle-mounted camera, extracts the image features of the current frame image, and obtains a feature map;

[0190] Secondly, the obtained feature map is subjected to an encoding (Encoder) operation (i.e., a down-sampling operation). The obtained feature map is subjected to multiple convolution operations in the encoding stage to obtain an encoded feature map.

[0191] In some embodiments, the feature map is subjected to convolution by a seventh convolution layer 501 to obtain a convolutional feature map. The convolutional feature map is input into a seventh pooling layer 511 for pooling to compress the size of the feature map. Then, the feature map is subjected to multiple convolution and pooling operations by an eighth convolution layer 502, an eighth pooling layer 512, a ninth convolution layer 503, and a ninth pooling layer 513 to obtain a compressed feature map output by the ninth pooling layer 513, i.e., an encoded feature map.

[0192] Again, the decoded feature map is subjected to a decoding (i.e., up-sampling) operation. The decoded feature map is subjected to multiple de-convolution operations in the decoding stage to obtain a decoded feature map.

[0193] In some embodiments, the encoded feature map is subjected to inverse pooling by the tenth pooling layer 514 to obtain an inverse-pooled feature map, which is input to the tenth convolution layer 504 for convolution to extract image features. The inverse-pooled feature map is then subjected to multiple inverse-pooling and convolution operations by the eleventh pooling layer 515, the eleventh convolution layer 505, the twelfth pooling layer 516, and the twelfth convolution layer 506 to obtain a feature map of the original size output by the twelfth convolution layer 506, i.e., a decoded feature map.

[0194] Finally, the decoded feature map is subjected to a classification operation by a normalization function (Softmax) to classify the types, colors, and solid / void of the lane lines, respectively. The results with a type confidence, a color confidence, and a solid / void confidence are retained, and the three results are combined to output the classified lane lines. For example, white solid lane lines, white dashed lane lines, yellow solid lane lines, yellow dashed lane lines, and white solid stop lines.

[0195] In some embodiments, for lane line extraction, the method shown in Figure 5 The method shown can also be used to extract lane lines by a deep neural network. Here, the extraction method of the lane lines is not limited.

[0196] For the projection operation of the lane lines, a space coordinate system is first constructed with the vehicle as the origin. Referring to Figure 6A , Figure 6A The vehicle coordinate system provided by the embodiments of the present application is shown in the schematic diagram. The vehicle coordinate system is constructed with the vehicle as the origin (O point), the direction in which the vehicle travels as the horizontal axis (x axis), the upper side of the vehicle as the vertical axis (z axis), and the left or right side of the vehicle as the longitudinal axis (y axis).

[0197] In some embodiments, referring to Figure 6B , Figure 6B The selection method of the origin of the vehicle coordinate system is shown. The intersection 601 of the four wheels of the vehicle on the vehicle chassis is selected as the origin of the vehicle coordinate system, and the position of the vehicle-mounted camera 602 is set behind the windshield.

[0198] Secondly, the position of the vehicle-mounted camera in the vehicle coordinate system is calibrated. Referring to Figure 6C, get the coordinate value of the vehicle-mounted camera in the vehicle coordinate system and the attitude angle compared to the vehicle coordinate system: roll angle, pitch angle, yaw angle, where the roll angle represents the rotation value of the vehicle-mounted camera on the vertical axis, the pitch angle represents the rotation value of the vehicle-mounted camera on the vertical axis, and the yaw angle represents the rotation value of the vehicle-mounted camera on the horizontal axis; see Figure 6D , get the intrinsic parameters f of the vehicle camera x 、f y 、c x 、c y , where f x 、f y Respectively represent the focal length of the image of the vehicle camera in the x and y directions of the image coordinate system, c x 、c y Respectively represent the offset of the optical axis of the vehicle camera in the x and y directions of the image coordinate system, where f x 、c x The positive direction is horizontally to the right, f y 、c y The positive direction is vertically downward.

[0199] Again, the homography matrix H from the image of the vehicle camera to the vehicle coordinate system is constructed through the coordinate values ​​of the vehicle camera in the vehicle coordinate system, the attitude angle compared to the car coordinate system, and the internal parameters of the vehicle camera. The homography matrix H is the mapping between the points of the vehicle camera image and the points of the vehicle coordinate system. Using the homography matrix H, the exact position of the corresponding point of any point in the vehicle camera image in the vehicle coordinate system can be found.

[0200] Finally, the detected lane lines are projected into the vehicle coordinate system through the homography matrix. Figure 6E For the first lane line 611, the second lane line 612, the third lane line 613, and the fourth lane line 614 detected in the current road image, the positions of the first lane line 611, the second lane line 612, the third lane line 613, and the fourth lane line 614 in the vehicle coordinate system are found through the physical relationship H between the image and the vehicle coordinate system, and the first lane line 611, the second lane line 612, the third lane line 613, and the fourth lane line 614 are detected by fitting using a quadratic polynomial in the vehicle coordinate system. The results of the quadratic polynomial fitting are used as the first projected lane line 621, the second projected lane line 622, the third projected lane line 623, and the fourth projected lane line 624. The projected lane line 624 is projected, and the lane center line is described using a quadratic polynomial between any two adjacent projected lane lines. For example, the lane center line between the first projected lane line 621 and the second projected lane line 622 is described as the first lane center line 631 in the vehicle coordinate system, the lane center line between the second projected lane line 622 and the third projected lane line 623 is described as the second lane center line 632 in the vehicle coordinate system, and the lane center line between the third projected lane line 623 and the fourth projected lane line 624 is described as the third lane center line 633 in the vehicle coordinate system.

[0201] In some embodiments, after projecting the lane lines into the vehicle coordinate system constructed with the vehicle as the origin, two current following lane lines need to be determined. Referring to Figure 7A , the two historical following lane lines of the previous frame of road surface image (i.e., the two closest projected lane lines to the vehicle in the previous frame) are assigned unique identifiers, for example, the two historical following lane lines of the previous frame of road surface image are the fifth lane line 701 and the sixth lane line 702. The projected lane lines of the current frame are also assigned unique identifiers, and the multiple projected lane lines of the current frame are respectively matched with the historical following lane lines, and the projected lane line that is closest to the historical following lane line and has a distance less than a distance threshold is determined as a current following lane line according to the nearest matching principle. For example, the projected lane lines of the current frame are the seventh lane line 711 and the eighth lane line 712, and the nearest matching principle is adopted, for example, the fifth lane line 701 is closest to the seventh lane line 711, and the sixth lane line 702 is closest to the eighth lane line 712, and the seventh lane line 711 and the eighth lane line 712 are determined as the following lane lines in the current frame.

[0202] In some embodiments, when matching according to the nearest matching principle, a threshold of one meter can be set to filter out invalid matches.

[0203] In some embodiments, after the navigation device obtains the navigation information from the server, the current pose of the vehicle also needs to be detected. Referring to Figure 8 , Figure 8 A vehicle pose detection schematic diagram is provided for the embodiments of the present application. First, the current frame image of the vehicle-mounted camera is obtained, and the image features of the current frame image are extracted as the estimated pose of the vehicle-mounted camera, which is used to correct the drift value of the inertial sensor. Compared with the inertial sensor, the data of the vehicle-mounted camera almost does not produce drift, and if the vehicle-mounted camera is fixed, the pose estimation is also fixed, therefore, the data of the vehicle-mounted camera can effectively correct the drift value generated in the inertial sensor, and the disadvantage of the vehicle-mounted camera is that motion blur will occur when the motion is too fast, which is just compensated by the inertial sensor.

[0204] Secondly, the inertial sensor data is obtained, which includes the vehicle speed in the adaptive cruise control (ACC) system and the angular velocity and acceleration generated by the six-axis gyro, and for short-time rapid motion, the inertial sensor can provide estimation, and the estimated pose of the inertial sensor is obtained through the inertial sensor data.

[0205] Thirdly, the proportion of the estimated pose of the vehicle camera is determined to determine the correction scale. When determining the correction scale, the proportion of the estimated pose of the vehicle camera can be calculated by weighted average, the value to be corrected is determined by multiplying the estimated pose of the vehicle camera by the proportion, and the estimated pose of the inertial sensor is corrected.

[0206] Finally, the current pose is updated according to the state quantity, the state quantity (State) includes the estimated pose (Posea) of the inertial sensor, the correction scale (Scale) and the drift value (Biases), and thus the current pose of the vehicle is obtained, and the current pose is a pose value including six degrees of freedom (i.e., movement degrees of freedom along x, y and z three orthogonal coordinate axes and rotation degrees of freedom around the three coordinate axes).

[0207] In some embodiments, when the vehicle is at the starting point and has not generated motion, the current frame image of the vehicle camera can be extracted to obtain image features, and the pose estimation value is obtained, and the pose estimation value is directly used as the current pose of the vehicle.

[0208] In some embodiments, referring to Figure 4C , 4D , 4E, the navigation device obtains navigation information from the server, generates corresponding types of ground navigation instructions according to the navigation information, and before generating the ground navigation instructions, first determines the starting generation position of the ground navigation instructions by the pose, wherein the starting generation position is located between two current following lane lines, and then generates the ground navigation instructions along the two current following lane lines from the starting generation position, and the types of the generated ground navigation instructions include straight paving instructions, lane changing instructions and intersection instructions.

[0209] In some embodiments, referring to Figure 9 , Figure 9 a ground navigation instruction lifecycle diagram provided by the embodiments of the present application. For the ground navigation instruction, 1, the ground navigation instruction is generated; 2, the ground navigation instruction is compared with the current pose of the vehicle; 3, when it is determined through the position comparison that the ground navigation instruction is located behind the vehicle, the ground navigation instruction is destroyed.

[0210] It can be understood that in the embodiments of the present application, data related to user information is involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0211] The following continues to explain an exemplary structure of the implementation of the real scene navigation device 455 provided by the embodiments of the present application as a software module. In some embodiments, as shown in Figure 2 , the software modules in the real scene navigation device 455 stored in the memory 440 can include:

[0212] The data acquisition module 4551 is configured to display a current frame road surface image in real time in a display interface, where the current frame road surface image is obtained by image acquisition on a road on which the vehicle travels; the lane projection module 4552 is configured to identify a plurality of lane lines from the current frame road surface image, and project the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines; the lane following module 4553 is configured to detect two current following lane lines from the plurality of projected lane lines, where the two current following lane lines are the two closest projected lane lines to the vehicle; the pose detection module 4554 is configured to detect a current pose of the vehicle from the current frame road surface image; and the guidance generation module 4555 is configured to display a ground-hugging navigation guidance between the two current following lane lines in the display interface based on the current pose.

[0213] In some embodiments, before the lane projection module 4552 projects the plurality of lane lines into the vehicle coordinate system to obtain the plurality of corresponding projected lane lines, the lane projection module 4552 is further configured to construct the vehicle coordinate system with a projection point of intersection of diagonals of four end points of the vehicle on a chassis as an origin, with a positive direction or a negative direction of a transverse axis of a driving direction of the vehicle as a horizontal axis, with a positive direction or a negative direction of a vertical axis of an upper side of the vehicle as a vertical axis, and with a positive direction or a negative direction of a longitudinal axis of a left side of the vehicle as a longitudinal axis.

[0214] In some embodiments, the lane projection module 4552 is further configured to perform downsampling processing on the current frame road surface image to obtain a compressed size of a downsampled feature map, perform upsampling processing on the downsampled feature map to obtain a restored size of an upsampled feature map, perform classification processing based on the upsampled feature map to respectively obtain a lane line category classification result, a lane line color classification result, a lane line solid-void classification result, and respective confidences, take the lane line category classification result exceeding a lane line category confidence threshold, the lane line color classification result exceeding a lane line color confidence threshold, and the lane line solid-void classification result exceeding a lane line solid-void confidence threshold as lane line identification results, and determine the plurality of lane lines included in the current frame road surface image based on the lane line identification results.

[0215] In some embodiments, the lane projection module 4552 is further configured to perform the following processing for each lane line: project the lane line into the vehicle coordinate system based on a homography matrix between the current frame road surface image and the vehicle coordinate system to obtain a plurality of corresponding mapping points, and construct a quadratic polynomial curve fitted with the plurality of mapping points as a projected lane line corresponding to the lane line.

[0216] In some embodiments, before projecting the lane line into the vehicle coordinate system based on the homography matrix between the current frame road surface image and the vehicle coordinate system, the lane projection module 4552 is further configured to construct the homography matrix by: calibrating a camera in the vehicle used for image acquisition to obtain a position of the camera in the vehicle coordinate system; obtaining intrinsic parameters of the camera, and constructing a homography matrix from the current frame road surface image to the vehicle coordinate system based on the position of the camera in the vehicle coordinate system and the intrinsic parameters of the camera; wherein different road surface images taken by the same camera share the homography matrix, and different positions of the camera correspond to different homography matrices.

[0217] In some embodiments, the position of the camera in the vehicle coordinate system includes: coordinates of the camera in the vehicle coordinate system, and an attitude angle of the camera relative to the vehicle coordinate system; the lane projection module 4552 is further configured to determine the coordinates of the camera in the vehicle coordinate system as a translation relationship of the camera to an origin of the vehicle coordinate system, determine the attitude angle of the camera in the vehicle coordinate system as a rotation relationship of the camera to the origin of the vehicle coordinate system, determine the physical transformation quantity as an addition of the translation relationship and the rotation relationship, construct an intrinsic parameter matrix based on the intrinsic parameters of the camera, and determine a product of the intrinsic parameter matrix, the physical transformation quantity, and a scale ratio as the homography matrix, wherein the scale ratio represents a conversion ratio of the current frame road surface image to the vehicle coordinate system.

[0218] In some embodiments, the pose detection module 4554 is further configured to: when the current frame road surface image is acquired, obtain inertial sensor data synchronously output by an inertial sensor in the vehicle, determine a first estimated pose of the vehicle based on the inertial sensor data, extract image features from the current frame road surface image, determine a second estimated pose of the vehicle based on the image features, and correct the first estimated pose based on the second estimated pose to obtain a current pose of the vehicle.

[0219] In some embodiments, the inertial sensor data includes: acceleration and angular velocity of the vehicle; the pose detection module 4554 is further configured to: obtain a time difference between a current time and a previous time of the vehicle, determine a displacement value of the vehicle relative to the time difference and the acceleration, and integrate the angular velocity with respect to the time difference as an angular displacement, wherein the displacement value and the angular displacement are with respect to the vehicle at the previous time; and determine the displacement value and the angular displacement as the first estimated pose of the vehicle.

[0220] In some embodiments, the pose detection module 4554 is further configured to obtain a proportion of the second estimated pose in the current pose, and determine a product of the proportion and the second estimated pose as a correction amount, and determine an addition of the first estimated pose and the correction amount as the current pose of the vehicle.

[0221] In some embodiments, the lane following module 4553 is further configured to, for each of the two historical following lane lines in the previous frame of the road image, perform the following processing: match the plurality of projected lane lines with the historical following lane line respectively, and take the projected lane line closest to the historical following lane line and having a distance less than a distance threshold from the historical following lane line as a current following lane line.

[0222] In some embodiments, the guidance generation module 4555 is further configured to determine a starting generation position of the on-road navigation guidance based on the current pose, wherein the starting generation position is located between the two current following lane lines and on a plane on which the two current following lane lines are located, and display the on-road navigation guidance between the two current following lane lines from the starting generation position, wherein the on-road navigation guidance includes at least one of the following: a straight-ahead paving guidance, a lane-changing guidance, and an intersection guidance.

[0223] In some embodiments, after displaying the on-road navigation guidance between the two current following lane lines based on the current pose, the guidance generation module 4555 is further configured to perform display position detection processing on each of the generated on-road guidance, and destroy the on-road guidance when a display position detection result indicates that a display position of the on-road guidance between the two current following lane lines is behind the vehicle.

[0224] In some embodiments, the guidance generation module 4555 is further configured to output a voice navigation guidance, and display an on-road navigation guidance between the two current following lane lines based on the current pose in response to the vehicle not driving according to the voice navigation guidance, wherein the voice navigation guidance includes at least one of the following: a straight-ahead voice guidance, a lane-changing voice guidance, and an intersection voice guidance.

[0225] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the real scene navigation method provided in the embodiment of the present application.

[0226] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor executes the real scene navigation method provided by the embodiment of the present application.

[0227] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, and the like; or can be various devices including one or any combination of the above storage.

[0228] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0229] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0230] As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected through a communication network.

[0231] To sum up, through the embodiment of the present application, a plurality of lane lines can be recognized from a current frame road surface image, the plurality of lane lines are projected into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines, two current following lane lines are detected from the plurality of projected lane lines, and a current pose of the vehicle is detected from the current frame road surface image; finally, a ground-hugging navigation guide is displayed between the two current following lane lines based on the pose, and accurate fitting of the navigation guide and the road surface in a real scene can be realized.

[0232] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method of live navigation, characterized by, The method comprises: real-time display of a current frame road surface image in a display interface, wherein the current frame road surface image is obtained by image acquisition of a road on which a vehicle travels; identifying a plurality of lane lines from the current frame road surface image, and projecting the plurality of lane lines into a vehicle coordinate system to obtain a plurality of corresponding projected lane lines; detecting two current following lane lines from the plurality of projected lane lines, wherein the two current following lane lines are the two closest projected lane lines to the vehicle; determining a first estimated pose of the vehicle based on inertial sensor data synchronously output by an inertial sensor in the vehicle when the current frame road surface image is acquired; determining a second estimated pose of the vehicle based on image features of the current frame road surface image; obtaining a proportion of the second pose estimate in a current pose of the vehicle, wherein at an initial time, a ratio of the first estimated pose obtained by the inertial sensor to the second estimated pose at the initial time is taken as a proportion of the first estimated pose; at a next time, the proportion of the first estimated pose at the initial time is used to calculate the first estimated pose at the next time, and a ratio of the first estimated pose at the next time to the second estimated pose at the next time is taken as a proportion of the second estimated pose at the next time; and a mean of the proportion of the second estimated pose at the initial time and the proportion of the second estimated pose at the next time is obtained as the proportion of the second estimated pose at the next time in the current pose; determining a correction amount as a product of the proportion and the second estimated pose; and determining the current pose as a sum of the first estimated pose and the correction amount. in the display interface, displaying a ground-hugging navigation guide between the two current following lane lines based on the current pose.

2. The method of claim 1, wherein, Before the projecting the plurality of lane lines into the vehicle coordinate system to obtain the plurality of corresponding projected lane lines, the method further comprises: constructing the vehicle coordinate system with a projection point of an intersection of diagonals of four end points of the vehicle on a chassis as an origin, a positive direction or a negative direction of a horizontal axis of a driving direction of the vehicle as a positive direction or a negative direction of a vertical axis of an upper side of the vehicle, and a positive direction or a negative direction of a longitudinal axis of a left side of the vehicle.

3. The method of claim 1, wherein, The identifying the plurality of lane lines from the current frame road surface image comprises: performing down-sampling processing on the current frame road surface image to obtain a compressed size down-sampled feature map; performing up-sampling processing on the down-sampled feature map to obtain a restored size up-sampled feature map; performing classification processing based on the up-sampled feature map to obtain lane line category classification results, lane line color classification results, lane line solid-void classification results, and respective confidences; taking the lane line category classification results exceeding a lane line category confidence threshold, the lane line color classification results exceeding a lane line color confidence threshold, and the lane line solid-void classification results exceeding a lane line solid-void confidence threshold as lane line identification results; determining the plurality of lane lines included in the current frame road surface image based on the lane line identification results.

4. The method of claim 1, wherein, The projecting the plurality of lane lines into the vehicle coordinate system comprises: For each of the lane lines, the following processing is performed: Projecting the lane line into the vehicle coordinate system based on a homography matrix between the current frame road surface image and the vehicle coordinate system to obtain a plurality of corresponding mapping points; Constructing a quadratic polynomial curve fitted with the plurality of mapping points, and taking the quadratic polynomial curve as a projected lane line corresponding to the lane line.

5. The method of claim 4, wherein, Before the projecting the lane line into the vehicle coordinate system based on the homography matrix between the current frame road surface image and the vehicle coordinate system, the method further comprises: The homography matrix is constructed in the following manner: Calibrating a camera in the vehicle used for image acquisition to obtain a position of the camera in the vehicle coordinate system; Obtaining intrinsic parameters of the camera, and constructing a homography matrix from the current frame road surface image to the vehicle coordinate system based on the position of the camera in the vehicle coordinate system and the intrinsic parameters of the camera; Wherein, different road surface images taken by the same camera share the homography matrix, and different positions of the camera correspond to different homography matrices.

6. The method of claim 5, wherein: The position of the camera in the vehicle coordinate system comprises coordinates of the camera in the vehicle coordinate system and an attitude angle of the camera relative to the vehicle coordinate system; The constructing of the homography matrix from the current frame road surface image to the vehicle coordinate system based on the position of the camera in the vehicle coordinate system and the intrinsic parameters of the camera comprises: Determining the coordinates of the camera in the vehicle coordinate system as a translation relationship of the camera to an origin of the vehicle coordinate system; Determining the attitude angle of the camera in the vehicle coordinate system as a rotation relationship of the camera to the origin of the vehicle coordinate system; Determining the physical transformation quantity as a sum of the translation relationship and the rotation relationship; Constructing an intrinsic parameter matrix based on the intrinsic parameters of the camera; Taking a product of the intrinsic parameter matrix, the physical transformation quantity and a scale ratio as the homography matrix, wherein the scale ratio represents a conversion ratio of the current frame road surface image to the vehicle coordinate system.

7. The method of claim 1, wherein: The inertial sensor data comprises acceleration and angular velocity of the vehicle; The determining of the first estimated pose of the vehicle based on the inertial sensor data comprises: Obtaining a time difference between a current time and a previous time of the vehicle, determining a displacement value of the vehicle relative to the time difference and the acceleration, and integrating the angular velocity with respect to the time difference as an angular displacement, wherein the displacement value and the angular displacement are referenced to the vehicle at the previous time; Determining the displacement value and the angular displacement as the first estimated pose of the vehicle.

8. The method of claim 1, wherein, The two current following lane lines are detected from the plurality of projected lane lines, and the method comprises the following steps: For the two historical following lane lines in the last frame road surface image, the following processing is performed respectively: The plurality of projected lane lines are matched with the historical following lane lines respectively, and the projected lane line which is closest to the historical following lane line and has a distance less than a distance threshold from the historical following lane line is taken as a current following lane line.

9. The method of claim 1, wherein, The ground navigation guide is displayed between the two current following lane lines based on the current pose, and the method comprises the following steps: The starting generation position of the ground navigation guide is determined based on the current pose, wherein the starting generation position is located between the two current following lane lines and on the plane where the two current following lane lines are located; The ground navigation guide is displayed between the two current following lane lines from the starting generation position; The type of the ground navigation guide comprises a straight paving guide, a lane changing guide, and an intersection guide.

10. The method of claim 9, wherein, After the ground navigation guide is displayed between the two current following lane lines from the starting generation position, the method further comprises the following steps: The display position of each generated ground navigation guide is detected; When the display position detection result indicates that the display position of the ground navigation guide between the two current following lane lines is behind the vehicle, the ground navigation guide is destroyed.

11. The method of claim 9, wherein, The starting generation position of the ground navigation guide is determined based on the current pose, and the method comprises the following steps: The current pose is added to a set pose to obtain the starting generation position of the ground navigation guide.

12. The method of claim 1, wherein, The ground navigation guide is displayed between the two current following lane lines based on the current pose, and the method comprises the following steps: Voice navigation guide is output, and in response to the vehicle not driving according to the voice navigation guide, the ground navigation guide is displayed between the two current following lane lines based on the current pose; The type of the voice navigation guide comprises at least one of the following: a straight voice guide, a lane changing voice guide, and an intersection voice guide.

13. A live navigation device, characterized by The device comprises: A data acquisition module is configured to display a current frame road surface image in a display interface in real time, wherein the current frame road surface image is obtained by image acquisition of a road on which a vehicle travels; A lane projection module is configured to identify a plurality of lane lines from the current frame road surface image, project the plurality of lane lines into a vehicle coordinate system, and obtain a plurality of corresponding projected lane lines; A lane following module is configured to detect two current following lane lines from the plurality of projected lane lines, wherein the two current following lane lines are the two closest projected lane lines to the vehicle. The pose detection module is configured to: when the current frame of the road surface image is collected, determine a first estimated pose of the vehicle based on inertial sensing data synchronously output by an inertial sensor in the vehicle; determine a second estimated pose of the vehicle based on image features of the current frame of the road surface image; and obtain a proportion of the second estimated pose in a current pose of the vehicle, wherein at an initial time, a ratio of the first estimated pose obtained by the inertial sensor to the second estimated pose at the initial time is taken as a proportion of the first estimated pose; at a next time, a proportion of the first estimated pose at the initial time is used to calculate the first estimated pose at the next time, a ratio of the first estimated pose at the next time to the second estimated pose at the next time is taken as a proportion of the second estimated pose at the next time, and an average of the proportion of the second estimated pose at the initial time and the proportion of the second estimated pose at the next time is taken as the proportion of the second estimated pose at the next time in the current pose; and determine a product of the proportion and the second estimated pose as a correction amount, and determine a sum of the first estimated pose and the correction amount as the current pose. The guidance generation module is configured to display, in the display interface, a ground-hugging navigation guidance between the two current following lane lines based on the current pose.

14. The apparatus of claim 13, wherein, The lane projection module is further configured to: construct the vehicle coordinate system with a projection point of an intersection of diagonal lines of four end points of the vehicle on a chassis as an origin, a positive direction or a negative direction of a horizontal axis of a driving direction of the vehicle as a horizontal axis, a positive direction or a negative direction of a vertical axis of an upper side of the vehicle as a vertical axis, and a positive direction or a negative direction of a longitudinal axis of a left side of the vehicle as a longitudinal axis.

15. The apparatus of claim 13, wherein, The lane projection module is further configured to: perform down-sampling processing on the current frame of the road surface image to obtain a down-sampled feature map with a compressed size; perform up-sampling processing on the down-sampled feature map to obtain an up-sampled feature map with a restored size; perform classification processing based on the up-sampled feature map to respectively obtain lane line type classification results, lane line color classification results, lane line solid-void classification results, and respective confidences; take the lane line type classification results exceeding a lane line type confidence threshold, the lane line color classification results exceeding a lane line color confidence threshold, and the lane line solid-void classification results exceeding a lane line solid-void confidence threshold as lane line recognition results; determine a plurality of lane lines included in the current frame of the road surface image based on the lane line recognition results.

16. A live navigation device characterized by The real-scene navigation device comprises: a memory configured to store executable instructions; a processor configured to execute the executable instructions or computer programs stored in the memory to implement the real-scene navigation method of any one of claims 1 to 12.

17. A computer-readable storage medium storing executable instructions or a computer program, characterized in that, The executable instructions are executed by the processor to implement the real-scene navigation method of any one of claims 1 to 12.

18. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the real-scene navigation method of any one of claims 1 to 12.

Images