Augmented reality-based navigation processing method, apparatus, device, and program product

Abstract

The present disclosure relates to an augmented reality-based navigation processing method, device, equipment and program product. The method comprises: acquiring a current frame image and historical vanishing point coordinates in historical frame images; predicting a road vanishing point in the current frame image based on the historical vanishing point coordinates, and determining predicted vanishing point coordinates; detecting a road vanishing point in the current frame image, and determining detected vanishing point coordinates; and determining current vanishing point coordinates in the current frame image based on the predicted vanishing point coordinates and the detected vanishing point coordinates. In this way, the accuracy, smoothness and anti-interference ability of vanishing point detection are improved, more accurate and better continuous basis data for subsequent calculation of a yaw angle corresponding to the current frame image are provided, and the phenomenon of direction jump of a virtual navigation mark can be greatly reduced, thereby improving the accuracy and continuity of AR navigation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of map, and particularly relates to an augmented reality based navigation processing method, device, equipment and program product. BACKGROUND

[0002] In an augmented reality (AR) navigation scene, a virtual navigation mark needs to be continuously and correctly displayed at a corresponding position in a real-world image to guide a user. The direction of the virtual navigation mark is mainly determined according to a camera yaw angle, and the camera yaw angle is mainly determined by a magnetometer.

[0003] However, the magnetometer is susceptible to magnetic field interference, so that when AR navigation is performed in a region with magnetic field interference, the yaw angle corresponding to each frame of image has errors and is not smooth and continuous, so that the direction of the virtual navigation mark deviates from the actual direction and the direction may be discontinuous between frames of images. SUMMARY

[0004] To solve the technical problem of low accuracy of the yaw angle in the AR navigation scene due to the susceptibility of the magnetometer to magnetic field interference, the present disclosure provides an augmented reality based navigation processing method, device, equipment and program product.

[0005] In a first aspect, an augmented reality based navigation processing method is provided, comprising:

[0006] obtaining historical vanishing point coordinates in a current frame of image and historical frames of image;

[0007] predicting a road vanishing point in the current frame of image based on the historical vanishing point coordinates, and determining predicted vanishing point coordinates;

[0008] detecting a road vanishing point in the current frame of image, and determining detected vanishing point coordinates;

[0009] determining current vanishing point coordinates in the current frame of image based on the predicted vanishing point coordinates and the detected vanishing point coordinates.

[0010] In a second aspect, an augmented reality based navigation processing device is also provided, comprising:

[0011] a data obtaining module configured to obtain historical vanishing point coordinates in a current frame of image and historical frames of image;

[0012] a predicted vanishing point coordinates determining module configured to predict a road vanishing point in the current frame of image based on the historical vanishing point coordinates, and determine predicted vanishing point coordinates;

[0013] a detected vanishing point coordinate determination module configured to detect a road vanishing point in the current frame image and determine a detected vanishing point coordinate;

[0014] a current vanishing point coordinate determination module configured to determine a current vanishing point coordinate in the current frame image based on the predicted vanishing point coordinate and the detected vanishing point coordinate.

[0015] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising:

[0016] a memory and a processor, wherein the memory is configured to store processor-executable instructions;

[0017] the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the augmented reality-based navigation processing method provided by any of the embodiments of the present disclosure.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the augmented reality-based navigation processing method provided by any of the embodiments of the present disclosure.

[0019] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product for executing the augmented reality-based navigation processing method provided by any of the embodiments of the present disclosure.

[0020] The technical solutions provided by the embodiments of the present disclosure have at least the following advantages compared with the prior art: on the one hand, the vanishing point coordinates in the real world are calculated to provide data basis for the calculation of the yaw angle, avoiding the problem that the yaw angle is interfered by the magnetic field, thereby improving the accuracy of the yaw angle by improving the accuracy of the vanishing point coordinates; on the other hand, through the fusion processing of the detected vanishing point coordinates in the current frame image, the vanishing point coordinates calculated based on the historical vanishing point coordinates in the historical frame image, and the detected vanishing point coordinates and the predicted vanishing point coordinates, more accurate current vanishing point coordinates in the current frame image can be obtained, and the smoothness and continuity of the vanishing point coordinates in the continuous frame images can be improved, thereby providing data basis for obtaining smooth and continuous yaw angle, and thus the accuracy and continuity of the subsequent virtual navigation mark pointing direction can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent by describing in detail the following specific embodiments with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0022] Figure 1 A flowchart of a navigation processing method based on augmented reality provided by an embodiment of the present disclosure is shown in FIG. 1;

[0023] Figure 2 A processing process diagram of a navigation processing method based on augmented reality considering uncertainty of vanishing points provided by an embodiment of the present disclosure is shown in FIG. 2;

[0024] Figure 3 A flowchart of another navigation processing method based on augmented reality provided by an embodiment of the present disclosure is shown in FIG. 3;

[0025] Figure 4 A display effect diagram of augmented reality navigation marks before and after yaw angle correction provided by an embodiment of the present disclosure is shown in FIG. 4;

[0026] Figure 5 A structure diagram of a navigation processing device based on augmented reality provided by an embodiment of the present disclosure is shown in FIG. 5;

[0027] Figure 6 A structure diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 6. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that each step described in the method embodiments of the present disclosure can be executed in different order, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions of other terms will be given in the description below.

[0031] It should be noted that the "first", "second", and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the "one", "multiple" modification mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0033] In an augmented reality (AR) navigation scenario, the yaw angle of a navigation device can be determined according to a magnetometer, and a virtual navigation mark is rendered in an image of the real world according to the yaw angle to guide the user. However, the magnetometer is susceptible to magnetic field interference, resulting in low accuracy of the yaw angle of the navigation device, and thus poor rendering effect of the virtual navigation mark.

[0034] In order to improve the accuracy of the yaw angle, vanishing point detection can be performed on the image, and a more accurate yaw angle can be calculated from the correspondence between the vanishing point in the image and the yaw angle, thereby improving the fitting degree of the virtual navigation mark and the ground features in the image in the AR navigation scenario. However, the vanishing point detection in the related art not only requires high computing resources, increasing the power consumption of the navigation device, but also mainly targets single-frame image vanishing point detection, which does not have the ability to process multiple consecutive images, resulting in poor vanishing point smoothness in consecutive images (such as video) and lack of anti-interference ability. Thus, for AR navigation, the virtual navigation mark is prone to freezing, direction jumping, and the like, reducing the accuracy and continuity of AR navigation.

[0035] Based on the above situation, the embodiment of the present disclosure provides an augmented reality-based navigation processing method to detect a road vanishing point in a current frame image, determine the coordinates of the detected vanishing point, and predict the road vanishing point in the current frame image using historical vanishing point coordinates in historical frame images, determine the predicted vanishing point coordinates, and fuse the detected vanishing point coordinates and the predicted vanishing point coordinates to obtain the current vanishing point coordinates in the current frame image that fuse historical information, thereby improving the accuracy, smoothness and anti-interference ability of continuously obtaining the vanishing point in the image during user travel, providing more accurate and better continuity of the basis data for subsequent calculation of the yaw angle corresponding to the current frame image, and thus greatly reducing the phenomenon of direction jumping of the virtual navigation mark, improving the accuracy and continuity of AR navigation.

[0036] Figure 1A flowchart of a navigation processing method based on augmented reality provided by an embodiment of the present disclosure can be applied to a scenario of AR navigation based on an electronic map. The electronic map can be a high-precision map, a high-definition map, or a three-dimensional map with high map accuracy, or a standard-precision map, a navigation map, or a two-dimensional map with relatively low map accuracy. The navigation processing method based on augmented reality can be executed by a navigation processing device based on augmented reality. The device can be implemented by software and / or hardware and can be integrated on an electronic device with certain computing capability and installed with an electronic map client. The electronic device can be a smart phone, a tablet computer, a palm computer, a smart wearable device, a notebook computer, a vehicle-mounted device, or the like.

[0037] As shown in Figure 1 The navigation processing method based on augmented reality provided by an embodiment of the present disclosure can include the following steps.

[0038] S110, obtaining a current frame image and historical vanishing point coordinates in a historical frame image.

[0039] The current frame image refers to an image obtained by shooting at the current moment. The historical frame image refers to an image obtained by shooting at a historical moment before the current moment, for example, a previous adjacent frame image or previous several frame images of the current frame image. The historical vanishing point coordinates refer to coordinates of a road vanishing point in the historical frame image, which are obtained when the historical frame image is processed. The vanishing point refers to a projection point of a visual intersection point of parallel lines in a real space in an imaging plane in the case of perspective deformation. In the embodiment of the present disclosure, the vanishing point is an imaging point of an intersection point of two edge lines of a road in user travel in an image.

[0040] Specifically, in the AR navigation scenario, the electronic device generates a virtual navigation mark in combination with map data and a navigation route, and obtains an image of a road in user travel, so as to project the virtual navigation mark on the road in user travel in the image by AR imaging technology, and guide the user. In order to improve the accuracy of navigation guidance, the embodiment of the present disclosure can improve the accuracy and smoothness of the yaw angle by processing the road vanishing point in the image before rendering the navigation mark.

[0041] In the process of processing the road vanishing point of each frame image, the electronic device fuses historical information in the historical frame image before the current frame image, so as to improve the continuity and smoothness of the road vanishing point. Therefore, the electronic device obtains the current frame image, and obtains the historical frame image and the historical vanishing point coordinates in the historical frame image, which are collectively used as input data for subsequent processing.

[0042] S120, predicting a road vanishing point in the current frame image based on the historical vanishing point coordinates, and determining predicted vanishing point coordinates.

[0043] wherein the predicted vanishing point coordinate refers to a coordinate of a possible position of the road vanishing point in the current frame image predicted by the historical information.

[0044] Specifically, there is information continuity between the front and rear frame images, so the electronic device can predict the predicted vanishing point coordinate in the current frame image according to the historical vanishing point coordinate in the historical frame image, with the aid of the relevant information in the current frame image and the historical frame image.

[0045] In some embodiments, the electronic device can predict the road vanishing point in the current frame image with the aid of the motion relationship of the pixel points. That is, S120 comprises: performing sparse optical flow tracking processing based on the current frame image, the historical frame image and the historical vanishing point coordinate, to determine the predicted vanishing point coordinate.

[0046] wherein the optical flow is the instantaneous velocity of the pixel motion of the spatial moving object on the observation imaging plane, is a method for finding the correspondence between the previous frame and the current frame by using the change of the pixels in the image sequence in the time domain and the correlation between the adjacent frames, thereby calculating the motion information of the object between the adjacent frames, and is a motion vector describing the motion of the moving object in the three-dimensional space reflected in the two-dimensional image. The sparse optical flow tracking algorithm (such as KLT algorithm) is an algorithm for performing optical flow tracking on the sparse feature point set of each frame image.

[0047] Specifically, the electronic device can take the current frame image I t , the historical frame image I t-1 and the historical vanishing point coordinate x t-1 as input data, and calculate the predicted vanishing point coordinate x′ t using the KLT algorithm shown in the following formula.

[0048] x′ t = KLT (I t-1 , I t , x t-1 ).

[0049] In other embodiments, the electronic device can predict the road vanishing point in the current frame image with the aid of the imaging rule of the adjacent frame images. That is, S120 comprises: performing camera projection transformation based on the historical vanishing point coordinate, the camera intrinsic parameter matrix and the camera relative pose from the historical frame image to the current frame image, to determine the predicted vanishing point coordinate.

[0050] wherein the camera intrinsic matrix is a matrix describing the correspondence between a point in the camera coordinate system and a point in the image coordinate system, which is represented by the focal length and the principal point coordinates (the offset of the camera optical center relative to the image plane). The camera intrinsic matrix can be determined by the factory parameters of the camera shooting the image, or can be determined by calibrating the parameters of the camera. The camera relative pose is used to describe the relative relationship between the pose of the camera at the current time and the pose of the camera at the historical time, which can be obtained by the above two pose calculations, and the pose at each time can be measured by the sensor associated with the camera, or obtained by calibrating the camera extrinsic parameters.

[0051] Specifically, the current frame image and the historical frame image are at least two frames of images in continuous images shot in a short time, and it can be considered that there are some same ground points therein. Due to the change of the camera pose at different times, the ground points may exhibit different image projection features when projected into the current frame image and the historical frame image. Therefore, the transformation rule of the ground points in the images at different times can be calculated according to the difference between the camera intrinsic and extrinsic parameters at the current time and the historical time, that is, the camera intrinsic matrix and the camera relative pose, and then the predicted vanishing point coordinates in the current frame image can be inferred in combination with the historical vanishing point coordinates in the historical frame image.

[0052] In specific implementation, the electronic device can use the historical vanishing point coordinates x t-1 , the camera intrinsic matrix C, and the camera relative pose between the historical frame image and the current frame image to calculate the predicted vanishing point coordinates x′ t .

[0053]

[0054] In still some embodiments, the electronic device can predict the road vanishing point in the current frame image by comprehensively considering the motion relationship of the pixel points and the imaging rule of the adjacent frame images. That is, S120 comprises: performing sparse optical flow tracking processing and camera projection transformation based on the current frame image, the historical frame image, the historical vanishing point coordinates, the camera intrinsic matrix, and the camera relative pose to determine the predicted vanishing point coordinates.

[0055] Specifically, the electronic device can use the KLT algorithm and the camera projection transformation algorithm to comprehensively predict the road vanishing point in the current image.

[0056] In an example, the electronic device can use the KLT algorithm and the camera projection transformation algorithm to calculate the initial vanishing point coordinates respectively, and then calculate the mean value of the two initial vanishing point coordinates as the predicted vanishing point coordinates.

[0057] In another example, the electronic device can first execute one of the two algorithms described above to obtain initial vanishing point coordinates; then, the electronic device executes the other algorithm by taking the initial vanishing point coordinates as input data instead of the historical vanishing point coordinates, to obtain predicted vanishing point coordinates. For example, the electronic device can first run the sparse optical flow tracking algorithm (such as the KLT algorithm) by taking the historical vanishing point coordinates, the current frame image, and the historical frame image as input data, to obtain initial vanishing point coordinates; then, the electronic device runs the camera projection transformation described above by taking the initial vanishing point coordinates, the camera intrinsic matrix, and the camera relative pose, to obtain final predicted vanishing point coordinates. In another example, the electronic device can first run the camera projection transformation described above by taking the historical vanishing point coordinates, the camera intrinsic matrix, and the camera relative pose, to obtain initial vanishing point coordinates; then, the electronic device runs the sparse optical flow tracking algorithm (such as the KLT algorithm) by taking the initial vanishing point coordinates, the current frame image, and the historical frame image as input data, to obtain predicted vanishing point coordinates. In this way, the accuracy of the predicted vanishing point coordinates can be improved to some extent.

[0058] S130, detecting a road vanishing point in the current frame image, and determining detected vanishing point coordinates.

[0059] The detected vanishing point coordinates refer to the coordinates of the road vanishing point obtained by image processing of the current frame image.

[0060] Specifically, the electronic device can perform corresponding image processing on the current frame image by using related technologies for vanishing point detection, to obtain the detected vanishing point coordinates. For example, the detected vanishing point coordinates in the current frame image can be estimated by using a combination of the Gabor filter and the voting method; in another example, the detected vanishing point coordinates can be obtained by inputting the current frame image into a pre-trained deep neural network model for determining road vanishing points for vanishing point detection; in yet another example, the detected vanishing point coordinates can be obtained by performing line detection and intersection filtering on the current frame image.

[0061] S140, determining current vanishing point coordinates in the current frame image based on the predicted vanishing point coordinates and the detected vanishing point coordinates.

[0062] The current vanishing point coordinates refer to the coordinates of the road vanishing point in the current frame image finally determined.

[0063] Specifically, according to the above description, if only the detected vanishing point coordinates in the current frame image are used, discontinuity and non-smoothness between the vanishing point coordinates in each frame image can occur. The predicted vanishing point coordinates are calculated using the historical vanishing point coordinates and the information continuity in the adjacent frame images. Therefore, in the embodiments of the present disclosure, the predicted vanishing point coordinates and the detected vanishing point coordinates can be fused to obtain the current vanishing point coordinates. For example, the predicted vanishing point coordinates and the detected vanishing point coordinates can be subjected to mean filtering, weighted filtering and the like, and a final vanishing point coordinate is output as the current vanishing point coordinate in the current frame image. The current vanishing point coordinate retains the information of the vanishing point in the current frame image and fuses the continuity information in the historical frame image, thereby ensuring the smoothness between the current vanishing point coordinate and the historical vanishing point coordinates in the historical frame image and improving the anti-interference capability.

[0064] In some embodiments, in order to further improve the accuracy and reliability of the vanishing point coordinates, the uncertainty of the calculation result can be increased in the process of calculating the vanishing point coordinates. On the one hand, the reliability of the vanishing point coordinates can be represented, and on the other hand, the uncertainty can be used to determine the weight of the vanishing point coordinate fusion, thereby further improving the accuracy of the current vanishing point coordinates.

[0065] On the basis of this embodiment, referring to Figure 2 The navigation processing method based on augmented reality can comprise:

[0066] S210, obtaining a current frame image, historical vanishing point coordinates and historical uncertainty in a historical frame image.

[0067] The historical uncertainty is used to represent the reliability of the historical vanishing point coordinates. The smaller the uncertainty, the higher the reliability. The uncertainty can be represented by confidence, standard deviation, covariance and uncertainty and the like.

[0068] Specifically, the electronic device can obtain the uncertainty of the historical vanishing point coordinates, i.e., the historical uncertainty, while obtaining the historical vanishing point coordinates.

[0069] In an example, when the historical frame image is the first frame image, the historical uncertainty can be determined by an empirical set value or a related value determined by a related person when evaluating the image quality of the first frame image.

[0070] In another example, the historical uncertainty can be calculated from the historical frame image according to the calculation method of the uncertainty measurement index and the requirement of the input data. For example, when the historical frame image is the first frame image, the historical uncertainty can be obtained by corresponding processing in this manner. It can be understood that when the historical frame image is not the first frame image, the historical uncertainty can also be obtained by processing in this manner.

[0071] In yet another example, the historical uncertainty can also be obtained in the manner that the uncertainty of the current vanishing point coordinate (i.e., the current uncertainty) is calculated in the embodiments of the present disclosure. For example, the historical frame image is not the first frame image, and when the historical frame image is processed as the current frame image at the corresponding time, it can be obtained in the manner of fusing historical information in the embodiments of the present disclosure. In this example, the historical uncertainty can be recorded in the associated information of the historical frame image together with the historical vanishing point coordinate, and then the electronic device can directly read the associated information to obtain the historical uncertainty.

[0072] S220, based on the historical vanishing point coordinate, predicting the road vanishing point in the current frame image, determining the predicted vanishing point coordinate, and based on the historical uncertainty, determining the predicted uncertainty corresponding to the predicted vanishing point coordinate.

[0073] The predicted uncertainty is used to represent the reliability of the predicted vanishing point coordinate.

[0074] Specifically, because the predicted vanishing point coordinate is calculated from the historical vanishing point coordinate, the predicted uncertainty is largely related to the historical uncertainty. Based on this, after the electronic device calculates the predicted vanishing point coordinate in the current frame image from the historical vanishing point coordinate, it can also calculate the reliability of the predicted vanishing point coordinate, i.e., the predicted uncertainty, from the historical uncertainty.

[0075] Taking the uncertainty as a covariance example, the electronic device can calculate the predicted uncertainty of the predicted vanishing point coordinate, i.e., the predicted covariance P t-1 , based on the historical covariance P t ′ .

[0076] P′ t = P t-1 + Q.

[0077] Wherein Q is a preset uncertainty increment value. Q can be an empirical setting (such as setting a unit matrix); or it can be an unknown parameter, which is solved by using the above-mentioned sparse optical flow tracking processing and / or camera projection transformation processing according to the information change law between the front and rear frame images.

[0078] S230, detecting the road vanishing point in the current frame image, determining the detected vanishing point coordinate, and based on the preset uncertainty index and the image features associated with the detected vanishing point coordinate in the current frame image, determining the detected uncertainty corresponding to the detected vanishing point coordinate.

[0079] The detection uncertainty is used to represent the reliability of the detected vanishing point coordinates. The preset uncertainty index refers to a preselected uncertainty measurement index, which can be any one of confidence, standard deviation, covariance, and uncertainty.

[0080] Specifically, in the process of vanishing point detection on the current frame image, the electronic device can extract corresponding image feature data from the current frame image according to the calculation method of the preset uncertainty index and the requirement of the input data, and calculate the detection uncertainty of the detected vanishing point coordinates.

[0081] In an example, in the case that the vanishing point detection method is a vanishing point detection method combining Gabor filter and voting method, or a vanishing point detection method of line detection and intersection screening, the corresponding feature data can be extracted from the current frame image according to the probability statistical method of the preset uncertainty index, and the detection uncertainty is calculated according to the probability statistical method.

[0082] In an example, in the case that the vanishing point detection method is a vanishing point detection method combining Gabor filter and voting method, or a vanishing point detection method of line detection and intersection screening, the corresponding feature data can be extracted from the current frame image according to the probability statistical method of the preset uncertainty index, and the detection uncertainty is calculated according to the probability statistical method.

[0083] In another example, in the case of using a deep neural network model for vanishing point detection, the detection uncertainty can be used as one of the output parameters of the model to participate in the training of the neural network model. In this way, when the deep neural network model is run to output the detected vanishing point coordinates, the detection uncertainty can be output at the same time.

[0084] In some embodiments, there is a lot of noise information in the real-world images involved in the walking navigation scene and the cycling navigation scene, such as the environment where the road in the image is located is relatively complex, so that there are many and miscellaneous straight lines in the image. The calculation resource requirement of the vanishing point detection method in the related art will increase with the increase of the number and complexity of the straight lines in the image, and the accuracy of the vanishing point detection method in the related art will decrease with the increase of the complexity of the straight lines in the image. Therefore, in order to reduce the calculation resource consumption of vanishing point detection, so that it can run smoothly on a mobile terminal or a vehicle-mounted terminal with relatively weak computing power, and in order to further improve the accuracy of vanishing point detection, the vanishing point detection can be performed by S231-S233 in the embodiment as follows.

[0085] S231, performing line detection on the current frame image to generate at least one initial straight line.

[0086] Specifically, the electronic device can perform line extraction in the current frame image by using a line extractor such as EDLines edge detection, LSD, Hough transform, and the like, to obtain a plurality of extracted lines, i.e., initial lines.

[0087] S232, filtering each initial line based on a preset condition to obtain a target line.

[0088] The preset condition is a condition for filtering lines that is set in advance. The preset condition includes at least one of the following: the line is on the ground, the angle between the line and the camera optical axis is within a preset angle range, the length of the line is within a preset length range, and the horizontal distance of the line from the camera optical center is within a preset distance range. The preset angle range, the preset length range, and the preset distance range are all critical values of the corresponding dimensions that are set empirically.

[0089] Specifically, there may be lines in the above-obtained initial lines that are not in the road area. These lines not only increase the data processing amount and cause unnecessary resource consumption, but also interfere with the subsequent vanishing point detection and reduce the accuracy of vanishing point detection. Therefore, the electronic device filters the above-obtained initial lines based on the preset condition, and the remaining initial lines after filtering are the target lines. Subsequent vanishing point detection using these target lines can not only reduce resource consumption and improve detection efficiency, but also reduce the influence of noise lines and improve detection accuracy.

[0090] S233, detecting a vanishing point based on each target line, determining the coordinates of the detected vanishing point, and determining the uncertainty of the detected vanishing point based on a preset uncertainty index, each target line, and the coordinates of the detected vanishing point.

[0091] Specifically, any two non-parallel target lines can intersect to obtain an intersection point, and each target line can obtain a plurality of intersection points. The electronic device can select the intersection point with the highest reliability or the intersection point with a reliability that meets certain requirements from the intersection points. Then, when the selected intersection point is one, it is directly used as the detected vanishing point; when the selected intersection point is multiple, a random intersection point can be selected as the detected vanishing point, or the selected intersection points can be sorted according to reliability, and the second or middle intersection point is selected as the detected vanishing point to eliminate the error influence of reliability calculation. The reliability can be represented by the number of target lines passing through the intersection point, for example, the more target lines passing through an intersection point, the higher the reliability of the intersection point.

[0092] In addition, the electronic device can determine the required input data from each target line and the coordinates of the detected vanishing point according to the calculation requirements of the preset uncertainty index, and calculate the detected uncertainty according to the calculation method of the preset uncertainty index.

[0093] For example, when the preset uncertainty index is the uncertainty in the measurement field, the perpendicular distance between the detected vanishing point and each target straight line can be calculated, the average of the perpendicular distances is calculated, and the maximum value of the difference between each perpendicular distance and the average is determined as the detected uncertainty.

[0094] For another example, when the preset uncertainty index is the covariance, the detected uncertainty can be calculated by using the above-mentioned perpendicular distance and the calculation formula of the covariance.

[0095] In some embodiments, in order to further improve the accuracy and calculation efficiency of the detected vanishing point and the detected uncertainty, a random sample consensus (RANSAC) algorithm can be used for vanishing point detection, that is, S233 comprises: determining the coordinates of the detected vanishing point by screening each intersection formed by each target straight line by using the random sample consensus algorithm; screening each associated straight line within a preset region range of the coordinates of the detected vanishing point from each target straight line; and determining the covariance corresponding to the coordinates of the detected vanishing point as the detected uncertainty based on the perpendicular distance between the coordinates of the detected vanishing point and each associated straight line.

[0096] The preset region range is a preset intersection screening range, which can be determined by a preset distance radius, and the preset distance radius can be empirically set.

[0097] Specifically, the electronic device calculates a plurality of intersections from each target straight line. Then, the RANSAC algorithm is performed on each intersection to screen out the most reliable intersection that meets the requirements of reliability from the intersections of any two target straight lines, as the coordinates of the detected vanishing point. Then, the electronic device can determine whether any of the above-mentioned intersections is within a preset region range of the coordinates of the detected vanishing point. If yes, the target straight line corresponding to the intersection is determined as an associated straight line (also referred to as an inlier straight line) having a correlation with the coordinates of the detected vanishing point. Moreover, the electronic device can calculate the perpendicular distance between the coordinates of the detected vanishing point and each associated straight line. For example, there are nine associated straight lines, and then nine perpendicular distances can be calculated. Subsequently, the electronic device can calculate the covariance by using the perpendicular distances as the detected uncertainty. For example, the covariance formula and the nine perpendicular distances are used to calculate the detected uncertainty; or the average of the nine perpendicular distances can be calculated, and the product of the average and the unit matrix is determined as the detected uncertainty.

[0098] S240, based on the predicted vanishing point coordinates, the predicted uncertainty, the detected vanishing point coordinates and the detected uncertainty, determining the current vanishing point coordinates and the current uncertainty in the current frame image.

[0099] The current uncertainty is used to represent the reliability of the current vanishing point coordinates.

[0100] Specifically, according to the above description, the predicted uncertainty characterizes the reliability of the predicted vanishing point coordinates, and the detected uncertainty characterizes the reliability of the detected vanishing point coordinates. Then, the electronic device can determine the one with smaller uncertainty from the predicted vanishing point coordinates and the detected vanishing point coordinates as the current vanishing point coordinates according to the numerical size relationship between the predicted uncertainty and the detected uncertainty. Similarly, the smaller numerical uncertainty can be determined as the current uncertainty.

[0101] Alternatively, the electronic device can take the predicted uncertainty and the detected uncertainty as weights, and perform fusion processing on the predicted vanishing point coordinates and the detected vanishing point coordinates to obtain the current vanishing point coordinates. Moreover, the current uncertainty can be calculated by the predicted uncertainty and the detected uncertainty in the fusion processing manner.

[0102] In some embodiments, S240 includes: performing weighted fusion processing on the predicted vanishing point coordinates and the detected vanishing point coordinates by Kalman filtering respectively with the predicted uncertainty and the detected uncertainty as weights, to generate the current vanishing point coordinates and the current uncertainty.

[0103] Specifically, the electronic device performs weighted filtering processing on the predicted vanishing point coordinates and the detected vanishing point coordinates respectively with the predicted uncertainty and the detected uncertainty as weighted weights according to the calculation manner of Kalman filtering, and can calculate and output the current vanishing point coordinates and the current uncertainty. Through such weighted filtering processing, the accuracy of the current vanishing point coordinates and the current uncertainty can be further improved.

[0104] In some embodiments, the current vanishing point coordinates and the current uncertainty calculated by the above embodiments can be used as the basis for rendering virtual navigation marks (i.e., augmented reality navigation marks) in AR navigation to improve the rendering effect and display accuracy of the augmented reality navigation marks. As shown in the following figure, the augmented reality-based navigation processing method provided in the present embodiment includes: Figure 3

[0105] S310, obtaining a current frame image, historical vanishing point coordinates and historical uncertainty in historical frame images.

[0106] S320, predicting a road vanishing point in the current frame image based on the historical vanishing point coordinates, determining the predicted vanishing point coordinates, and determining the predicted uncertainty corresponding to the predicted vanishing point coordinates based on the historical uncertainty.

[0107] S330, detecting a road vanishing point in the current frame image to determine the detected vanishing point coordinates, and determining the detected uncertainty corresponding to the detected vanishing point coordinates based on a preset uncertainty index and image features associated with the detected vanishing point coordinates in the current frame image.​

[0108] S340, determine a current vanishing point coordinate and a current uncertainty in the current frame image based on the predicted vanishing point coordinate, the predicted uncertainty, the detected vanishing point coordinate and the detected uncertainty.

[0109] S350, determine a current yaw angle and a current pitch angle corresponding to the current frame image based on the current vanishing point coordinate.

[0110] Specifically, in the AR navigation process, the augmented reality navigation mark (such as a virtual three-dimensional arrow or a virtual guide line) is positioned and displayed in the current frame image according to the yaw angle, the pitch angle and the like. The yaw angle (i.e. the initial yaw angle) and the pitch angle (i.e. the initial pitch angle) in the camera pose when the current frame image is captured can have a certain deviation, thereby causing the augmented reality navigation mark to be displayed inaccurately. According to the principle of projective geometry, there is a certain calculation relationship between the vanishing point in the image and the yaw angle and the pitch angle therein. Therefore, in the embodiment, the current vanishing point coordinate and the current uncertainty can be used to calculate a new yaw angle (i.e. the current yaw angle) and a new pitch angle (i.e. the current pitch angle) in the current frame image with higher accuracy, to provide a more accurate and reliable basis for rendering of the augmented reality navigation mark.

[0111] The initial yaw angle and the initial pitch angle can be measured by a sensor (such as a magnetometer) associated with the camera, or can be calculated by a camera extrinsic calibration method.

[0112] In some embodiments, the electronic device can determine the current yaw angle from the current vanishing point coordinate according to the corresponding relationship between the yaw angle and the vanishing point coordinate in the image. For example, the corresponding relationship between the yaw angle and the vanishing point in the image is that if the yaw angle of the camera is greater than 0, the vanishing point of the road will move to the right of the road center, and the greater the yaw angle, the greater the distance of the vanishing point moving to the right. If the yaw angle of the camera is less than 0, the vanishing point of the road will move to the left of the road center, and the smaller the yaw angle, the greater the distance of the vanishing point moving to the left. Then, the electronic device can calculate the current yaw angle according to the deviation direction and the deviation distance of the current vanishing point coordinate relative to the road center.

[0113] Similarly, the electronic device can determine the current pitch angle from the current vanishing point coordinate according to the corresponding relationship between the pitch angle and the vanishing point coordinate in the image. For example, the corresponding relationship between the pitch angle and the vanishing point in the image is that if the pitch angle of the camera is greater than 0, the vanishing point of the road will move upward from the road vertical reference line when the pitch angle is 0, and the greater the pitch angle, the greater the distance of the vanishing point moving upward. If the pitch angle of the camera is less than 0, the vanishing point of the road will move downward from the road vertical reference, and the smaller the pitch angle, the greater the distance of the vanishing point moving downward. Then, the electronic device can calculate the current pitch angle according to the deviation direction and the deviation distance of the current vanishing point coordinate relative to the road vertical reference.

[0114] In some other embodiments, S350 includes: if the current uncertainty is less than a preset threshold, then determining the current yaw angle and the current pitch angle based on the current vanishing point coordinates and the camera intrinsic parameter matrix; if the current uncertainty is greater than or equal to the preset threshold, then determining the initial yaw angle and the initial pitch angle corresponding to the current frame image as the current yaw angle and the current pitch angle, respectively.

[0115] The preset threshold is a pre-set critical value for uncertainty, which can be determined based on the precision and recall required by the business. The preset threshold can be a single numerical value or a threshold matrix consistent with the covariance dimension.

[0116] Specifically, given that the current vanishing point coordinates may also be inaccurate, this embodiment determines the current yaw angle and current pitch angle based on the current uncertainty to further ensure the accuracy of the angles on which the augmented reality navigation sign rendering is based.

[0117] For example, taking uncertainty as a two-dimensional covariance matrix, the electronic device compares the current uncertainty with a preset threshold.

[0118] For example, when the preset threshold is a single value, the determinant of the two-dimensional covariance matrix can be calculated, and the result of the determinant calculation can be compared with the preset threshold.

[0119] For example, when the preset threshold is a single value, another threshold can be calculated from the preset threshold and an empirically set threshold coefficient. Then, the preset threshold and the other threshold are used as the elements on the diagonal of a two-dimensional matrix to form a two-dimensional threshold matrix. Alternatively, the preset threshold can be a two-dimensional threshold matrix containing the elements on the diagonal of a two-dimensional matrix. In this case, the elements on the diagonal of the two-dimensional covariance matrix can be compared with the corresponding elements in the two-dimensional threshold matrix to characterize the reliability of the current vanishing point coordinates in the x and y directions.

[0120] If the comparison result shows that the current uncertainty is greater than or equal to the preset threshold, it indicates that the reliability of the current vanishing point coordinates is low, and the accuracy of the yaw angle calculated from the current vanishing point coordinates may also be low. In this case, the initial yaw angle and initial pitch angle can be directly used as the basis for rendering augmented reality navigation markers, that is, the initial yaw angle and initial pitch angle are determined as the current yaw angle and current pitch angle, respectively.

[0121] If the comparison result shows that the current uncertainty is less than a preset threshold, it indicates that the reliability of the current vanishing point coordinates is high. Therefore, the current yaw angle and current pitch angle can be calculated from the current vanishing point coordinates. At this point, the electronic equipment can use the following formula to calculate the yaw angle and pitch angle from the current vanishing point coordinates x... t The current yaw angle is calculated from the camera intrinsic parameter matrix C.′ and the current pitch angle ′ .

[0122]

[0123] wherein a, b and g represent the first dimension row vector, the second dimension row vector and the third dimension row vector in the matrix obtained by the first formula respectively.

[0124] S360, rendering the augmented reality navigation mark based on the current yaw angle and the current pitch angle, and displaying the rendering result in the current frame image.

[0125] Specifically, the electronic device determines the orientation of the rendering and display of the augmented reality navigation mark with the current yaw angle and the current pitch angle, and renders it into the current frame image using the relevant rendering settings, and displays the rendering result in the visual screen. In this way, the rendering orientation of the augmented reality navigation mark of AR navigation can be calculated using the vanishing point coordinates with higher accuracy, better smoothness and stronger anti-interference ability, thereby greatly reducing the phenomenon of direction jump of the augmented reality navigation mark, and improving the accuracy and continuity of AR navigation.

[0126] Referring to Figure 4 , the voice and text mark display of AR navigation is a navigation guide of "straight ahead". However, in the left (a) figure, due to the error offset of the initial yaw angle, the augmented reality navigation mark 410 in the (a) figure without yaw angle correction does not point to the straight ahead direction of the road, but is offset to the right side direction of the road, which is easy to cause the problem of AR navigation guide error.

[0127] After the processing of the above embodiments, the current vanishing point coordinates can be used to obtain the more accurate current yaw angle required for rendering. Then, as Figure 4 shown in the right (b) figure, the augmented reality navigation mark 420 after yaw angle correction points to the straight ahead direction of the road, which is consistent with the navigation guide displayed by the text mark, and improves the correctness and intuitiveness of AR navigation guide.

[0128] Figure 5 A structure schematic diagram of a navigation processing device based on augmented reality provided by the embodiments of the present disclosure. As Figure 5 shown, the navigation processing device based on augmented reality 500 provided by the embodiments of the present disclosure can include:

[0129] The data acquisition module 510 is configured to acquire the current frame image and the historical vanishing point coordinates in the historical frame image.

[0130] The predicted vanishing point coordinate determination module 520 is configured to determine the predicted vanishing point coordinate based on the historical vanishing point coordinates.

[0131] The detected vanishing point coordinate determination module 530 is configured to detect a road vanishing point in the current frame image, and determine a detected vanishing point coordinate.

[0132] The current vanishing point coordinate determination module 540 is configured to determine a current vanishing point coordinate in the current frame image based on the predicted vanishing point coordinate and the detected vanishing point coordinate.

[0133] In some embodiments, the augmented reality based navigation processing apparatus 500 further comprises an uncertainty representing a reliability degree of the vanishing point coordinate;

[0134] Correspondingly, the data acquisition module 510 is further configured to:

[0135] acquire a historical uncertainty corresponding to the historical vanishing point coordinate;

[0136] The predicted vanishing point coordinate determination module 520 is configured to:

[0137] predict a road vanishing point in the current frame image based on the historical vanishing point coordinate, determine a predicted vanishing point coordinate, and determine a predicted uncertainty corresponding to the predicted vanishing point coordinate based on the historical uncertainty;

[0138] The detected vanishing point coordinate determination module 530 is configured to:

[0139] detect a road vanishing point in the current frame image, determine a detected vanishing point coordinate, and determine a detected uncertainty corresponding to the detected vanishing point coordinate based on a preset uncertainty index and an image feature in the current frame image associated with the detected vanishing point coordinate;

[0140] The current vanishing point coordinate determination module 540 is configured to:

[0141] determine a current vanishing point coordinate and a current uncertainty in the current frame image based on the predicted vanishing point coordinate, the predicted uncertainty, the detected vanishing point coordinate and the detected uncertainty.

[0142] In some embodiments, the current vanishing point coordinate determination module 540 is specifically configured to:

[0143] respectively take the predicted uncertainty and the detected uncertainty as a weight, and perform a weighted fusion processing on the predicted vanishing point coordinate and the detected vanishing point coordinate by using a Kalman filter to generate the current vanishing point coordinate and the current uncertainty.

[0144] In some embodiments, the augmented reality based navigation processing apparatus 500 further comprises:

[0145] The current yaw angle determination module is configured to determine a current yaw angle and a current pitch angle corresponding to the current frame image based on the current vanishing point coordinate after determining the current vanishing point coordinate and the current uncertainty in the current frame image based on the predicted vanishing point coordinate, the predicted uncertainty, the detected vanishing point coordinate and the detected uncertainty.

[0146] The AR rendering module is configured to render an augmented reality navigation mark based on the current yaw angle and the current pitch angle, and display the rendering result in the current frame image.

[0147] Further, the current yaw angle determination module is specifically configured to:

[0148] if the current uncertainty is less than the preset threshold, determine the current yaw angle and the current pitch angle based on the current vanishing point coordinate and the camera intrinsic matrix;

[0149] if the current uncertainty is greater than or equal to the preset threshold, determine the initial yaw angle and the initial pitch angle corresponding to the current frame image as the current yaw angle and the current pitch angle respectively; wherein the initial yaw angle and the initial pitch angle are the yaw angle and the pitch angle when the current frame image is captured.

[0150] In some embodiments, the predicted vanishing point coordinate determination module 520 is specifically configured to realize the function of determining the predicted vanishing point coordinate by predicting the road vanishing point in the current frame image based on the historical vanishing point coordinates in any one of the following ways:

[0151] performing sparse optical flow tracking processing based on the current frame image, the historical frame image and the historical vanishing point coordinates to determine the predicted vanishing point coordinate;

[0152] performing camera projection transformation based on the historical vanishing point coordinates, the camera intrinsic matrix and the camera relative pose from the historical frame image to the current frame image to determine the predicted vanishing point coordinate;

[0153] performing sparse optical flow tracking processing and camera projection transformation based on the current frame image, the historical frame image, the historical vanishing point coordinates, the camera intrinsic matrix and the camera relative pose to determine the predicted vanishing point coordinate.

[0154] In some embodiments, the detected vanishing point coordinate determination module 530 comprises:

[0155] The initial straight line generation submodule is configured to perform straight line detection on the current frame image to generate at least one initial straight line.

[0156] The target straight line obtaining submodule is configured to obtain target straight lines by screening the initial straight lines based on preset conditions, wherein the preset conditions include at least one of the following: the straight line is on the ground, the straight line and the camera optical axis included angle is within a preset angle range, the straight line length is within a preset length range, and the horizontal distance of the straight line from the camera optical center is within a preset distance range.

[0157] The detected vanishing point coordinate determining submodule is configured to determine the detected vanishing point coordinates based on the target straight lines, and determine the detected uncertainty based on a preset uncertainty index, the target straight lines and the detected vanishing point coordinates.

[0158] Further, the detected vanishing point coordinate determining submodule is specifically configured to:

[0159] The random sample consensus algorithm is used to screen the intersection points formed by the target straight lines to determine the detected vanishing point coordinates.

[0160] The associated straight lines within the preset area range of the detected vanishing point coordinates are screened from the target straight lines.

[0161] The covariance corresponding to the detected vanishing point coordinates is determined based on the perpendicular distance between the detected vanishing point coordinates and the associated straight lines, as the detected uncertainty.

[0162] The navigation processing device based on augmented reality provided by the embodiments of the present disclosure can execute the navigation processing method based on augmented reality provided by any embodiment of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method. The contents not described in detail in the device embodiments of the present disclosure can be referred to the description in any method embodiment of the present disclosure.

[0163] Figure 6 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure is used to exemplarily illustrate the electronic device for implementing the navigation processing method based on augmented reality in any embodiment of the present disclosure, and should not be understood as a specific limitation of the embodiments of the present disclosure.

[0164] As shown in Figure 6 The electronic device 600 can include a processor (for example, a central processing unit, a graphics processing unit, etc. 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0165] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. While the electronic device 600 is shown with various devices, it is understood that all of the shown devices are not required to be implemented or possessed. More or less devices can alternatively be implemented or possessed.

[0166] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the functions defined in the augmented reality based navigation processing method provided by any embodiments of the present disclosure can be performed.

[0167] It should be noted that the computer readable medium in the above disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0168] In some embodiments, the client, server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communications (e.g., a communications network) of any form or medium, such as the Internet. Examples of communications networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0169] The computer readable medium described above can be included in the electronic device described above; or can exist separately, without being assembled into the electronic device.

[0170] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the augmented reality based navigation processing method provided by any embodiment of the present disclosure.

[0171] Computer program code for carrying out operations of the present disclosure can be written in any one or more of a variety of programming languages or combinations of languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0172] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0173] The modules involved in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0174] The functions described above in the detailed description can be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0175] In the context of the present disclosure, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium would include a tangible, non-transitory memory such as a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] The above description is only preferred embodiments of the present disclosure and the explanation of the applied technical principles. It should be understood by those skilled in the art that the disclosure range involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0177] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0178] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A navigation processing method based on augmented reality, characterized by, The method comprises: obtaining historical vanishing point coordinates and historical uncertainty in a current frame image and a historical frame image, wherein the uncertainty represents the reliability of the vanishing point coordinates; based on the historical vanishing point coordinates, predicting the road vanishing point in the current frame image, determining the predicted vanishing point coordinates, and based on the historical uncertainty, determining the predicted uncertainty corresponding to the predicted vanishing point coordinates; detecting the road vanishing point in the current frame image, determining the detected vanishing point coordinates, and based on a preset uncertainty indicator and image features associated with the detected vanishing point coordinates in the current frame image, determining the detected uncertainty corresponding to the detected vanishing point coordinates; based on the predicted vanishing point coordinates, the predicted uncertainty, the detected vanishing point coordinates and the detected uncertainty, determining the current vanishing point coordinates and the current uncertainty in the current frame image, to improve the accuracy and smoothness of the yaw angle.

2. The method of claim 1, wherein, The method comprises: respectively weighting the predicted uncertainty and the detected uncertainty, and using Kalman filtering to perform weighted fusion processing on the predicted vanishing point coordinates and the detected vanishing point coordinates, to generate the current vanishing point coordinates and the current uncertainty.

3. The method of claim 1, wherein, After the current vanishing point coordinates and the current uncertainty in the current frame image are determined based on the predicted vanishing point coordinates, the predicted uncertainty, the detected vanishing point coordinates and the detected uncertainty, the method further comprises: based on the current vanishing point coordinates, determining the current yaw angle and the current pitch angle corresponding to the current frame image; based on the current yaw angle and the current pitch angle, rendering an augmented reality navigation mark, and displaying the rendering result in the current frame image.

4. The method of claim 3, wherein, The method comprises: if the current uncertainty is less than a preset threshold, determining the current yaw angle and the current pitch angle based on the current vanishing point coordinates and a camera intrinsic matrix; if the current uncertainty is greater than or equal to the preset threshold, determining the initial yaw angle and the initial pitch angle corresponding to the current frame image as the current yaw angle and the current pitch angle respectively; wherein the initial yaw angle and the initial pitch angle are the yaw angle and the pitch angle when the current frame image is captured.

5. The method of claim 1, wherein, The method comprises: based on the current frame image, the historical frame image and the historical vanishing point coordinates, performing sparse optical flow tracking processing to determine the predicted vanishing point coordinates; based on the historical vanishing point coordinates, a camera intrinsic matrix and a camera relative pose from the historical frame image to the current frame image, performing camera projection transformation to determine the predicted vanishing point coordinates; The sparse optical flow tracking processing and camera projection transformation are performed based on the current frame image, the historical frame image, the historical vanishing point coordinates, the camera intrinsic parameter matrix and the camera relative pose, to determine the predicted vanishing point coordinates.

6. The method of claim 1, wherein, The detecting the road vanishing point in the current frame image, determining the detected vanishing point coordinates, and determining the detected uncertainty corresponding to the detected vanishing point coordinates based on a preset uncertainty index and image features associated with the detected vanishing point coordinates in the current frame image include: Performing straight line detection on the current frame image to generate at least one initial straight line; Screening each of the initial straight lines based on a preset condition to obtain a target straight line, wherein the preset condition includes at least one of the following: the straight line is on the ground, the straight line and the camera optical axis have an included angle within a preset angle range, the straight line has a length within a preset length range, and the straight line has a horizontal distance from the camera optical center within a preset distance range; Performing vanishing point detection based on each of the target straight lines, determining the detected vanishing point coordinates, and determining the detected uncertainty based on the preset uncertainty index, each of the target straight lines and the detected vanishing point coordinates.

7. The method of claim 6, wherein, In a case where the preset uncertainty index is a covariance, the performing vanishing point detection based on each of the target straight lines, determining the detected vanishing point coordinates, and determining the detected uncertainty based on the preset uncertainty index, each of the target straight lines and the detected vanishing point coordinates include: Screening each intersection formed by each of the target straight lines using a random sample consensus algorithm to determine the detected vanishing point coordinates; Screening each associated straight line within a preset area range of the detected vanishing point coordinates from each of the target straight lines; Determining the covariance corresponding to the detected vanishing point coordinates based on the perpendicular distance between the detected vanishing point coordinates and each of the associated straight lines as the detected uncertainty.

8. An augmented reality-based navigation processing apparatus characterized by comprising: The method includes: A data acquisition module is configured to acquire a current frame image, historical frame images, historical vanishing point coordinates and historical uncertainty, wherein the uncertainty represents the reliability of the vanishing point coordinates; A predicted vanishing point coordinate determination module is configured to predict a road vanishing point in the current frame image based on the historical vanishing point coordinates, determine predicted vanishing point coordinates, and determine a predicted uncertainty corresponding to the predicted vanishing point coordinates based on the historical uncertainty; A detected vanishing point coordinate determination module is configured to detect a road vanishing point in the current frame image, determine detected vanishing point coordinates, and determine a detected uncertainty corresponding to the detected vanishing point coordinates based on a preset uncertainty index and image features associated with the detected vanishing point coordinates in the current frame image; A current vanishing point coordinate determination module is configured to determine current vanishing point coordinates and current uncertainty in the current frame image based on the predicted vanishing point coordinates, the predicted uncertainty, the detected vanishing point coordinates and the detected uncertainty, to improve the accuracy and smoothness of the yaw angle.

9. An electronic device, comprising: The method includes: a memory for storing the processor-executable instructions; the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the augmented reality based navigation processing method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, the computer program product is configured to implement the augmented reality based navigation processing method according to any one of claims 1 to 7.

Images