A vision-assisted navigation method and system in an urban environment based on an omnidirectional camera

Abstract

The present invention relates to the field of global satellite navigation and positioning technology, and in particular to a visually assisted navigation method and system in an urban environment based on an omnidirectional camera. The urban sky image in the form of an omnidirectional image taken by an omnidirectional camera is restored to an omnidirectional image after correction and sky area detection, the GNSS signal at the time of collecting the original urban sky image is obtained, and the satellite in the GNSS signal is projected onto the omnidirectional sky area image, and the LOS satellite and NLOS satellite are determined according to the positional relationship between the satellite and the sky area. After the NLOS satellite is eliminated, only the LOS satellite is used for positioning, which can not only improve the positioning accuracy and stability, but also has a small computational load, does not need to rely on the accuracy and precision of modeling, and does not require a large amount of data sets for training and learning compared to the existing NLOS satellite signal elimination method.

Description

Technical Field

[0001] The present invention relates to the field of global satellite navigation and positioning technology, and in particular to a vision-assisted navigation method and system in an urban environment based on an omnidirectional camera. Background Art

[0002] With the continuous development and improvement of the global satellite navigation system, the global satellite navigation system has played a huge role in the field of intelligent transportation and unmanned driving. Due to the availability of multi-constellation global navigation satellite systems (GPS, Beidou, GLONASS, Galileo and QZSS), GNSS solutions are becoming more and more popular. If the GNSS receiver receives enough direct signals transmitted by satellites, the so-called line-of-sight signals (LOS), GNSS positioning can achieve good performance. However, in highly urbanized areas, GNSS propagation may be reflected, diffracted or blocked by skyscrapers and moving objects, causing additional signal transmission delays. The resulting multipath effect and non-line-of-sight (NLOS) signal reception will increase the error of the solution result, which may result in positioning errors of several meters or even thousands of meters, seriously affecting the positioning accuracy of the global satellite navigation system. In order to improve the availability and positioning accuracy of the global satellite navigation system in urban environments, the importance of eliminating non-line-of-sight signals is self-evident.

[0003] The existing methods for removing NLOS signals are:

[0004] Spatial statistical method: The spatial statistical method is used to generate multipath signals with spatial distribution inherent to the user's location to reduce multipath phenomena and improve GNSS positioning effects.

[0005] 3D Building Model Approach: 3D building models are used to improve cross-track positioning accuracy in urban canyons by predicting which satellites are visible from different locations and comparing them with measured satellite visibility to determine the location.

[0006] Machine learning method: classify the received signals, detect NLOS signals, and thus improve the GNSS positioning effect.

[0007] Prior technology shortcomings:

[0008] Spatial statistical method: As the number of satellites increases, the computational load increases, and the computational efficiency decreases when multiple satellites receive multipath errors.

[0009] Three-dimensional building model method: It is necessary to obtain three-dimensional information of the surrounding environment in advance or in real time. It is difficult for the algorithm to successfully identify all NLOS satellites and it is extremely dependent on the accuracy and precision of the modeling.

[0010] Machine learning methods: require a large amount of data sets for training and need to be labeled. Summary of the invention

[0011] The present invention provides a vision-assisted navigation method and system in an urban environment based on an omnidirectional camera, which solves the problems of low efficiency of NLOS satellite signal elimination methods in satellite positioning in existing urban environments, inability to meet the requirements of solution accuracy, or the algorithm being extremely dependent on the accuracy and precision of the model, thereby improving the availability and positioning accuracy of the global satellite navigation system in an urban environment.

[0012] In a first aspect, an embodiment of the present invention provides a method for visually assisted navigation in an urban environment based on an omnidirectional camera, comprising the following steps:

[0013] An omnidirectional camera is used to collect an original city sky image in the form of an omnidirectional image, and a normalized plane correction is performed on the original city sky image to obtain a corrected city sky image;

[0014] Performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image;

[0015] Acquire the GNSS signal at the time when the original city sky image is collected, and project the satellites in the GNSS signal onto the omnidirectional sky area image;

[0016] Determine whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite;

[0017] Positioning is performed via the LOS satellites.

[0018] In a further implementation of the first aspect of the present invention, the step of performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image further comprises:

[0019] extracting a sky boundary from the corrected city sky image;

[0020] According to the sky boundary, the sky area is marked from the corrected urban sky image, and then converted back to an omnidirectional image to obtain an omnidirectional sky area image.

[0021] In a further embodiment of the first aspect of the present invention, the step of extracting a sky boundary from the corrected city sky image further comprises:

[0022] Extracting the corrected city sky image gradient information;

[0023] Define the sky boundary function border(x), which is:

[0024] 1≤border(x)≤H(1≤x≤W),

[0025] Where W and H represent the width and height of the corrected city sky image, and the sky area and non-sky area are defined as:

[0026] sky={(x,y)|1≤x≤W,1≤y≤border(x)},

[0027] ground={(x,y)|1≤x≤W,border(x)≤y≤H};

[0028] The improved energy function is determined as:

[0029]

[0030] Among them, Σ s and Σ g represents the covariance matrix of the sky area and the non-sky area represented by RGB values. γ is the parameter of the uniformity of the sky area and its value is 2. and corresponds to the largest eigenvalue of the two matrices, Σ s and Σ g The definition is as follows:

[0031]

[0032]

[0033] Among them, N s and N g Represents the number of pixels in the sky area and the non-sky area, I is the input RGB image, μ s and μ g The average RGB values ​​of the sky area and the non-sky area respectively;

[0034] Define t as the threshold of the sky energy function border(x), where 5≤t≤600, given a first threshold, obtain the sky boundary of the corrected city sky image corresponding to the first threshold, which is the first sky boundary, and obtain an improved energy function value according to the first sky boundary, which is the first improved energy function value;

[0035] Using a step search strategy with a step size of 5 to obtain a second sky boundary, a third sky boundary, ..., an nth sky boundary, and corresponding second improved energy function values, third improved energy function values, ..., and an nth improved energy function value;

[0036] A maximum improved energy function value is obtained, wherein the sky boundary corresponding to the maximum improved energy function value is the sky boundary of the corrected city sky image.

[0037] In a further embodiment of the first aspect of the present invention, extracting the sky boundary from the corrected city sky image further comprises detecting a pseudo sky area within the sky boundary of the corrected city sky image, the steps being:

[0038] Define a diff_abs(x) function, wherein diff_abs(x)=|border(x+1)-border(x)|(1≤x≤W-1);

[0039] If diff_abs(x)>H / 3, it is determined that a pseudo sky area exists in the sky area corresponding to the sky boundary function;

[0040] The K-means algorithm is used to divide the area with the pseudo sky area into two clusters, and the average vector μ of the two clusters is obtained. s1 and μ s2 , the covariance matrix Σ s1 and Σ s2 , calculate μ s1 and μ g , μ s2 and μ g The Mahalanobis distance between them is the cluster with a larger Mahalanobis distance, and the cluster with a smaller Mahalanobis distance is the pseudo-sky region.

[0041] In a further implementation of the first aspect of the present invention, the step of acquiring the GNSS signal at the time of collecting the original city sky image and projecting the satellites in the GNSS signal onto the omnidirectional sky area image further includes:

[0042] Align the timestamps of the original city sky image and the GNSS signal collected by the receiver to obtain the GNSS signal received when the original city sky image was collected;

[0043] Calculate the pseudorange, altitude and azimuth of the satellite from the GNSS signal, and represent the satellite in the ENU coordinate system to obtain a star map at the time of original city sky image acquisition;

[0044] The star map is projected onto the omnidirectional sky area image according to the imaging principle of an omnidirectional camera.

[0045] In a further implementation of the first aspect of the present invention, before projecting the star map onto the omnidirectional sky area image according to the omnidirectional camera imaging principle, it is also necessary to solve the angle between the original city sky image and the true north direction, and the specific steps are as follows:

[0046] Calculate the speed of the camera in the ENU coordinate system from the GNSS signal;

[0047] The angle between the original city sky image and the true north direction is calculated using north_velocity and east_velocity.

[0048] In a second aspect, an embodiment of the present invention provides a visually assisted navigation system in an urban environment based on an omnidirectional camera, the system comprising:

[0049] Image acquisition module: used to collect original city sky images in the form of omnidirectional images through an omnidirectional camera;

[0050] Image correction module: used for performing normalized plane correction on the original city sky image to obtain a corrected city sky image;

[0051] Sky area detection module: used to perform sky area detection on the corrected urban sky image and convert it back into an omnidirectional image to obtain an omnidirectional sky area image;

[0052] Satellite projection module: used to obtain the GNSS signal at the time of collecting the original city sky image, and project the satellite in the GNSS signal onto the omnidirectional sky area image;

[0053] Satellite judgment module: used to judge whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite;

[0054] Positioning module: used for positioning through the LOS satellite.

[0055] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor, and the processor can execute the program instructions stored in the memory to execute the control method described in any one of claims 1 to 6.

[0056] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed, the method according to any one of claims 1 to 6 is implemented.

[0057] The present invention provides a visual assisted navigation method, system, computer device and computer storage medium in an urban environment based on an omnidirectional camera. An omnidirectional original city sky image is obtained by an omnidirectional camera, and image correction, sky area detection and image restoration are performed on the original city sky image to obtain an omnidirectional sky area image. The GNSS signal at the time of collecting the original city sky image is obtained, and the satellite in the GNSS signal is projected onto the omnidirectional sky area image, and the LOS satellite and NLOS satellite are determined according to the positional relationship between the satellite and the sky area. The NLOS satellite is eliminated and only the LOS satellite is used in a global satellite navigation system in an urban environment, so as to improve the satellite positioning effect, solve the problems that the NLOS satellite signal elimination method in the existing navigation system has low calculation efficiency, the solution accuracy cannot meet the requirements, or the algorithm is extremely dependent on the accuracy and precision of the model, and improve the availability and positioning accuracy of the global satellite navigation system in an urban environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a method for visually assisted navigation in an urban environment based on an omnidirectional camera according to an embodiment of the present invention;

[0059] Figure 2 is an omnidirectional original city sky image taken by the omnidirectional camera in an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of an omnidirectional image forming process in an embodiment of the present invention;

[0061] Figure 4 is a corrected city sky image in an embodiment of the present invention;

[0062] Figure 5 It is a schematic diagram of a preferred process of performing sky area detection on a corrected city sky image and converting it back to an omnidirectional image in an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of a process for extracting a sky boundary from a corrected city sky image in an embodiment of the present invention;

[0064] Figure 7 is a sobel operator convolution template in an embodiment of the present invention;

[0065] Figure 8 is a sky area detection result of correcting the city sky image in an embodiment of the present invention;

[0066] Fig. 9 is a sky area detection result restored to an omnidirectional image in an embodiment of the present invention;

[0067] Fig.10 is an omnidirectional city sky image projected by a satellite in an embodiment of the present invention;

[0068] Fig.11 is a schematic diagram of a process for solving the angle between an omnidirectional city image and the true north direction in an embodiment of the present invention;

[0069] Fig.12 It is a structural schematic diagram of a visually assisted navigation system module in an urban environment based on an omnidirectional camera in an embodiment of the present invention.

[0070] Fig.13 It is a schematic diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following is a detailed description of the implementation of the present invention in conjunction with the accompanying drawings. The embodiments are provided only for illustrative purposes and cannot be understood as limiting the present invention. The accompanying drawings are only for reference and illustration purposes and do not constitute a limitation on the scope of patent protection of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] In one embodiment, see Figure 1 As shown, the present invention provides a visually assisted navigation method in an urban environment based on an omnidirectional camera, comprising the following steps:

[0073] S10, collecting an original city sky image in the form of an omnidirectional image through an omnidirectional camera, and performing normalized plane correction on the original city sky image to obtain a corrected city sky image.

[0074] Utilize the 360° shooting function of the omnidirectional camera and point it toward the sky to collect original city sky images in omnidirectional form.

[0075] like Figure 2 The figure shows the original city sky image in omnidirectional form. It can be seen from the figure that the city sky image taken by the omnidirectional camera has certain distortion, which does not conform to the normal visual conventions of humans and is not convenient for sky detection of subsequent images. Therefore, it is necessary to correct the distorted city sky omnidirectional image.

[0076] In this embodiment, the following scheme is used to correct the original omnidirectional city sky image.

[0077] Compared to the pinhole model, which can directly project 3D points onto a normalized plane, the imaging process of an omnidirectional camera has an additional intermediate process: first projecting the 3D points onto a unit sphere, and then projecting the points on the unit sphere onto a normalized plane. Figure 3 Shown is a schematic diagram of the imaging process of an omnidirectional camera.

[0078] First, the spatial point χ=[X,Y,Z] is projected onto the unit sphere, χs is the coordinate of a space point on the unit sphere:

[0079]

[0080] Then the χ on the sphere s Transform to O p In the coordinate system with the origin as the coordinate:

[0081]

[0082] Where ξ represents the distance from the center of the unit sphere to the origin of the normalized plane coordinate system.

[0083] Then transform the coordinates of the point on the unit sphere to the normalized plane:

[0084]

[0085] Taking into account the radial distortion and tangential distortion in the projection process, the distortion matrix D = [k 1 ,k 2 ,p 1 ,p 2 ]Get m u The distorted point m d =(x d ,y d ),in:

[0086] x d =x u (1+k 1 r 2 +k 2 r 4 )+2p 1 x u y u +p 2 (r 2 +2x u 2 ),

[0087] y d =y u (1+k 1 r 2 +k 2 r 4 )+p 1 (r 2 +2y u 2 )+2p 2 x u y u ,

[0088] where r 2 =xu 2 +y u 2 .

[0089] Using the camera internal parameter K, m d Transform to the pixel plane and get point p,

[0090]

[0091] The above is the imaging process of the omnidirectional camera.

[0092] Obtaining a normal image that conforms to human visual habits from an omnidirectional image is the correction of the omnidirectional image, which is the inverse process of the above-mentioned omnidirectional camera imaging. u The inverse transformation gives χ s , the specific steps are:

[0093] m is calculated by the following formula u The inverse transformation gives χ s ,

[0094]

[0095] χ s Image correction can be completed without transforming the unit sphere into normalized plane coordinates.

[0096]

[0097] Figure 2 The image of the city sky taken by the omnidirectional camera before correction is shown. Figure 4 This is the corrected city sky image. From the images before and after correction, we can see that the corrected city sky image is obviously more in line with normal human visual habits, and is also more convenient for subsequent sky detection of images.

[0098] In addition to the omnidirectional image correction method provided in the embodiment of the present application, other omnidirectional image correction methods may also be used, and the present invention does not impose any specific limitation on this.

[0099] S20, performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image.

[0100] like Figure 5 As shown, in this embodiment, the sky area detection is performed on the corrected city sky image and converted back to an omnidirectional image to obtain an omnidirectional sky area image, and the method further includes:

[0101] S201: extracting a sky boundary from the corrected city sky image.

[0102] S202: Mark the sky area from the corrected urban sky image according to the sky boundary, and convert it back into an omnidirectional image to obtain an omnidirectional sky area image.

[0103] Figure 6 In this embodiment, Figure 5 Flowchart of the steps for extracting the sky boundary from a rectified city sky image.

[0104] S2011. Extracting gradient information of the corrected city sky image.

[0105] like Figure 7 As shown, the Sobel operator is used to extract gradient information in the x and y directions of the image and calculate the gradient amplitude of each pixel.

[0106] S2012. Define a sky boundary function border(x), wherein the sky boundary function is:

[0107] 1≤border(x)≤H(1≤x≤W),

[0108] Where W and H represent the width and height of the corrected city sky image, and the sky area and non-sky area are defined as:

[0109] sky={(x,y)|1≤x≤W,1≤y≤border(x)},

[0110] ground={(x,y)|1≤x≤W,border(x)≤y≤H};

[0111] S2013. Determine the improved energy function as:

[0112]

[0113] Among them, Σ s and Σ g represents the covariance matrix of the sky area and the non-sky area represented by RGB values. γ is the parameter of the uniformity of the sky area and its value is 2. and corresponds to the largest eigenvalue of the two matrices, Σ s and Σ g The definition is as follows:

[0114]

[0115]

[0116] Among them, N s and N g Represents the number of pixels in the sky area and the non-sky area, I is the input RGB image, μ s and μg are the average RGB values ​​of the sky area and non-sky area, respectively.

[0117] S2014. Define t as the threshold of the sky energy function border(x), where 5≤t≤600. Given a first threshold, obtain the sky boundary of the corrected urban sky image corresponding to the first threshold, which is the first sky boundary. According to the first sky boundary, obtain the improved energy function value, which is the first improved energy function value.

[0118] Threshold t and J n The relationship between (t) is quite complex and nonlinear. How to select the value of t to maximize the energy function is a complex problem. Therefore, a step search strategy is proposed to determine the value of t.

[0119] S2015. Use a step search strategy with a step size of 5 to obtain the second sky boundary, the third sky boundary, ..., the nth sky boundary, and the corresponding second improved energy function value, the third improved energy function value, ..., the nth improved energy function value.

[0120] S2016. Obtain a maximum improved energy function value, wherein the sky boundary corresponding to the maximum improved energy function value is the sky boundary of the corrected city sky image.

[0121] The maximum improved energy function value is searched from all improved energy function values, and the sky boundary corresponding to the maximum improved energy function value is the final sky boundary of the corrected urban sky image.

[0122] During the image capture process, sometimes very tall buildings are captured in the lens, which leads to the detection of false sky areas in the middle of the image during the sky boundary extraction process. The reason is that the algorithm we proposed implicitly assumes that there are sky area pixels in each column. To detect the false sky area, we propose the diff_abs(x) function, where:

[0123] diff_abs(x)=|border(x+1)-border(x)|(1≤x≤W-1),

[0124] If diff_abs(x)>H / 3, it is determined that a pseudo sky area exists in the sky area corresponding to the sky boundary function.

[0125] The K-means algorithm is used to divide the area with the pseudo sky area into two clusters, and the average vector μ of the two clusters is obtained. s1 and μ s2 , the covariance matrix Σ s1 and Σ s2 , calculate μ s1 and μ g , μs2 and μ g The Mahalanobis distance between them is the cluster with a larger Mahalanobis distance, and the cluster with a smaller Mahalanobis distance is the pseudo-sky region.

[0126] The sky area of ​​the corrected city sky image can be determined based on the detected sky boundary, and the false sky area can be eliminated, so that the real sky area can be accurately detected and the sky area can be marked.

[0127] like Figure 8 The figure shows the sky area detection result of the corrected city sky image. It can be seen that the sky area is accurately detected and most of the tall buildings are not mistakenly detected as the sky area.

[0128] In addition to the sky area detection method given in the embodiment of the present application, a sky area segmentation method can also be used to detect the sky area of ​​a city image, and the present invention does not make any specific limitations on this.

[0129] The corrected city sky image after sky area detection is converted back to an omnidirectional image. The result is as follows: Fig. 9 As shown, the sky area detection result restored to the omnidirectional image.

[0130] S30, acquiring the GNSS signal at the time of collecting the original city sky image, and projecting the satellites in the GNSS signal onto the omnidirectional sky area image.

[0131] The city sky omnidirectional image and the GNSS signal timestamp collected by the receiver are aligned to obtain the GNSS signal received at the time of collecting the original city sky image.

[0132] The pseudorange, altitude and azimuth of the satellite are calculated from the GNSS signal, and the satellite is represented in the ENU coordinate system to obtain a starry sky map at the time of omnidirectional city sky image acquisition.

[0133] In this embodiment, the pseudo range r, altitude angle el and azimuth angle az of the satellite can be calculated from the GNSS signal obtained by the GNSS receiver, and the satellite S can be expressed as (r, el, az). The satellite S is expressed in the ENU coordinate system using the following formula: S e =(x,y,z). The x, y, and z axes of the ENU coordinate system point to the east, north, and up directions respectively.

[0134]

[0135] By projecting the satellites in the star map onto the omnidirectional sky area image using the imaging principle of the omnidirectional camera, the position of each satellite on the omnidirectional sky area image can be obtained, such as Fig.10 shown.

[0136] Before projecting the satellites in the GNSS signal onto the omnidirectional sky area image, it is also necessary to solve the angle between the omnidirectional image and the true north direction so that the star map and the omnidirectional sky area image can be aligned. Fig.11 As shown, in this embodiment, the following scheme is used to solve the angle between the original omnidirectional city sky image and the true north direction:

[0137] S301 , calculating the speed of the camera in the ENU coordinate system from the GNSS signal.

[0138] S302, using north_velocity and east_velocity to calculate the angle between the original city sky image and the true north direction.

[0139] S40, determining whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite.

[0140] like Fig.10 As shown in FIG. 1 , satellites falling within the sky area of ​​the omnidirectional sky area image represent visible satellites, and satellites not falling within the sky area of ​​the omnidirectional sky area image represent invisible satellites. Visible satellites are LOS satellites, and the signals they send are LOS signals. Invisible satellites are NLOS satellites, and the signals they send are NLOS signals.

[0141] S50: Perform positioning through the LOS satellite.

[0142] After eliminating the NLOS satellites at each moment, only LOS satellites are used for positioning. Compared with receiving all signals, the algorithm that eliminates NLOS signals effectively improves positioning accuracy and stability.

[0143] In this embodiment, an omnidirectional camera is used to capture a city sky image, which is then corrected and restored to an omnidirectional image after the sky area is marked. The star map when the omnidirectional city sky image is collected is projected onto the omnidirectional city sky image using the imaging principle of the omnidirectional camera. LOS satellites and NLOS satellites are determined based on the positional relationship between the satellite and the sky area. After eliminating the NLOS satellites, only LOS satellites are used for satellite positioning in urban environments, which can effectively improve the accuracy and stability of satellite positioning.

[0144] like Fig.12 As shown, in one embodiment, the present invention provides a visually assisted navigation system in an urban environment based on an omnidirectional camera, the system comprising:

[0145] Image acquisition module 1: used to acquire original city sky images in the form of omnidirectional images through an omnidirectional camera;

[0146] Image correction module 2: used for performing normalized plane correction on the original city sky image to obtain a corrected city sky image;

[0147] Sky area detection module 3: used to perform sky area detection on the corrected urban sky image and convert it back into an omnidirectional image to obtain an omnidirectional sky area image;

[0148] Satellite projection module 4: used to obtain the GNSS signal at the time of collecting the original city sky image, and project the satellite in the GNSS signal onto the omnidirectional sky area image;

[0149] Satellite determination module 5: used to determine whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite;

[0150] Positioning module 6: used for positioning through the LOS satellite.

[0151] For the specific definition of a visually assisted navigation system in an urban environment based on an omnidirectional camera, please refer to the above-mentioned definition of a visually assisted navigation method in an urban environment based on an omnidirectional camera, which will not be repeated here. A person of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] The visually assisted navigation method, system and device in an urban environment based on an omnidirectional camera in this embodiment are aimed at the problems in the existing navigation methods, such as the low computational efficiency of the NLOS satellite signal removal method, the inability to meet the requirements of the solution accuracy, or the algorithm being extremely dependent on the accuracy and precision of the model. The omnidirectional city sky image taken by the omnidirectional camera is restored to an omnidirectional image after correction and sky area detection, the GNSS signal at the time of collecting the original city sky image is obtained, and the satellite in the GNSS signal is projected onto the omnidirectional sky area image, and the LOS satellite and NLOS satellite are determined according to the positional relationship between the satellite and the sky area. After removing the NLOS satellite and only using the LOS satellite for positioning, not only can the accuracy and stability of positioning be improved, but also, compared with the existing NLOS satellite signal removal method, the computational load is small, it does not need to rely on the accuracy and precision of modeling, and does not require a large amount of data sets for training and learning.

[0153] like Fig.13As shown, a computer device provided by an embodiment of the present invention includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor, and the processor can execute the program instructions stored in the memory to execute the steps of the above-mentioned visual-assisted navigation method in an urban environment based on an omnidirectional camera.

[0154] The memory may include a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories; the processor may be a central processing unit, a microprocessor, an application-specific integrated circuit, a programmable logic device, or a combination thereof. By way of example but not limitation, the programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0155] Additionally, the memory may be a physically separate unit or may be integrated with the processor.

[0156] It can be understood by those skilled in the art that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0157] In one embodiment, a computer-readable storage medium is provided, wherein the storage medium is used to store one or more computer programs, wherein the one or more computer programs include program codes, and when the computer programs are run on a computer, the program codes are used to execute the above-mentioned visually assisted navigation method in an urban environment based on an omnidirectional camera.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD), etc.

[0159] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0160] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A vision-assisted navigation method in an urban environment based on an omnidirectional camera, characterized in that: The following steps are involved: An omnidirectional camera is used to collect an original city sky image in the form of an omnidirectional image, and a normalized plane correction is performed on the original city sky image to obtain a corrected city sky image; Performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image; Acquire the GNSS signal at the time when the original city sky image is collected, and project the satellites in the GNSS signal onto the omnidirectional sky area image; Determine whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite; Positioning is performed through the LOS satellite; The step of performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image includes: extracting a sky boundary from the corrected city sky image; According to the sky boundary, a sky area is marked from the corrected urban sky image, and converted back to an omnidirectional image to obtain an omnidirectional sky area image; The step of extracting a sky boundary from the corrected city sky image comprises: Extracting the corrected city sky image gradient information; Define the sky boundary function border(x), which is: 1≤border(x)≤H, Where 1≤x≤W, W represents the width of the corrected city sky image, and H represents the height of the corrected city sky image. The sky area and non-sky area are defined as: sky={(x,y)|1≤x≤W,1≤y≤border(x)}, ground={(x,y)|1≤x≤W,border(x)≤y≤H}; The improved energy function is determined as: Among them, Σ s and Σ g represents the covariance matrix of the sky area and the non-sky area represented by RGB values. γ is the parameter of the uniformity of the sky area and its value is 2. and λ1 g corresponds to the largest eigenvalue of the two matrices, Σ s and Σ g The definition is as follows: Among them, N s and N g Indicates the number of pixels in the sky area and non-sky area, I s (y,x) is the RGB image input for the sky area, I g (y,x) is the RGB image input for the non-sky area, μ s and μ g The average RGB values ​​of the sky area and the non-sky area respectively; Define t as the threshold of the sky boundary function border(x), where 5≤t≤600, give a first threshold, obtain the sky boundary of the corrected city sky image corresponding to the first threshold, which is the first sky boundary, and obtain an improved energy function value according to the first sky boundary, which is the first improved energy function value; Using a step search strategy with a step size of 5 to obtain a second sky boundary, a third sky boundary, ..., an nth sky boundary, and corresponding second improved energy function values, third improved energy function values, ..., and an nth improved energy function value; A maximum improved energy function value is obtained, wherein the sky boundary corresponding to the maximum improved energy function value is the sky boundary of the corrected city sky image.

2. The visually assisted navigation method in an urban environment based on an omnidirectional camera as claimed in claim 1, characterized in that: The extracting of the sky boundary from the corrected city sky image further comprises detecting a pseudo sky area within the sky boundary of the corrected city sky image, the steps being: Define a diff_abs(x) function, wherein diff_abs(x)=|border(x+1)-border(x)|, Among them, 1≤x≤W-1; If diff_abs(x)>H / 3, it is determined that a pseudo sky area exists in the sky area corresponding to the sky boundary function; The K-means algorithm is used to divide the area with the pseudo sky area into two clusters, and the average vector μ of the two clusters is obtained. s1 and μ s2 , the covariance matrix Σ s1 and Σ s2 , calculate μ s1 and μ g , μ s2 and μ g The Mahalanobis distance between them is the cluster with a larger Mahalanobis distance, and the cluster with a smaller Mahalanobis distance is the pseudo-sky region.

3. The visually assisted navigation method in an urban environment based on an omnidirectional camera as claimed in claim 1, characterized in that: The acquiring of the GNSS signal at the time of collecting the original city sky image and projecting the satellite in the GNSS signal onto the omnidirectional sky area image comprises: Align the timestamps of the original city sky image and the GNSS signal collected by the receiver to obtain the GNSS signal received when the original city sky image was collected; Calculate the pseudorange, altitude and azimuth of the satellite from the GNSS signal, and represent the satellite in the ENU coordinate system to obtain a star map at the time of original city sky image acquisition; The star map is projected onto the omnidirectional sky area image according to the imaging principle of an omnidirectional camera.

4. The visually assisted navigation method in an urban environment based on an omnidirectional camera as claimed in claim 3, characterized in that: Before projecting the star map onto the omnidirectional sky area image according to the omnidirectional camera imaging principle, it is also necessary to solve the angle between the original city sky image and the true north direction. The specific steps are as follows: Calculate the speed of the camera in the ENU coordinate system from the GNSS signal; The angle between the original city sky image and the true north direction is calculated using north_velocity and east_velocity.

5. A visually assisted navigation system in an urban environment based on an omnidirectional camera, characterized in that: The system comprises: Image acquisition module: used to collect original city sky images in the form of omnidirectional images through an omnidirectional camera; Image correction module: used for performing normalized plane correction on the original city sky image to obtain a corrected city sky image; Sky area detection module: used to perform sky area detection on the corrected urban sky image and convert it back into an omnidirectional image to obtain an omnidirectional sky area image; Satellite projection module: used to obtain the GNSS signal at the time of collecting the original city sky image, and project the satellite in the GNSS signal onto the omnidirectional sky area image; Satellite judgment module: used to judge whether the satellite is located in the sky area corresponding to the omnidirectional sky area image, if so, it is a LOS satellite; otherwise, it is a NLOS satellite; Positioning module: used for positioning through the LOS satellite; The step of performing sky area detection on the corrected urban sky image and converting it back into an omnidirectional image to obtain an omnidirectional sky area image includes: extracting a sky boundary from the corrected city sky image; According to the sky boundary, a sky area is marked from the corrected urban sky image, and converted back to an omnidirectional image to obtain an omnidirectional sky area image; The step of extracting a sky boundary from the corrected city sky image comprises: Extracting the corrected city sky image gradient information; Define the sky boundary function border(x), which is: 1≤border(x)≤H, Where 1≤x≤W, W represents the width of the corrected city sky image, and H represents the height of the corrected city sky image. The sky area and non-sky area are defined as: sky={(x,y)|1≤x≤W,1≤y≤border(x)}, ground={(x,y)|1≤x≤W,border(x)≤y≤H}; The improved energy function is determined as: Among them, Σ s and Σ g represents the covariance matrix of the sky area and the non-sky area represented by RGB values. γ is the parameter of the uniformity of the sky area and its value is 2. and λ1 g corresponds to the largest eigenvalue of the two matrices, Σ s and Σ g The definition is as follows: Among them, N s and N g Indicates the number of pixels in the sky area and non-sky area, I s (y,x) is the RGB image input for the sky area, I g (y,x) is the RGB image input for the non-sky area, μ s and μ g The average RGB values ​​of the sky area and the non-sky area respectively; Define t as the threshold of the sky boundary function border(x), where 5≤t≤600, give a first threshold, obtain the sky boundary of the corrected city sky image corresponding to the first threshold, which is the first sky boundary, and obtain an improved energy function value according to the first sky boundary, which is the first improved energy function value; Using a step search strategy with a step size of 5 to obtain a second sky boundary, a third sky boundary, ..., an nth sky boundary, and corresponding second improved energy function values, third improved energy function values, ..., and an nth improved energy function value; A maximum improved energy function value is obtained, wherein the sky boundary corresponding to the maximum improved energy function value is the sky boundary of the corrected city sky image.

6. A computer device, characterized in that: It includes a memory, a processor and a transceiver, which are connected by a bus; the memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor, and the processor can execute the program instructions stored in the memory to execute the visual-assisted navigation method in an urban environment based on an omnidirectional camera as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method for visually assisted navigation in an urban environment based on an omnidirectional camera as claimed in any one of claims 1 to 4 is implemented.

Images