Visual positioning method, device, equipment and medium

Abstract

The invention provides a visual positioning method and device, equipment and a medium, and the method comprises the steps: determining a camera target pose based on camera reference poses corresponding to a plurality of reference images of a target scene, obtaining a prediction image based on the camera target pose, and if the difference between the prediction image and the target reference image meets an image difference condition, determining a visual positioning result. And determining the predicted image as an enhanced image, and performing visual positioning at least based on the enhanced image and a target reference image. According to the method, the enhanced image can be generated based on the original reference image of the target scene, and it is ensured that the enhanced image has no redundancy relative to the reference image, so that the VPS positioning complexity is simplified, and the VPS positioning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual positioning method, device, equipment and medium. Background Art

[0002] Visual Positioning System (VPS) is a system or service that uses image information for positioning. The core of VPS technology is to estimate the six degrees of freedom (6DOF) posture by matching the global descriptor and local descriptor of the image captured by the image acquisition device with the global descriptor in the map data space.

[0003] In related technologies, VPS technology relies on the acquisition quality of scene images. However, it is difficult to fully acquire all observation information of the scene space during the acquisition process. Insufficient acquisition information will affect the subsequent VPS positioning accuracy. Therefore, the scene image can be enhanced to enhance the VPS positioning accuracy. However, the enhanced scene image contains many redundant images, which increases the complexity of VPS positioning and is not conducive to improving the efficiency of VPS positioning. Summary of the invention

[0004] According to one aspect of the present disclosure, a visual positioning method is provided, comprising:

[0005] Determine the camera target pose based on the camera reference poses corresponding to the multiple reference images of the target scene;

[0006] Acquire a predicted image based on the camera target pose;

[0007] If the difference between the predicted image and the target reference image satisfies an image difference condition, it is determined that the predicted image is an enhanced image.

[0008] Visual positioning is performed based at least on the enhanced image and the target reference image.

[0009] According to another aspect of the present disclosure, a visual positioning device is provided, characterized in that it includes:

[0010] A determination module, configured to determine a camera target pose based on a camera reference pose corresponding to a plurality of reference images of a target scene, and determine a predicted image based on the camera target pose;

[0011] A screening module, configured to determine that the predicted image is an enhanced image if a difference between the predicted image and the target reference image satisfies an image difference condition;

[0012] A positioning module is used to perform visual positioning based on at least the enhanced image and the target reference image.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] processor; and,

[0015] Memory for storing programs,

[0016] The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to the exemplary embodiment of the present disclosure.

[0018] One or more technical solutions provided in the exemplary embodiments of the present disclosure can determine the camera target pose based on the camera reference pose corresponding to multiple reference images of the target scene, and then determine the predicted image based on the camera target pose. After that, it can be determined whether the difference between the predicted image and the target reference image meets the image difference condition. If the image difference condition is met, it means that the difference between the predicted image and the target reference image is relatively large and does not belong to a redundant image. Therefore, the predicted image can be confirmed as an enhanced image. It can be seen that the visual positioning method of the exemplary embodiment of the present disclosure can generate an enhanced image based on the original reference image of the target scene, and ensure that the enhanced image has no redundancy relative to the reference image. Therefore, when visual positioning is performed based on at least the enhanced image and the target reference image, not only can the accuracy of visual positioning be improved, but also the complexity of VPS positioning can be simplified and the efficiency of VPS positioning can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram showing a flow chart of a visual positioning method according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 2 A schematic diagram showing the architecture of an image generation model of an exemplary embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram of a characteristic difference determination process of an exemplary embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic block diagram of functional modules of a visual positioning device according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] Before introducing the embodiments of the present disclosure, the following definitions are given for the terms involved in the embodiments of the present disclosure:

[0031] Three-dimensional reconstruction is the reconstruction of three-dimensional objects in the virtual world. In layman's terms, it is the inverse operation of the camera (the camera presents real objects in two-dimensional images, and three-dimensional reconstruction is the display of the information in the two-dimensional image in the three-dimensional virtual space).

[0032] Camera pose is the collective name for the position and attitude of the camera, which is mainly composed of the rotation matrix R and the translation vector t. It describes the transformation relationship between the world coordinate system and the camera coordinate system.

[0033] Spline interpolation is a mathematical method that uses variable splines to create a smooth curve passing through a series of points. The interpolation spline is composed of some polynomials, each of which is determined by two adjacent data points. In this way, any two adjacent polynomials and their derivatives are continuous at the connection point.

[0034] Neural Radiance Fields (NeRF) is a method for generating high-quality 3D reconstruction models. It uses deep learning technology to extract the geometric shape and texture information of objects from images from multiple perspectives, and then uses this information to generate a continuous 3D radiation field, which can present highly realistic 3D models at any angle and distance.

[0035] Information entropy is a basic concept in information theory. It describes the uncertainty of possible events in information sources. In the 1940s, C.E. Shannon borrowed the concept of thermodynamics and called the average amount of information after excluding redundancy "information entropy".

[0036] Reprojection error refers to the difference between the projection and reprojection of a real 3D space point on the image plane, which is used to optimize the camera pose parameters and the coordinates of the 3D space point.

[0037] Pyramid resolution means that the pyramid of an image is a collection of images arranged in a pyramid shape with gradually decreasing resolutions and originating from the same original image.

[0038] Volume rendering is a method of tracing rays and integrating or accumulating them to generate images.

[0039] An exemplary embodiment of the present disclosure provides a visual positioning method, which can obtain a predicted image of a camera target pose based on a camera reference pose of an existing reference image, and then determine whether the predicted image is a redundant image through an image difference condition, so as to reduce redundant images generated by image enhancement, thereby simplifying VPS isotope complexity and improving VPS positioning efficiency.

[0040] The method of the exemplary embodiment of the present disclosure may be executed by an electronic device or a chip applied to the electronic device, and the electronic device may be a user device with a display function, or a computer device without a display function. When the electronic device is a user device, the user device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a PDA, and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology.

[0041] Exemplarily, when the user device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0042] Figure 1 FIG. 2 is a flow chart showing a visual positioning method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the visual positioning method of the exemplary embodiment of the present disclosure may include:

[0043] Step 101: Determine a target camera pose based on camera reference poses corresponding to multiple reference images of a target scene. The camera reference pose corresponding to each reference image may be determined based on each reference image and three-dimensional spatial position information corresponding to the reference image.

[0044] In practical applications, the exemplary embodiments of the present disclosure may determine the camera target pose by interpolation. Taking spline interpolation as an example, the exemplary embodiments of the present disclosure may determine the camera target pose based on the camera reference pose corresponding to multiple reference images of the target scene, and may include: determining the spline curve relationship between the interpolation position and the camera pose based on the camera reference pose corresponding to the multiple reference images, and then determining the camera target pose corresponding to the target interpolation position based on the spline curve relationship between the interpolation position and the camera pose.

[0045] Exemplarily, when determining the spline curve relationship between the interpolation position and the camera pose in the exemplary embodiment of the present disclosure, the camera reference pose corresponding to the multiple reference images can be used as the control point to obtain the spline basis function, and then based on the spline basis function and the camera reference pose corresponding to the multiple reference images, the spline curve relationship is obtained. The spline curve relationship can be expressed as Formula 1:

[0046]

[0047] Where d represents the function degree of the spline curve, i+1 represents the control point number, N represents the total number of control points, c i represents the camera reference pose corresponding to the i+1th control point, t represents the value of the interpolation position, B i,d (t) represents the spline basis function of the i+1th control point, which is a polynomial with respect to t and can be expressed as Equation 2 and Equation 3 using the Cox-de Boor recursive formula.

[0048]

[0049]

[0050] Among them, t k represents the value of the k+1th control point, t k+d-1 Indicates k+d The value of the control point, t k+d represents the value of the k+d+1th control point, t k+1 Represents the value of the k+2th control point.

[0051] After calculating with equations 1 to 3, the spline curve relationship can be obtained, and then the parameters of the target interpolation position are substituted into the spline curve relationship to obtain the camera target pose corresponding to the target interpolation position.

[0052] Step 102: Determine a predicted image based on the camera target pose. In some embodiments, the camera target pose may be used as input to generate a predicted image corresponding to the camera target pose based on a neural radiance field or an image construction technique similar to a neural radiance field.

[0053] For example, the target radiation field features corresponding to the camera target posture may be determined first, and then the target radiation field features may be rendered to obtain a predicted image corresponding to the camera target posture. The exemplary embodiment of the present disclosure may obtain the target radiation field features corresponding to the camera target posture by a multi-layer convolution method, and then render the target radiation field features by a volume rendering method to obtain a predicted image of the camera target posture.

[0054] Exemplarily, the camera target posture of the exemplary embodiment of the present disclosure includes camera target rotation data and camera target translation data, and the target radiation field features include target volume density data and target color data. Here, in the process of determining the target radiation field features corresponding to the camera target posture, the target volume density data and multi-dimensional feature data can be extracted through a multi-layer convolutional network, and considering that the color data is related to the direction, the target color data can be generated based on the multi-dimensional feature data and the camera target rotation data.

[0055] Exemplarily, when the target radiation field characteristics of the exemplary embodiment of the present disclosure include target volume density data and target color data, it is taken into account that the target volume density data is only related to the observation position but not to the observation angle, while the target color data is related not only to the observation position but also to the observation angle. Therefore, volume rendering can be performed in the following manner: first, the target cumulative transparency is determined based on the target volume density and the volume density data included in the radiation field characteristics of the target scene, and then a predicted image is obtained based on the target cumulative transparency, the target color data and the volume density data included in the radiation field characteristics of the target scene.

[0056] The target radiation field characteristics of the exemplary embodiment of the present disclosure include target volume density data and target color data. The target cumulative volume density is obtained through the target volume density data. The volume density suppression function of the target cumulative volume density is used to make the obtained predicted image closer to the actual one. Therefore, the method of the exemplary embodiment of the present disclosure can be applied to image enhancement of target scenes with complex lighting, occlusion effects and infinite details.

[0057] Step 103: If the difference between the predicted image and the target reference image satisfies the image difference condition, the predicted image is determined to be an enhanced image.

[0058] When the difference between the predicted image and the target reference image meets the image difference condition, it means that the difference between the predicted image and the target reference image is relatively large and does not belong to a redundant image. Therefore, the predicted image can be confirmed as an enhanced image. If the difference between the predicted image and the target reference image does not meet the image difference condition, it means that the difference between the predicted image and the target reference image is relatively small and belongs to a redundant image. The predicted image can be deleted. It can be seen that the exemplary embodiment of the present disclosure can filter redundant predicted images through the image difference condition to ensure that the obtained enhanced image has a relatively large difference with the target reference image.

[0059] Step 104: Perform visual positioning based on at least the enhanced image and multiple target reference images. Since the enhanced image and the target reference image are quite different, when performing visual positioning based on the enhanced image and the target reference image, both the enhanced image and the target reference image can be used as the maximum visual positioning reference information for visual positioning, ensuring the accuracy and authenticity of the final visual positioning. Moreover, since the predicted images that are too redundant through the image difference condition can reduce unnecessary calculations in the visual positioning process, the method of the exemplary embodiment of the present disclosure can simplify the complexity of VPS positioning and improve the efficiency of VPS positioning.

[0060] In an optional manner, the image difference condition of the exemplary embodiment of the present disclosure may include: a feature difference between the predicted image and the target reference image is greater than or equal to a preset feature difference.

[0061] Exemplarily, when the feature difference between the predicted image and the target reference image is greater than or equal to the preset feature difference, it means that the feature difference between the predicted image and the target reference image is greater than or equal to the preset feature difference, and relative to the target reference image, the preset image has added new content. Therefore, it can be determined that the preset image is an enhanced image.

[0062] The number of the reference images is at least two, and the target reference images can be one or at least two. Based on this, after obtaining the predicted image, one or more target reference images can be obtained from multiple reference images, and then the difference between the predicted image and the target reference image is compared, and the difference between the two is measured by the image difference condition, so as to determine whether the predicted image is redundant relative to the target reference image of the target scene.

[0063] When there are at least two target reference images, the feature difference between the predicted image and each target reference image can be calculated. If the minimum value of the feature difference is greater than or equal to the preset feature difference, it means that the predicted image is an enhanced image, otherwise the predicted image is a redundant image.

[0064] Considering that there may be a relationship between image redundancy and the magnitude of the change in camera pose, for example, the smaller the change in the camera reference pose, the smaller the change in the reference image taken by the camera may be, while the larger the change in the camera reference pose, the larger the change in the reference image taken by the camera. Therefore, the exemplary embodiment of the present disclosure can determine whether the predicted image is a redundant image relative to the target reference image when the difference between the camera target pose and the camera reference pose corresponding to the target reference image is small.

[0065] For example, the exemplary embodiment of the present disclosure can determine whether the difference between the camera target pose and the camera reference pose corresponding to the reference image is less than a preset pose difference, and divide all reference images into target reference images and non-target reference images. The number of target reference images and non-target reference images can be one or more.

[0066] For the target reference image, the difference between the camera target pose and the camera reference pose corresponding to the target reference image is less than the preset pose difference. In this case, it can be determined whether the feature difference between the predicted image and the first target reference image is greater than or equal to the preset feature difference.

[0067] If the feature difference between the predicted image and the target reference image is greater than or equal to the preset feature difference, it means that although the difference between the camera target pose and the camera reference pose corresponding to the target reference image is relatively small, the predicted image has a larger information increment relative to the target reference image. Therefore, relative to this target reference image, the predicted image is an enhanced image.

[0068] If the feature difference between the predicted image and the target reference image is smaller than the preset feature difference, it means that the difference between the camera target pose and the camera reference pose corresponding to the target reference image is relatively small, and the predicted image has a small information increment or even no information increment relative to the target reference image. Therefore, relative to this target reference image, the predicted image is a redundant image and can be deleted as a redundant image.

[0069] For the non-target reference image, the difference between the camera target pose and the camera reference pose corresponding to the non-target reference image is greater than or equal to the preset pose difference. In this case, the feature difference between the predicted image and the first target reference image is theoretically very large, and the possibility that the predicted image is a redundant image relative to the non-target reference image is extremely low.

[0070] Based on the above analysis, the exemplary embodiments of the present disclosure can perform visual positioning based on the enhanced image, the target reference image and the non-target reference image, thereby further improving the accuracy of visual positioning.

[0071] In an optional manner, the exemplary embodiment of the present disclosure can determine the camera reference pose corresponding to each reference image based on the three-dimensional spatial position information of each reference image and the target scene. This process essentially uses the camera projection model to solve the camera reference pose corresponding to the reference image.

[0072] Exemplarily, the three-dimensional spatial position information of the target scene in the exemplary embodiments of the present disclosure may include spatial point cloud information of the target scene. For example, three-dimensional reconstruction may be performed based on multiple reference images of the target scene to obtain spatial point cloud information matching the two-dimensional image position information of each reference image, or spatial point cloud information may be directly obtained through a laser radar, and then the spatial point cloud information of the target scene and the two-dimensional image position information of the reference image may be matched by 2D-3D features using relevant technologies.

[0073] Exemplarily, when the three-dimensional spatial position information of the target scene may include the three-dimensional spatial coordinates of multiple feature points, the two-dimensional image coordinates of the multiple feature points may be extracted from the reference image. Then, when the camera intrinsic parameters are known, a conversion relationship from the two-dimensional image coordinates of each feature point to the three-dimensional spatial coordinates may be established through a camera projection model as shown in Formula 4. Subsequently, the conversion relationship from the two-dimensional image coordinates of the multiple feature points to the three-dimensional spatial coordinates may be solved to obtain the camera reference pose of the reference image.

[0074] λp=KTP w Formula 4

[0075] Among them, λ represents the scaling factor, K represents the camera intrinsic parameter, T represents the camera reference pose, P w represents the three-dimensional space coordinates of the feature point, and p represents the two-dimensional image coordinates of the feature point.

[0076] The above process of the exemplary embodiment of the present disclosure can be implemented by an image generation model based on the principle of neural radiation field. Figure 2 The architectural diagram of the image generation model shown describes the generation process of the predicted image.

[0077] like Figure 2 As shown, the image generation model 200 of the exemplary embodiment of the present disclosure may include: a volume density extraction module 201, a color extraction module 202 and a volume rendering module 203. It should be understood that the architecture of the image generation model 200 of the exemplary embodiment of the present disclosure may refer to the NeRF model or other image generation models.

[0078] like Figure 2 As shown, the camera target matrix includes the rotation matrix R s Input the opacity extraction module 201 to obtain the target body density data and multi-dimensional feature data. The multi-dimensional feature data and the translation vector t included in the camera target matrix are s The target volume density data and the target color data are input into the color extraction module 202 to obtain the target volume density data and the target color data. After the target volume density data and the target color data are obtained, the two data are input into the volume rendering module 203. The volume rendering module 203 can perform image rendering based on the target volume density data and the target color data to obtain a predicted image.

[0079] In the training phase, the exemplary embodiment of the present disclosure can collect multiple image samples and spatial point cloud information of different camera pose samples in the same scene, and then use the camera pose samples corresponding to the multiple image samples as input and the multiple image samples as supervision information to train the image generation model. When the training is completed, the image generation network has high generalization ability and robustness. Therefore, when a new camera target pose is obtained, the image generation model can also obtain a predicted image close to the real one based on the camera target pose.

[0080] In an optional manner, the feature difference between the predicted image and the target reference image of the exemplary embodiment of the present disclosure may include the information entropy difference between the predicted image and the target reference image, or the reprojection error between the predicted image and the target reference image on the same projection plane, or may also include both the information entropy difference between the predicted image and the target reference image, and the reprojection error between the predicted image and the target reference image on the same projection plane.

[0081] When the feature difference between the predicted image and the target reference image includes the information entropy difference between the predicted image and the target reference image, the information entropy difference between the predicted image and the target reference image can be determined by calculating the information entropy of the predicted image and the information entropy of the target reference image, and then solving the difference between the information entropy of the predicted image and the information entropy of the target reference image. It can be seen that the exemplary embodiments of the present disclosure can judge the feature difference between the predicted image and the target reference image from the perspective of information increment, ensuring that the determined enhanced image has a large increment in image content relative to the target reference image.

[0082] When the feature difference between the predicted image and the target reference image of the exemplary embodiment of the present disclosure includes the reprojection error including the reprojection error of the predicted image and the target reference image on the same projection plane, the exemplary embodiment of the present disclosure may project the predicted image and the target reference image to the same projection plane through a camera projection model, and then determine the projection error between the two, thereby determining the feature difference between the predicted image and the target reference image.

[0083] When the feature difference between the predicted image and the target reference image includes the information entropy difference between the predicted image and the target reference image, and the reprojection error between the predicted image and the target reference image on the same projection plane, the method of the exemplary embodiment of the present disclosure actually comprehensively judges the feature difference between the predicted image and the target reference image from the two perspectives of information increment and camera projection, which can improve the accuracy of judging the feature difference between the predicted image and the target reference image.

[0084] The target reference image of the exemplary embodiment of the present disclosure may include two-dimensional position measurement information of multiple second reference feature points, and the predicted image may include two-dimensional position measurement information of multiple second reference feature points. There is a corresponding relationship between the first reference feature point and the second reference feature point, that is, the first reference feature point and the corresponding second reference feature point may correspond to the same three-dimensional space target.

[0085] For the first reference feature points, the multiple first reference feature points can be evenly or randomly distributed on the target reference image, or the distribution method of the first reference feature points on the target reference image can be determined based on the extraction method of the first reference feature points. For example, the two-dimensional position measurement information of the multiple first reference feature points can be obtained from the target reference image through image feature point detection algorithms such as Harris corner detection and SIFT algorithm.

[0086] For the second reference feature points, the plurality of second reference feature points may be evenly or randomly distributed on the predicted image, or the distribution of the second reference feature points on the predicted image may be determined based on the extraction method of the second reference feature points. For example, the two-dimensional position measurement information of the plurality of second reference feature points may be obtained from the predicted image through image feature point detection algorithms such as Harris corner detection and SIFT algorithm.

[0087] Figure 3 FIG. 2 shows a schematic diagram of a characteristic difference determination process of an exemplary embodiment of the present disclosure. Figure 3 As shown, the method of the exemplary embodiment of the present disclosure also includes:

[0088] Step 301: Based on the two-dimensional position measurement information of each first reference feature point and the corresponding camera reference pose, determine the three-dimensional position information of each first reference feature point. Here, you can refer to the relevant description above, and obtain the three-dimensional position information of each first reference feature point in combination with the camera intrinsic parameters of the captured target reference image.

[0089] Step 302: Based on the three-dimensional position information of each first reference feature point and the camera target pose, determine the two-dimensional position prediction information of each first reference feature point. This process is essentially the process of obtaining the projection information of multiple first two-dimensional position measurement information in the predicted image with reference to the relevant description above. For example, the two-dimensional position prediction information of each first reference feature point can be obtained based on the three-dimensional position information of each first reference feature point, the camera target pose and the camera internal parameters.

[0090] Step 303: Determine the feature difference between the predicted image and the target reference image based on the two-dimensional position measurement information of the plurality of second reference feature points and the two-dimensional position prediction information of the corresponding first reference feature points.

[0091] The process of determining the feature difference between the above-mentioned predicted image and the target reference image may include: based on the two-dimensional position measurement information of each second reference feature point and the two-dimensional position prediction information of the corresponding first reference feature point, the projection error of the first reference feature point and the corresponding second reference feature point on the plane where the predicted image is located can be obtained, and then the feature difference between the predicted image and the target reference image is determined based on the projection error of multiple first reference feature points and the corresponding second reference feature points on the plane where the predicted image is located. For example: the exemplary embodiment of the present disclosure can determine the feature difference between the predicted image and the target reference image by the feature difference formula shown in Formula 5, and then determine whether the predicted image is a redundant image in combination with the threshold judgment method.

[0092] h(k)=3(1+log(2π))+0.5 log(|Σ|) Formula 5

[0093] |Σ| represents the determinant calculation result formed by the projection errors of multiple first reference feature points and the corresponding second reference feature points on the plane where the predicted image is located, and h(k) represents the feature difference between the predicted image and the target reference image.

[0094] It can be seen that the projection error between the first reference feature point and the corresponding second reference feature point in the plane where the predicted image is located in the exemplary embodiment of the present disclosure is determined by the two-dimensional position measurement information of the second reference feature point and the two-dimensional position prediction information corresponding to the first reference feature point (hereinafter referred to as the reprojection error of the reference feature point), and the two-dimensional position prediction information of the first reference feature point is determined based on the two-dimensional position measurement information of the first reference feature point. Therefore, the method of the exemplary embodiment of the present disclosure can use the principle of reprojection error to calculate the feature difference between the target reference image and the predicted image.

[0095] Since the reprojection error of the reference feature points in the local area cannot fully reflect the reprojection error of the reference feature points in all areas of the predicted image and the target reference image, the accuracy of the feature difference between the predicted image and the target reference image determined based on the reprojection error of the reference feature points in the local area is not high. In this case, when the feature difference between the predicted image and the target reference image is greater than or equal to the preset feature difference, the predicted image may be mistaken for the enhanced image, and when the feature difference between the predicted image and the target reference image is less than the preset feature difference, the predicted image may be mistaken for a redundant image.

[0096] In order to overcome the above problems, the multiple first reference feature points of the exemplary embodiment of the present disclosure correspond to different areas of the target reference image, and the multiple first reference feature points correspond to different areas of the predicted image. The reprojection errors of the multiple reference feature points can fully reflect the reprojection errors of the reference feature points of each area of ​​the predicted image and the target reference image. Therefore, based on the reprojection errors of the multiple reference feature points, the feature difference between the predicted image and the target reference image can be determined more accurately, reducing the occurrence of misjudgment of the predicted image.

[0097] One or more technical solutions provided in the exemplary embodiments of the present disclosure can determine the camera target pose based on the camera reference pose corresponding to multiple reference images of the target scene, and then determine the predicted image based on the camera target pose. After that, it can be determined whether the difference between the predicted image and the target reference image meets the image difference condition. If the image difference condition is met, it means that the difference between the predicted image and the target reference image is relatively large and does not belong to a redundant image. Therefore, the predicted image can be confirmed as an enhanced image. It can be seen that the visual positioning method of the exemplary embodiment of the present disclosure can generate an enhanced image based on the original reference image of the target scene, and ensure that the enhanced image has no redundancy relative to the reference image. Therefore, when visual positioning is performed based on at least the enhanced image and the target reference image, not only can the accuracy of visual positioning be improved, but also the complexity of VPS positioning can be simplified and the efficiency of VPS positioning can be improved.

[0098] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of an electronic device. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0099] The disclosed embodiment can divide the electronic device into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0100] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a visual positioning device, which may be a server or a chip applied to a server. Figure 4 FIG. 1 shows a schematic block diagram of functional modules of a visual positioning device according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the visual positioning device 400 includes:

[0101] A determination module 401 is used to determine a camera target pose based on a camera reference pose corresponding to a plurality of reference images of a target scene, and to obtain a predicted image based on the camera target pose;

[0102] The screening module 402 is configured to determine that the predicted image is an enhanced image if the difference between the predicted image and the target reference image satisfies an image difference condition.

[0103] The positioning module 403 is used to perform visual positioning based on at least the enhanced image and the target reference image.

[0104] In a possible implementation, the determination module 401 is further configured to determine a camera reference pose corresponding to each of the reference images based on the three-dimensional spatial position information of each of the reference images and the target scene.

[0105] In a possible implementation, the determination module 401 is used to determine a spline curve relationship between an interpolation position and a camera pose based on camera reference poses corresponding to a plurality of reference images, and to determine a camera target pose corresponding to a target interpolation position based on the spline curve relationship between the interpolation position and the camera pose.

[0106] In a possible implementation, the determination module 401 is used to determine the target radiation field characteristics corresponding to the camera target posture, render the target radiation field characteristics, and obtain a predicted image corresponding to the camera target posture.

[0107] In a possible implementation, the camera target posture includes camera target rotation data and camera target translation data, and the target radiation field characteristics include target volume density data and target color data.

[0108] In a possible implementation, the determination module 401 is used to determine the target volume density data and the multi-dimensional feature data based on the camera target translation data, and obtain the target color data based on the multi-dimensional feature data and the camera target rotation data.

[0109] In one possible implementation, the target radiation field characteristics include target volume density data and target color data, and the prediction module 402 is used to determine the target cumulative transparency based on the target volume density and the volume density data included in the radiation field characteristics of the target scene, and obtain the predicted image based on the target cumulative transparency, the target color data, and the volume density data included in the radiation field characteristics of the target scene.

[0110] In a possible implementation manner, the image difference condition includes:

[0111] A feature difference between the predicted image and the target reference image is greater than or equal to a preset feature difference, and a difference between the camera target pose and a camera reference pose corresponding to the target reference image is less than a preset pose difference.

[0112] In a possible implementation manner, a difference between the camera target pose and a camera reference pose corresponding to the target reference image is smaller than a preset pose difference.

[0113] In a possible implementation, the feature difference between the predicted image and the target reference image includes an information entropy difference between the predicted image and the target reference image, and / or a reprojection error between the predicted image and the target reference image on the same projection plane.

[0114] In a possible implementation, the target reference image includes two-dimensional position measurement information of multiple first reference feature points, the predicted image includes two-dimensional position measurement information of multiple second reference feature points, the first reference feature points correspond to the second reference feature points, and the determination module 401 is further used to determine the three-dimensional position information of each first reference feature point based on the two-dimensional position measurement information of each first reference feature point and the corresponding camera reference pose; determine the two-dimensional position prediction information of each first reference feature point based on the three-dimensional position information of each first reference feature point and the camera target pose; determine the feature difference between the predicted image and the target reference image based on the two-dimensional position measurement information of multiple second reference feature points and the two-dimensional position prediction information corresponding to the first reference feature point. .

[0115] Figure 5 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 5 As shown, the chip 500 includes one or more (including two) processors 501 and a communication interface 502. The communication interface 502 can support the electronic device to perform the data receiving and sending steps in the above method, and the processor 501 can support the electronic device to perform the data processing steps in the above method.

[0116] Optional, such as Figure 5 As shown, the chip 500 also includes a memory 503, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0117] In some embodiments, Figure 5 As shown, the processor 501 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 501 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 503 may include a read-only memory and a random access memory, and provides instructions and data to the processor 501. A portion of the memory 503 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 5 Various buses are labeled as bus system 504 .

[0118] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0119] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0120] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0121] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0122] refer to Figure 6 , a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0123] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 606 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0124] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 606, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0125] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiment of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 606. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the method of the exemplary embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0126] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0132] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A visual positioning method, characterized in that: include: Determine the camera target pose based on the camera reference poses corresponding to the multiple reference images of the target scene; Acquire a predicted image based on the camera target pose; If the difference between the predicted image and the target reference image satisfies an image difference condition, determining that the predicted image is an enhanced image; Visual positioning is performed based at least on the enhanced image and the target reference image.

2. The method according to claim 1, characterized in that The method further comprises: Based on each of the reference images and the three-dimensional spatial position information of the target scene, a camera reference pose corresponding to each of the reference images is determined.

3. The method according to claim 1, characterized in that The determining of the camera target pose based on the camera reference poses corresponding to the multiple reference images of the target scene includes: Based on the camera reference poses corresponding to the multiple reference images, determine the spline curve relationship between the interpolation position and the camera pose; The camera target pose corresponding to the target interpolation position is determined based on the spline curve relationship between the interpolation position and the camera pose.

4. The method according to claim 1, characterized in that The step of determining a predicted image based on the camera target pose comprises: Determining target radiation field characteristics corresponding to the camera target posture; The target radiation field features are rendered to obtain a predicted image corresponding to the camera target posture.

5. The method according to claim 4, characterized in that The camera target posture includes camera target rotation data and camera target translation data, the target radiation field feature includes target body density data and target color data, and determining the radiation field feature corresponding to the camera target posture based on the camera target posture includes: Determine the target body density data and multi-dimensional feature data based on the camera target translation data; The target color data is obtained based on the multi-dimensional feature data and the camera target rotation data.

6. The method according to claim 4, characterized in that The target radiation field feature includes target body density data and target color data, and the predicted image of the camera target pose is rendered based on the target radiation field feature, including: Determining target cumulative transparency based on the target volume density and volume density data included in the radiation field characteristics of the target scene; The predicted image is obtained based on the target cumulative transparency, the target color data and the volume density data included in the radiation field characteristics of the target scene.

7. The method according to any one of claims 1 to 6, characterized in that: The image difference condition includes: a feature difference between the predicted image and the target reference image is greater than or equal to a preset feature difference.

8. The method according to claim 7, characterized in that The difference between the camera target pose and the camera reference pose corresponding to the target reference image is less than a preset pose difference.

9. The method according to claim 7, characterized in that: The feature difference between the predicted image and the target reference image includes an information entropy difference between the predicted image and the target reference image, and / or a reprojection error between the predicted image and the target reference image on the same projection plane.

10. The method according to claim 7, characterized in that The target reference image includes two-dimensional position measurement information of a plurality of first reference feature points, the predicted image includes two-dimensional position measurement information of a plurality of second reference feature points, the first reference feature points correspond to the second reference feature points, and the method further includes: Determine the three-dimensional position information of each of the first reference feature points based on the two-dimensional position measurement information of each of the first reference feature points and the corresponding camera reference pose; Determine two-dimensional position prediction information of each first reference feature point based on the three-dimensional position information of each first reference feature point and the camera target pose; Based on the two-dimensional position measurement information of a plurality of the second reference feature points and the two-dimensional position prediction information corresponding to the first reference feature points, a feature difference between the predicted image and the target reference image is determined.

11. A visual positioning device, characterized in that: include: A determination module, configured to determine a camera target pose based on a camera reference pose corresponding to a plurality of reference images of a target scene, and to obtain a predicted image based on the camera target pose; A screening module, configured to determine that the predicted image is an enhanced image if a difference between the predicted image and the target reference image satisfies an image difference condition; A positioning module is used to perform visual positioning based on at least the enhanced image and the target reference image.

12. An electronic device, characterized in that: include: processor; as well as, Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the method according to any one of claims 1 to 10.

Images