Cross-view six-degree-of-freedom positioning method and device, computer device and storage medium

Abstract

This application relates to a cross-view six-DOF localization method, apparatus, computer device, and storage medium. The method includes: inputting prior information obtained from sensors into a priori preprocessing module for preprocessing to obtain an initial state; inputting the initial state, a 3D texture model, and a given query image into a coarse localization module to obtain a coarse localization pose; inputting the coarse localization pose and the query image into a fine localization module to obtain a cross-view six-DOF localization result; the fine localization module is similar to the coarse localization module, except that it generates a synthetic image based on the same intrinsic parameters and coarse pose as the given query image. This method is used for accurate cross-view localization in large-scale scenes; a novel coarse localization module is introduced, utilizing a synthetic panoramic image to assist localization, achieving a robust and fast localization process.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to a cross-view... Figure 6 Methods, apparatus, computer equipment, and storage media for positioning with degrees of freedom. Background Technology

[0002] Visual localization is a fundamental task in computer vision, crucial for various applications such as augmented reality, autonomous driving, and virtual reality. The goal of visual localization is to estimate the 6-DOF pose of a camera relative to a known map or reference image. This can be achieved using various techniques, such as feature detection and matching, structure-of-motion, and perspective n-point (PnP) solvers. Traditional large-scale ground image 6-DOF estimation for visual localization typically requires both the ground image and a textured 3D model based on it, the latter being very time-consuming to construct.

[0003] To avoid this situation, some researchers have proposed cross-view positioning methods, which utilize bird's-eye views for ground positioning. Unfortunately, due to the lack of ground maps, these methods can only perform 3-DoF positioning, that is, positioning based on xy coordinates and azimuth.

[0004] Meanwhile, with the rapid development of 3D reconstruction technology, constructing city-level digital 3D models using aerial oblique photogrammetry has become a reality. However, the results of using aerial 3D models and traditional visual positioning algorithms for six-degree-of-freedom localization of ground images are unsatisfactory. This is because there are significant differences in the viewing angles between ground images and aerial images, making the matching process extremely challenging. Summary of the Invention

[0005] Therefore, it is necessary to provide a cross-view solution to address the aforementioned technical problems. Figure 6 Methods, apparatus, computer equipment, and storage media for positioning with degrees of freedom.

[0006] A cross-vision Figure 6 A method for locating degrees of freedom, which includes:

[0007] The prior information obtained from the sensor is input into the prior preprocessing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw angle.

[0008] The initial state, 3D texture model, and given query image are input into the coarse localization module to obtain the coarse localization pose. The coarse localization module is used to draw a synthetic panoramic image based on the initial state and 3D texture model, and to perform template matching based on the query image and the synthetic panoramic image to obtain the refined pose. Based on the refined pose, a synthetic image with a field of view of 90° and a depth map are generated. Then, the synthetic image and the query image are matched and the PnP algorithm is used to obtain the coarse localization pose.

[0009] The coarse localization pose and the query image are input into the fine localization module to obtain cross-view results. Figure 6 The results of the degree-of-freedom localization are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image based on the same intrinsic parameters and coarse pose as the given query image.

[0010] In one embodiment, prior information obtained from the sensor is input into a priori preprocessing module for preprocessing to obtain an initial state, including:

[0011] The prior information obtained from the sensor will be input into the prior preprocessing module. If it is a special location point, the location point will be close to the center of the road. Special location points include: the middle or nearby of buildings and bushes.

[0012] The height of the location points is adjusted using a digital surface model to obtain the initial state.

[0013] In one embodiment, the coarse localization module includes: a synthetic image generation module, a refined initial pose solver, and a coarse pose solver.

[0014] The initial state, 3D texture model, and given query image are input into the coarse localization module to obtain the coarse localization pose, including:

[0015] The initial state and 3D texture model are input into the synthetic image generation module to generate a synthetic panoramic image.

[0016] The synthesized panoramic image and the given query image are input into the thinning initial pose solver to obtain the thinned pose.

[0017] The refined pose is input into the coarse pose solver to obtain the coarse localization pose.

[0018] In one embodiment, the refined initial pose solver includes: a template matching module, a QATM module, and an NMS module; the template matching module includes two structurally identical feature extraction networks sharing parameters.

[0019] The synthesized panoramic image and the given query image are input into the thinning initial pose solver to obtain the thinned pose, including:

[0020] The given query image and the synthesized panoramic image are input into two feature extraction networks respectively to obtain query image features and panoramic image features.

[0021] Input the query image features and panoramic image features into the QATM module to calculate the matching quality score.

[0022] The matching quality score is input into the NMS module to obtain the refined pose.

[0023] In one embodiment, the query image features and panoramic image features are input into the QATM module, and the matching quality score is calculated as follows:

[0024] Q map ←Max(QATM|P)

[0025] QATM(p,q)=L(q|p)·L(p|q)

[0026]

[0027]

[0028] Among them, Q map For the highest matching quality score, QATM(p,q) is the product of the likelihood function value of p matching in the query graph Q and the likelihood function value of q matching in the template graph P, where q is the query patch, p is the template patch, L(q|p) is the probability that the template patch p will be matched given the query patch q, L(p|q) is the probability that the template patch p will be matched given the template patch p, p' is the remaining sub-patches, P is the template graph, and F Q [x,y] and F P [i,j] represent query patch features and panoramic patch features, respectively. α is a hyperparameter, and Dist(·) is a predefined similarity calculation function.

[0029] In one embodiment, the matching quality score is input into the NMS module to obtain the refined pose, including:

[0030] Using the maximum response value in the matching quality score, NMS is used to obtain detection boxes that may contain the target object.

[0031] Let C be the center of the detection box, 0 degrees be the center of the panoramic image, and ±180 degrees be the left and right sides respectively; using C as the target point, calculate the optimized yaw value:

[0032]

[0033] Among them W p C is the width of the panoramic image. x The x-coordinate of the center detection box.

[0034] The initial pose is updated based on the optimized yaw value to obtain the refined pose.

[0035] In one embodiment, the refined pose and the synthesized panoramic image are input into a coarse pose solver to obtain a coarse localization pose, including:

[0036] Based on the refined pose, a synthetic image and a depth map with a field of view of 90° are generated.

[0037] The query image and the information are combined into an image and matched to generate matching pairs.

[0038] Based on the matching pairs, intrinsic parameters, and depth map, determine the correspondence between 2D and 3D points in the query image.

[0039] The coarse localization pose is obtained by using a PnP solver based on the correspondence.

[0040] A cross-vision Figure 6 A degree-of-freedom positioning device, comprising:

[0041] The prior preprocessor is used to input the prior information obtained from the sensor into the prior preprocessing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw.

[0042] The coarse localizer is used to input the initial state, 3D texture model, and given query image into the coarse localization module to obtain the coarse localization pose. The coarse localization module is used to draw a synthetic panoramic image based on the initial state and 3D texture model, and perform template matching based on the query image and the synthetic panoramic image to obtain the refined pose. Based on the refined pose, a synthetic image with a field of view of 90 degrees and a depth map are generated. Then, the synthetic image and the query image are matched and the PnP algorithm is used to obtain the coarse localization pose.

[0043] The fine locator is used to input the coarse localization pose and the query image into the fine localization module to obtain cross-view results. Figure 6 The results of the degree-of-freedom localization are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image using the same intrinsic parameters and coarse pose as the given query image.

[0044] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0046] The above cross-view Figure 6A method, apparatus, computer device, and storage medium for degree-of-freedom localization, the method comprising: inputting prior information obtained from sensors into a prior preprocessing module for preprocessing to obtain an initial state; the prior information includes latitude, longitude, altitude, and yaw; inputting the initial state, a 3D texture model, and a given query image into a coarse localization module to obtain a coarse localization pose; the coarse localization module is used to draw a synthetic panoramic image based on the initial state and the 3D texture model, and to perform template matching between the query image and the synthetic panoramic image to obtain a refined pose; based on the refined pose, generating a synthetic image with a field of view of 90° and a depth map, and then matching the synthetic image and the query image using the PnP algorithm to obtain the coarse localization pose; inputting the coarse localization pose and the query image into a fine localization module to obtain a cross-field-of-view pose. Figure 6 The localization results for degrees of freedom are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image based on the same intrinsic parameters and coarse pose as the given query image. This method is used for accurate localization in large-scale scenes with extreme appearance differences; a novel coarse localization module is introduced, which uses a synthetic panoramic image to assist localization, achieving a robust and fast localization process. Attached Figure Description

[0047] Figure 1 Cross-view in one embodiment Figure 6 The basic process of the degree-of-freedom positioning method;

[0048] Figure 2 Cross-view in one embodiment Figure 6 A flowchart illustrating the degree-of-freedom positioning method;

[0049] Figure 3 This is a schematic diagram of the workflow of the prior preprocessing module in one embodiment;

[0050] Figure 4 In another embodiment, cross-view Figure 6 A flowchart illustrating the steps of a degree-of-freedom positioning framework;

[0051] Figure 5 This is the template matching process in another embodiment;

[0052] Figure 6 This is a schematic diagram of the template matching algorithm in another embodiment;

[0053] Figure 7 Cross-view in one embodiment Figure 6 Structural block diagram of a degree-of-freedom positioning device;

[0054] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The significant differences in viewpoint between ground and aerial images make the matching process extremely challenging. Therefore, this application addresses these challenges by achieving cross-view, six-DOF localization. To accomplish this task, this application proposes a cross-view localization method assisted by synthetic panoramic images. Figure 6 The Six Degrees of Freedom Localization (CLaSP) method explores the possibility of ground-based six-degrees-of-freedom localization using aerial databases in large-scale scenarios. Inspired by recent research in view synthesis, the core idea of ​​the proposed model is to recover pose from coarse to fine using a model framework. (Cross-view...) Figure 6 The basic process of the degree-of-freedom positioning method is as follows: Figure 1 As shown. Specifically, given prior information from device sensors (i.e., GPS, compass, and gravity direction), this method uses a semantic segmentation map to optimize the prior information to obtain an initial pose. To obtain a wider continuous search space, the model generates a synthetic panorama from the initial pose using a textured mesh model, followed by a refined pose generated through template matching. The refined pose is used to generate a synthetic perspective image and a depth map. Then, based on pixel projection relationships, 2D-2D matching is converted to 2D-3D matching, and the coarse result is recovered through PnP RANSAC. Finally, fine localization repeats the above process, but based on different intrinsic parameters to obtain the final pose result.

[0057] This method is a cross-view coarse-to-fine localization framework that leverages synthetic panoramic images to facilitate cross-view localization in large-scale scenes. Figure 6 -DoF localization. Compared to ordinary images, synthetic panoramas can obtain a wider and more continuous search space, thus achieving robust coarse pose. We then formulate the refined localization process as dense matching and pose refinement to obtain the final result.

[0058] In one embodiment, such as Figure 2 As shown, a cross-view Figure 6 The method for locating degrees of freedom includes the following steps:

[0059] Step 100: Input the prior information obtained from the sensor into the prior preprocessing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw.

[0060] Specifically, the Prior Preprocessing (PPM) module processes prior information from the device's sensor logs to generate the prior pose T0. It primarily refines the geographic coordinates, namely latitude (lat), longitude (lon), and altitude (alt).

[0061] Latitude, longitude, altitude, and yaw are acquired from sensors and used as prior information in subsequent processes. Furthermore, pitch and roll values ​​are set to 90° and 0° respectively to ensure a normal composite panorama. This means the method only uses 4-DoF prior information as input. It's easy to notice that a normal device's GPS signal is sometimes unstable, with errors of several meters. Such errors seem acceptable in navigation, but are unacceptable for precise ground positioning. For example, if the image is captured next to a building, tree, or cliff, tiny changes in latitude or length coordinates can be rendered as if the image is on the facade of a building, above a tree, or below a cliff. Therefore, it's necessary to handle this situation beforehand.

[0062] Due to the confined space, directly using prior information from sensors often leads to erroneous pose results. To address this issue, existing research proposes a method for randomly initializing prior poses, but this method is rigid and time-consuming. This application employs a smarter module to obtain robust coarse poses. We optimize latitude, longitude, and altitude, and the module uses semantic information to pull illogical points (e.g., points inside buildings and below cliffs) to the center of the road. Furthermore, we utilize synthetic panoramas and template matching methods to adjust the rotation matrix to better values. Subsequent experiments show that this method produces robust coarse results and requires less time than the random seed method.

[0063] Step 102: Input the initial state, 3D texture model, and given query image into the coarse localization module to obtain the coarse localization pose; the coarse localization module is used to draw a synthetic panoramic image based on the initial state and 3D texture model, and perform template matching based on the query image and synthetic panoramic image to obtain the refined pose. Based on the refined pose, a synthetic image with a field of view of 90 and a depth map are generated. Then, the synthetic image and the query image are matched and the PnP algorithm is used to obtain the coarse localization pose.

[0064] Specifically, orientation values ​​obtained from sensors are often inaccurate, and if directly used for rendering and matching, the final result may be inaccurate. This is especially true for ground image localization; even adjusting the x, y, and z coordinates can significantly affect the final result due to orientation differences. However, the random seed method is a time-consuming pose localization process with poor robustness.

[0065] The coarse localization (CLM) module is used to obtain a wider continuous search space and robust coarse pose results T.c .

[0066] Step 104: Input the coarse localization pose and query image into the fine localization module to obtain cross-view... Figure 6 The results of the degree-of-freedom localization are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image based on the same intrinsic parameters and coarse pose as the given query image.

[0067] Specifically, coarse localization can achieve approximate localization, but it cannot yield precise 6-DoF pose results. Therefore, it is necessary to further optimize fine localization.

[0068] In the coarse localization module, the solver uses a 90-FoV reference image. Due to inaccurate initial poses, the model needs to present the widest possible view to increase the likelihood of having the same objects as the query image. However, when we set large view parameters for rendering, the scale difference between the synthesized image and the query image can be very large, leading to matching errors.

[0069] To overcome this problem, after obtaining the coarse positioning pose T c Then, the composite image I is rendered using the same intrinsic parameters as the query image. k This ensures the same scale and better matching results. The final result T is then obtained using the matching algorithm and the PnP solver. f .

[0070] The prior preprocessing module, coarse localization module, and fine localization module constitute the cross-view module. Figure 6 Degrees of freedom positioning frame (abbreviated as: CLaSP frame).

[0071] The above cross-view Figure 6 In the degree-of-freedom localization method, the method includes: inputting prior information obtained from sensors into a prior preprocessing module for preprocessing to obtain an initial state; the prior information includes latitude, longitude, altitude, and yaw; inputting the initial state, 3D texture model, and a given query image into a coarse localization module to obtain a coarse localization pose; the coarse localization module is used to draw a synthetic panoramic image based on the initial state and 3D texture model, and to perform template matching based on the query image and the synthetic panoramic image to obtain a refined pose; based on the refined pose, generating a synthetic image with a field of view of 90° and a depth map; then matching the synthetic image and the query image using the PnP algorithm to obtain the coarse localization pose; inputting the coarse localization pose and the query image into a fine localization module to obtain a cross-field-of-view pose. Figure 6The localization results for degrees of freedom are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image based on the same intrinsic parameters and coarse pose as the given query image. This method is used for accurate localization in large-scale scenes with extreme appearance differences; a novel coarse localization module is introduced, which uses a synthetic panoramic image to assist localization, achieving a robust and fast localization process.

[0072] In one embodiment, step 100 includes: inputting prior information obtained from the sensor into a priori preprocessing module; if it is a special location point, then moving the location point closer to the center of the road; special location points include: points in or near buildings or bushes; adjusting the height of the location point using a digital surface model to obtain an initial state.

[0073] Specifically, based on common sense, outdoor street images are typically taken on roads. Therefore, the purpose of the prior preprocessing module (PPM) is to bring location points (e.g., points between buildings and bushes) closer to the center of the road. It then adjusts the height using a digital surface model (DSM). The PPM workflow is as follows: Figure 3 As shown, semantic segmentation maps were manually annotated from orthophotos in the Airloc dataset for this purpose. Roads, buildings, trees, and other objects were annotated. Figure 3 This module uses semantic segmentation information and DSM to optimize locations marked with "×" to locations marked with "○". The dashed line in the second image above represents the fitted centerline of the road.

[0074] After manual labeling, roads are manually extracted and fitted as straight lines R0, R0, ... R m At the center. As shown in Algorithm 1. PPM looks at the prior coordinates of each location, and if the coordinates are inside or near a building / tree, it forces them to be moved to the center of the nearest road. This step adjusts the latitude and altitude coordinates, but in most cases, the alt coordinates are often inaccurate. Furthermore, after obtaining the geographic coordinates, PPM adjusts the alt coordinates using the DSM of the texture model. Based on the DSM, the altitude of each site is increased by 1.5 meters to simulate the height of a person holding a device.

[0075] Algorithm 1: Geographic Coordinate Optimization Algorithm

[0076] Input: Initial geographic coordinates: T 0 ,T 1 ,…,T n ;

[0077] Road semantic information: R0, R1, ..., R m ;

[0078] Semantic segmentation graph: MAP;

[0079] Digital Surface Model: DSM;

[0080] Output: Optimized geographic coordinates: T0;

[0081] 1: for all T i ∈T do / / For each T in T i implement;

[0082] 2if T i If it is near or within the semantic area of ​​buildings or trees, then:

[0083] 3: T i Drag to the nearest road R and calculate T. i Perpendicular to the road segment

[0084] T i Drag to the nearest road R and calculate T. i Perpendicular to the road segment

[0085] 4:else

[0086] 5:

[0087] 6:end if

[0088] 7: According to Update elevation values ​​in conjunction with DSM information;

[0089] 9: Return to T0

[0090] In one embodiment, the coarse localization module includes: a synthetic image generation module, a refined initial pose solver, and a coarse pose solver; step 102 includes the following steps:

[0091] Step 200: Input the initial state and 3D texture model into the composite image generation module to generate a composite panoramic image.

[0092] Specifically, the compositing engine renders composite images from a specific viewpoint based on existing texture models. Blender is a free, open-source 3D graphics software with a built-in Python development environment and equipped with the Cycles and EEVEE compositing engines. We wrote scripts in the Python environment, employing both panoramic and perspective compositing modes.

[0093] Panoramic Compositing: Using the texture model and pose T0 as input, Blender is set to Cycles, and the rendering mode is defined as Panoramic. The output will be a composite panoramic image. Lighting conditions need to be considered in this step; otherwise, the composite image will be very dark.

[0094] Viewpoint synthesis: using texture models, intrinsic parameters, and pose T0' / T c The input is set to EEVEE, and the rendering mode is set to perspective. The output is a composite perspective image. The intrinsic parameter matrix needs to be set in advance, and differences in the intrinsic parameters will significantly affect the image content. In this framework, different parameters are selected for each process to achieve the best results.

[0095] Step 202: Input the synthesized panoramic image and the given query image into the thinning initial pose solver to obtain the thinned pose.

[0096] Specifically, the Refine Initial Pose Solver (RIP) takes the query image and panoramic image as input, and uses the idea of ​​template matching algorithm to optimize the yaw value to obtain the refined pose T0'.

[0097] Step 204: Input the refined pose and the synthesized panoramic image into the coarse pose solver to obtain the coarse localization pose.

[0098] Specifically, the coarse pose solver (CPS) generates a synthetic image with a 90° field of view as a reference based on the refined pose T0'. This image contains more information without distortion. Finally, the CPS obtains the coarse localization pose T by matching the query image and the reference image and applying the PnP algorithm within LO-RANSAC. c Template matching is essentially a basic pattern recognition method where specific object patterns are located in an image, and then the objects are located. The specific process of this method is as follows: Figure 4 As shown, the template matching process is like finding a smaller object in a larger image; first, the query image I is used... Q For template image I p A full scan is performed, and high-confidence bounding boxes are obtained, indicating that template matching is complete. In template matching, a template is used to search for the target in a large image, while the template is a smaller image. Therefore, template matching is well-suited for coarse localization and determining optimized yaw values ​​in this task.

[0099] from Figure 4 The input options are: query image, prior information, and 3D texture model.

[0100] The specific process of this method includes the following steps:

[0101] (1) The prior processing module uses the semantic segmentation graph to move the viewpoint at a special location to the center of the road to obtain the initial pose;

[0102] (2) In the coarse localization module, a composite panoramic image is rendered based on the initial pose; (coarse localization module)

[0103] (3) In the coarse localization module, template matching is performed between the synthesized panoramic image and the query image to obtain the refined initial pose;

[0104] (4) In the coarse positioning module, a composite image with a 90-degree field of view is rendered based on the refined initial pose;

[0105] (5) In the coarse positioning module, the composite image of the 90-degree field of view is matched with the query image to obtain a matching pair. Then, the coarse pose is obtained by using the matching pair, the intrinsic parameter matrix and the PnP solution technique.

[0106] (6) Based on the coarse pose, synthesize a composite image with the same viewpoint as the query image, and repeat step 5 to obtain the final pose result.

[0107] In one embodiment, the refined initial pose solver includes a template matching module, a QATM module, and an NMS module; the template matching module includes two structurally identical feature extraction networks sharing parameters. Step 202 includes: inputting the given query image and synthesized panoramic image into the two feature extraction networks respectively to obtain query image features and panoramic image features; inputting the query image features and panoramic image features into the QATM module to calculate the matching quality score; and inputting the matching quality score into the NMS module to obtain the refined pose.

[0108] Specifically, Figure 5 The image shows the template matching process. Figure 6 The diagram shows the template matching algorithm flow. The query image and template image are processed by a feature extraction network to obtain corresponding feature maps, and then the QATM module calculates the highest matching quality score Q. map Finally, the detection boxes are obtained by filtering using the NMS algorithm.

[0109] RIP employs template matching and the QATM concept to achieve accurate localization results. For example... Figure 6 As shown, this module queries image I Q And Panoramic I P As input, a shared feature extraction network is used to extract features F. Q and F P In this work, RIP uses a pre-trained VGG19 as the backbone network. More precisely, the first 16 layers operate on VGG19, comprising 8 convolutional layers, 7 ReLU layers, and 4 max-pooling layers. To combine high-level and low-level features, the results of the second and eighth layers are directly concatenated. Therefore, given H... Q ×W Q ×3 size I Q and H p ×W p ×3 size I pThe feature extractor generates H Q / 4×W Q / 4×320 size F Q and H p / 4×W p / 4×320 size F p , where parameter 4 is the result of two downsampling operations in VGG19, and the channel size 320 is the sum of the two convolutional layers.

[0110] In one embodiment, the query image features and panoramic image features are input into the QATM module, and the matching quality score is calculated as follows:

[0111] Q map ←Max(QATM|P) (1)

[0112] QATM(p,q)=L(q|p)·L(p|q) (2)

[0113]

[0114]

[0115] Among them, Q map For the highest matching quality score, QATM(p,q) is the product of the likelihood function value of p matching in the query graph Q and the likelihood function value of q matching in the template graph P, where q is the query patch, p is the template patch, L(q|p) is the probability that the template patch p will be matched given the query patch q, L(p|q) is the probability that the template patch p will be matched given the template patch p, p' is the remaining sub-patches, P is the template graph, and F Q [x,y] and F P [i,j] represent query patch features and panoramic patch features, respectively. α is a hyperparameter, and Dist(·) is a predefined similarity metric function.

[0116] Specifically, unlike the usual matching strategy, the likelihood value is used instead of the similarity score to determine the similarity between two images. The likelihood value is calculated using formula (2) based on a predefined similarity value, and then QATM(p,q) is calculated using formula (2). Finally, the similarity is determined by Q... map ←Max(QATM|P) yields the highest matching quality score.

[0117] In one embodiment, the matching quality score is input into the NMS module to obtain the refined pose, including: using the maximum response value in the matching quality score, using NMS to obtain detection boxes that may contain the target object; denoting the center of the detection box as C, with the center of the panoramic image at 0 degrees, and the left and right sides at ±180 degrees respectively; and calculating the optimized yaw value with C as the target point:

[0118]

[0119] Among them W p C is the width of the panoramic image. x The x-coordinate of the center detection box.

[0120] The initial pose is updated based on the optimized yaw value to obtain the refined pose.

[0121] Specifically, using the matching quality score Q map The maximum response value in the bounding box is used to obtain the bounding box that may contain the target object.

[0122] When updating the initial pose, the center of the detection box is denoted as C, and the center of the panoramic image is 0 degrees; the left and right sides are ±180 degrees respectively. Using C as the target point, the optimized yaw value can be calculated according to equation (5). Based on the optimized yaw value... The initial pose T0 is updated to obtain the refined pose T0'.

[0123] In one embodiment, step 204 includes: generating a synthetic image and a depth map with a field of view of 90° based on the refined pose; matching the query image and the information synthetic image to generate a matching pair; determining the correspondence between 2D and 3D points in the query image based on the matching pair, intrinsic parameters, and the depth map; and using a PnP solver to calculate the coarse localization pose based on the correspondence.

[0124] In one embodiment, such as Figure 3 The cross-view shown Figure 6 The flowchart illustrates the steps of a degree-of-freedom positioning frame. Solid arrows (→) indicate the main process, while dashed arrows display detailed information for each part. x Indicates 6-DoF pose; D x For depth map; I 90 / I k This is a composite image of 90 / original-FoV. The prior preprocessing module, coarse localization module, and fine localization module constitute a cross-view image. Figure 6 Degrees-of-Freedom Localization Frame (CLaSP Frame). Given a query image I Q Using prior information and texture models as input, the CLaSP framework utilizes a coarse-to-fine process to recover I... Q The accurate pose.

[0125] The pose recovery method within the CLaSP framework is based on traditional camera localization methods. It pairs a reference image with a query image and recovers the final pose using a 3D model and a PnP solver. Unlike traditional retrieval methods that rely on comparison and localization, this method utilizes prior information obtained from sensors to generate a reference image using the Blender engine. To address the issues of inaccurate GPS or CMU measurements and significant visual discrepancies, the model employs prior preprocessing, coarse localization, and fine localization.

[0126] The CLaSP framework's workflow is as follows: Given a single ground query image, prior information, and a texture model as input, PPM processes the prior information into an initial pose T0. The algorithm includes two solvers: a refined initial pose solver and a coarse pose solver. The refined initial pose solver uses the synthesized panorama from T0 for template matching, obtaining a better yaw angle value than the initial one and updating T0 to refine the pose T0'. The coarse localization solver takes T0' as input and generates a synthesized image I with a field of view of 90 degrees. 90 and depth map D 90 . in I Q and I 90 A matching algorithm is used to find matching pairs. Based on these matching pairs, the intrinsic parameter matrix, and D... 90 The coarse pose solver estimates the 2D-3D corresponding points of the query image and uses the PnP algorithm to solve the coarse pose T. c Finally, it's worth noting that the steps of FLM are similar to those of a coarse pose solver. The difference is that FLM uses intrinsic parameters of the original image to generate a synthetic image, ensuring it has the same scale as the query image and thus obtaining a more accurate pose T. f .

[0127] Many studies have proposed using synthetic images for camera localization, among which the most relevant to this application is iNeRF, which estimates the pose of objects in an image by comparing the predicted appearance of objects in the model with their actual appearance in the image through inversion of the NeRF model. However, due to its sensitivity to noise, iNeRF is not suitable for the task described in this application. Meshloc has proven that meshes can be used for pose estimation and describes the overall localization process. However, its accuracy is affected when there are significant differences in viewpoints, such as in cross-viewpoint tasks. The R&C model utilizes a rendering engine to assist in six-DOF pose localization to compensate for large viewpoint differences. The R&C approach requires the use of random seeds and iterative methods based on prior information, which increases the likelihood of including objects. This is a time-consuming process, and the results are unreliable. The CLaSP framework proposed in this application not only has advantages in terms of time consumption but also achieves better localization accuracy through a coarse-to-fine approach.

[0128] In a validation embodiment, the experiments were conducted on the Airloc dataset, which was collected over a large urban area, including buildings, streets, and vegetation. Airloc contains a total of 1970 reference images and 1432 query images, captured from multiple angles by smartphones and drones (UAVs). Furthermore, the dataset includes a 3D reconstruction model built from aerial imagery, providing intrinsic parameters, prior information, and the true pose for each reference and query image. Therefore, Airloc is suitable for cross-view localization tasks.

[0129] Localization estimation experiments were conducted using Airloc ground images, and the experiments were divided into two parts: a daytime dataset containing 533 query images and a nighttime dataset containing 546 query images. In addition, the data used in the experiments also included 3D reconstruction models, prior information, intrinsic parameters, and true poses.

[0130] (1) Baseline and evaluation indicators

[0131] Our proposed method is compared with current mainstream visual localization algorithms HLoc, MeshLoc, and render-compare (R&C). Each algorithm employs different feature points and matching strategies, such as traditional SIFT, learning-based sparse feature point superpoint (SPP), distance-based nearest neighbor (NN) matching, and graph neural network-based superglue (SPG). We use SPP+SPG and LoFTR as our methods. Importantly, these feature points and matching algorithms are directly adopted, rather than fine-tuned or retrained for the AirLoc dataset.

[0132] Evaluation metrics: The method in this application solves the same visual localization problem as R&C, so its localization recall thresholds (25cm, 2°), (50cm, 5°) and (1m, 10°) are used as evaluation metrics.

[0133] (2) Implementation details

[0134] The CLaSP framework in this application comprises three modules. Here, we will introduce the significant experimental setup of these three modules. During the rendering process, we rendered the panoramic image to a size of 1984×992, the 90-FoV image to a size of 960×960, and the original intrinsic parameter image to a size of 2736×3648. In CLM, we utilize a pre-trained weighted VGG19 model as the backbone to extract template matching features, allowing only template matching results with topk=3 to proceed to the next step. For the rendered images, CLM utilizes the intrinsic parameter matrix calculated based on 90-FoV. For FLM, the matching filtering threshold is set to 0.2, and outliers are removed using the RANSAC algorithm during the matching stage. Furthermore, during each matching process, the algorithm renders the original size image I... Q or I k Shrink by 3 times to prevent CUDA errors.

[0135] AirLoc is a public dataset spanning multiple views, eliminating the need to simply use global descriptors to retrieve coarse localizations. We leverage prior information from HLoc and MeshLoc to improve image retrieval.

[0136] If two images are similar, then their 3D point clouds are also very similar. Based on the 3D model and prior knowledge, depth maps of the query image and the reference image can be obtained using backprojection points. Using 2048 random uniform sampling, P can be obtained. q and P r The distance to the back projection point is calculated using the improved chamfer distance MCD, where P q* This represents the 50 query points rearranged in ascending order. Through this calculation, for each query image, we can obtain the top 50 nearest neighbor images {I}. r 1,I 2 r ,…,I r 50}

[0137]

[0138] For the R&C method, the final positioning accuracy is based on a random seed and three iterations. To be fair, we compare the results of two iterations of the R&C method with our proposed method.

[0139] All experiments were conducted on servers configured with i9-11900K CPUs and RTX 3090 GPUs.

[0140] (3) Evaluation and Analysis

[0141] Table 1 shows the visual localization results of our method and the comparative model. It is readily apparent that our method significantly outperforms image retrieval-based localization methods when using the same matching algorithm. It is almost four times better than the daytime dataset and three times better than the more difficult nighttime dataset. Our method outperforms all other methods except for (1m, 10°), which uses the same feature points and matching strategy as the R&C method. In particular, our method achieves significant improvements in high-precision metrics. This embodiment presents recalls for (25cm, 2°) / (50cm, 5°) / (1m, 10°) and compares them with HLoc, MeshLoc, and R&C, different feature matching methods.

[0142] Table 1 Visual localization results of the method and comparison model in this application.

[0143]

[0144] The data "0.0 / 0.0 / 0.0" in the table means that the percentages within the range of (25cm, 2°) / (50cm, 5°) / (1m, 10°) are 0%, 0%, and 0%, respectively. That is, the percentages of the refined pose results within the range of 25cm and 2 degrees are 0%, and so on.

[0145] The method described in this application yielded reliable matching results.

[0146] To ensure a fair comparison, we chose the R&C i2 model for comparison, as the CLaSP framework is similar to iterating twice, but not identically. Notably, the proposed model lags behind the R&C i2 model in accuracy (1m, 10°). This failure can be attributed to exceptional instances where the template matching process fails or results in incorrect matches. This leads to coarse localization results in the synthesized image that may not contain the target object, resulting in a significant discrepancy between the subsequently calculated pose and the ground truth. R&C employs a seeding method, ensuring that at least one direction covers the target region, thus yielding more results in the 1m and 10° range. However, the seeding method is stringent and requires significant time during rendering. The iterative comparison results are shown in Table 2, presenting recalls for (25cm, 2deg) / (50cm, 5deg) / (1m, 10deg) compared to R&C.

[0147] Table 2 Iterative Comparison Results

[0148]

[0149] In Table 2, compared to all iterations, the proposed method outperforms R&C at (25cm, 2°). The model lags behind the (50cm, 5°) and (1m, 10°) metrics, likely due to outliers in the template matching process, as described above. This issue could be caused by tree occlusion and repetitive textures. Notably, once the coarse localization module detects the target object, the framework achieves higher accuracy than other comparative models. This indicates that the CLaSP framework can obtain very robust coarse localization results, thus facilitating the formation of the final pose.

[0150] As shown in Table 2, our proposed method has a significant advantage in terms of time cost. The R&C model takes 61.20 seconds to obtain the final result, but its accuracy is lower than that of our proposed model. Meanwhile, our framework only takes 34.30 seconds, which is 43% less than the time taken by the R&C model.

[0151] The visualization of the matching results shows that the higher the matching result, the better the pose solution.

[0152] (4) Ablation Experiment Analysis

[0153] Module validity: To conduct reasonable comparative experiments, we assume that the following models use FLM, and the ablation experiments only compare PPM and CLM. The ablation experiments use the following variables to analyze the validity of the modules: (1) CLaSP-PP is a model that uses only PPM and CLM; (2) CLaSP-CL is a model that uses only CLM and FLM; (3) The CLaSP-OP model uses the original prior information without any processing; (4) CLaSP is the complete framework proposed in this application.

[0154] The experimental results on the model's effectiveness show that each module has its own advantages compared to having no modules. When they are combined, the optimal results are achieved. A comparison of the module effectiveness results is shown in Table 3.

[0155] Table 3 Comparison of model validity results

[0156]

[0157] Table 3 shows the effectiveness of the proposed model. Regardless of which module is added, the results are better than CLaSP-OP. Combining these two modules yields the optimal solution. The model effectiveness results demonstrate that the proposed model has strong matching ability and provides more accurate pose estimation compared to other models.

[0158] Scale: Synthetic images of different scales can be presented using different intrinsic parameters, which will affect the matching results and thus the final pose. Table 4 shows a comparison of the localization results obtained using synthetic images of different scales. The method in this application uses CLM based on 90-FoV to calculate the focal length because 90-FoV represents the maximum field of view presented without distortion, which helps capture more features. In FLM, to obtain more accurate localization results, the intrinsic parameters of the query image are used to ensure that the query image and synthetic image are presented at the same scale. In Table 4, C.90° means 90-fov CLM, Fk represents the FLM inherent to the phone size, and C.90°+Fk represents a combination of the two modules. If a coarse localization result is found using the 90-FoV method, and the same query image intrinsic is used in FLM, a more accurate FLM matching result can be obtained.

[0159] Comparing the various modules under different rendering strategies, C.90+FK yielded the best results.

[0160] Table 4 Comparison of localization results

[0161]

[0162]

[0163] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0164] In one embodiment, such as Figure 7 As shown, a cross-view Figure 6 The degree-of-freedom positioning device includes: a priori preprocessor, a coarse locator, and a fine locator, wherein:

[0165] The prior preprocessor is used to input the prior information obtained from the sensor into the prior preprocessing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw.

[0166] The coarse localizer is used to input the initial state, 3D texture model, and given query image into the coarse localization module to obtain the coarse localization pose. The coarse localization module is used to synthesize a panoramic image based on the initial state and 3D texture model, and to perform template matching based on the query image and the synthesized panoramic image to obtain the refined pose. Based on the refined pose, a synthesized image with a field of view of 90 degrees and a depth map are generated. Then, the synthesized image and the query image are matched and the PnP algorithm is used to obtain the coarse localization pose.

[0167] The fine locator is used to input the coarse localization pose and the query image into the fine localization module to obtain cross-view results. Figure 6 The results of the degree-of-freedom localization are similar to those of the coarse localization module, except that the fine localization module generates a synthetic image using the same intrinsic parameters and coarse pose as the given query image.

[0168] In one embodiment, the prior preprocessor is further configured to input prior information obtained from the sensor into the prior preprocessing module, and if it is a special location point, to place the location point closer to the center of the road; special location points include: points in or near buildings, bushes; and to adjust the height of the location point using a digital surface model to obtain an initial state.

[0169] In one embodiment, the coarse localization module includes: a synthetic image generation module, a refined initial pose solver, and a coarse pose solver; the coarse localizer is further configured to input the initial state and 3D texture model into the synthetic image generation module to generate a synthetic panoramic image; input the synthetic panoramic image and a given query image into the refined initial pose solver to obtain a refined pose; and input the refined pose and the synthetic panoramic image into the coarse pose solver to obtain a coarse localization pose.

[0170] In one embodiment, the refined initial pose solver includes: a template matching module, a QATM module, and an NMS module; the template matching module includes two structurally identical feature extraction networks sharing parameters; a coarse localizer is also used to input a given query image and a synthesized panoramic image into the two feature extraction networks respectively to obtain query image features and panoramic image features; the query image features and panoramic image features are input into the QATM module to calculate the matching quality score; the matching quality score is input into the NMS module to obtain the refined pose.

[0171] In one embodiment, the coarse locator is also used to input query image features and panoramic image features into the QATM module, and calculate the matching quality score using expressions as shown in equations (1) to (4).

[0172] In one embodiment, the coarse locator is further used to obtain a detection box that may contain the target object using NMS with the maximum response value in the matching quality score; the center of the detection box is denoted as C, the center of the panoramic image is 0 degrees, and the left and right sides are ±180 degrees respectively; with C as the target point, the expression for calculating the optimized yaw value is shown in Equation (5); the initial pose is updated according to the optimized yaw value to obtain the refined pose.

[0173] In one embodiment, the coarse locator is further configured to generate a synthetic image with a field of view of 90° and a depth map based on the refined pose; match the query image and the information synthetic image to generate a matching pair; determine the correspondence between 2D and 3D points in the query image based on the matching pair, intrinsic parameters, and the depth map; and calculate the coarse localization pose using a PnP solver based on the correspondence.

[0174] Regarding cross-vision Figure 6 For specific limitations on the degree-of-freedom positioning device, please refer to the section above regarding cross-viewpoint positioning. Figure 6 The limitations of the degree-of-freedom positioning method will not be elaborated upon here. The aforementioned cross-view... Figure 6 The modules in a degree-of-freedom positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0175] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a cross-view... Figure 6 A method for positioning with degrees of freedom. The display screen of the computer device can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0176] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to perform the steps in the method embodiments described above.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cross-view six-DOF positioning method, characterized in that, The method includes: The prior information obtained from the sensor is input into the prior preprocessing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw angle; The initial state, 3D texture model, and given query image are input into the coarse localization module to obtain the coarse localization pose. The coarse localization module is used to draw a synthetic panoramic image based on the initial state and the 3D texture model, and to perform template matching between the query image and the synthetic panoramic image to obtain a refined pose. Based on the refined pose, a synthetic image with a field of view of 90° and a depth map are generated. Then, the synthetic image and the query image are matched and the PnP algorithm is used to obtain the coarse localization pose. The coarse localization pose and the query image are input into the fine localization module to obtain a cross-view six-DOF localization result. The fine localization module is similar to the coarse localization module, except that a synthetic image is generated in the fine localization module based on the same intrinsic parameters as the given query image and the coarse localization pose. The coarse localization module includes: a synthetic image generation module, a refined initial pose solver, and a coarse pose solver. The initial state, 3D texture model, and given query image are input into the coarse localization module to obtain the coarse localization pose, including: The initial state and the 3D texture model are input into the synthetic image generation module to generate a synthetic panoramic image; The synthesized panoramic image and the given query image are input into the thinning initial pose solver to obtain the thinned pose. The thinning initial pose solver includes a template matching module, a QATM module, and an NMS module. The template matching module includes two structurally identical feature extraction networks sharing parameters. Specifically, it includes: inputting the given query image and the synthesized panoramic image into the two feature extraction networks respectively to obtain query image features and panoramic image features; inputting the query image features and the panoramic image features into the QATM module to calculate the matching quality score; and inputting the matching quality score into the NMS module to obtain the thinned pose. The matching quality score is: in, For the highest matching quality score, Let p be the product of the likelihood function value of p matching in the query graph Q and the likelihood function value of q matching in the template graph P. q To find the patch, p For template patch, For a given query patch q Time template patch p The probability of being matched, For a given template patch p Patch Query q The probability of a match, For the remaining sub-patches, P This is a template image. and These represent query patch features and panoramic patch features, respectively. It is a hyperparameter. It is a predefined function for calculating similarity.

2. The method according to claim 1, characterized in that, The prior information obtained from the sensor is input into the prior preprocessing module for preprocessing to obtain the initial state, including: The prior information obtained from the sensor is input into the prior preprocessing module. If it is a special location point, the location point is placed close to the center of the road. The special location points include: the middle or nearby of buildings and bushes. The height of the location points is adjusted using a digital surface model to obtain the initial state.

3. The method according to claim 1, characterized in that, The matching quality score is input into the NMS module to obtain the refined pose, including: Using the maximum response value in the matching quality score, NMS is used to obtain detection boxes that may contain the target object; The center of the detection frame is denoted as... C The center of the panoramic image is 0 degrees, and the left and right sides are ±180 degrees respectively; C For the target point, the optimized yaw value is calculated as follows: in The width of the panoramic image. For the center detection box x coordinate; The initial pose is updated based on the optimized yaw value to obtain the refined pose.

4. The method according to claim 1, characterized in that, The refined pose is input into the coarse pose solver to obtain the coarse localization pose, including: Based on the refined pose, a synthetic image and a depth map with a field of view of 90 degrees are generated; The query image and the information composite image are matched to generate matching pairs; Based on the matching pairs, intrinsic parameters, and the depth map, determine the correspondence between 2D and 3D points in the query image; The coarse localization pose is obtained by using a PnP solver based on the aforementioned correspondence.

5. A six-degree-of-freedom positioning device spanning a view, characterized in that, The device includes: The prior information processor is used to input the prior information obtained from the sensor into the prior information processing module for preprocessing to obtain the initial state; the prior information includes: latitude, longitude, altitude and yaw angle; A coarse localizer is used to input the initial state, 3D texture model, and given query image into the coarse localization module to obtain a coarse localization pose. The coarse localization module is used to draw a synthetic panoramic image based on the initial state and the 3D texture model, and perform template matching based on the query image and the synthetic panoramic image to obtain a refined pose. Based on the refined pose, a synthetic image with a field of view of 90° and a depth map are generated. Then, matching is performed between the synthetic image and the query image using the PnP algorithm to obtain the coarse localization pose. The coarse localization module includes: a synthetic image generation module, a refined initial pose solver, and a coarse pose solver. Specifically, this includes: inputting the initial state and the 3D texture model into the synthetic image generation module to generate a synthetic panoramic image; inputting the synthetic panoramic image and the given query image into the thinning initial pose solver to obtain the thinned pose; wherein, the thinning initial pose solver includes: a template matching module, a QATM module, and an NMS module; the template matching module includes two feature extraction networks with identical structures and shared parameters; specifically, this includes: inputting the given query image and the synthetic panoramic image into the two feature extraction networks respectively to obtain query image features and panoramic image features; inputting the query image features and the panoramic image features into the QATM module to calculate the matching quality score; inputting the matching quality score into the NMS module to obtain the thinned pose; wherein the matching quality score is: in, For the highest matching quality score, Let p be the product of the likelihood function value of p matching in the query graph Q and the likelihood function value of q matching in the template graph P. q To find the patch, p For template patch, For a given query patch q Time template patch p The probability of being matched, For a given template patch p Patch Query q The probability of a match, For the remaining sub-patches, P This is a template image. and These represent query patch features and panoramic patch features, respectively. It is a hyperparameter. It is a predefined function for calculating similarity; A fine locator is used to input the coarse localization pose and the query image into the fine localization module to obtain a cross-view six-DOF localization result. The fine localization module is similar to the coarse localization module, except that it generates a synthetic image using the same intrinsic parameters as the given query image and the coarse localization pose.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images