Image pose correction method, device, equipment and storage medium

Abstract

The application discloses an image pose correction method and device, equipment and storage medium, the method comprises the steps that the repositioning pose of the to-be-corrected pose frame is obtained after repositioning by a cloud end; the to-be-corrected pose frame is uploaded to the cloud end by a user end; the pose of the to-be-corrected pose frame, a current frame and a transition frame is coarsely corrected based on the repositioning pose; the current frame is a frame played when the user end receives the pose returned by the cloud end, and the transition frame is a frame between the to-be-corrected pose frame and the current frame; the pose of each frame after coarse correction is finely corrected. The trajectory curve composed of the pose of each frame after fine correction is smoother, so that the position information does not jump when the pose is updated.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image pose correction method, apparatus, device and storage medium. Background Technology

[0002] How to solve the problem of accumulated error is an unavoidable challenge in the field of SLAM (Simultaneous Localization and Mapping).

[0003] Currently, the problem of accumulated errors is solved by performing real-time relocation on the user end. When performing real-time relocation on the user end, the pose is updated immediately after successful relocation, which causes the position information of the current frame on the user end to change. In AR (Augmented Reality) applications, this manifests as the user's "position" suddenly teleporting, breaking the sense of realism of the immersive experience. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image pose correction method, apparatus, device, and storage medium, which aim to avoid position information jumps when updating pose.

[0005] To achieve the above objectives, this application provides an image pose correction method, the method comprising:

[0006] The repositioned pose of the pose frame to be corrected is obtained after repositioning from the cloud; the pose frame to be corrected is uploaded from the user terminal to the cloud.

[0007] Based on the repositioning pose, the poses of the pose frame to be corrected, the current frame, and the transition frame are coarsely corrected; the current frame is the frame played when the user receives the pose transmitted back from the cloud, and the transition frame is the frame between the pose frame to be corrected and the current frame.

[0008] Fine correction is performed on the pose of each frame after coarse correction.

[0009] For example, the fine correction of the pose of each frame after coarse correction includes:

[0010] Obtain the first map point set matched by the cloud from the pose frame to be corrected, and obtain the second map point set matched by the user terminal from the pose frame to be corrected;

[0011] Determine a new map point corresponding to the pose frame to be corrected from the first map point set, wherein the new map point is a map point not present in the second map point set;

[0012] Based on the new map points and the second map point set, the pose of each frame after coarse correction is finely corrected.

[0013] For example, determining the new map points corresponding to the pose frame to be corrected from the first map point set includes:

[0014] The third map point set is obtained by taking the difference between the first map point set and the second map point set;

[0015] Project the third map point set onto the pose frame to be corrected to obtain the feature point set;

[0016] Determine the candidate point set for each first feature point in the feature point set;

[0017] Based on the map points corresponding to the candidate point set, new map points corresponding to the pose frame to be corrected are determined.

[0018] For example, determining the new map points corresponding to the pose frame to be corrected based on the map points corresponding to the candidate point set includes:

[0019] Iterate through each candidate point set;

[0020] Calculate the descriptor distance between each second feature point in the traversed candidate point set and the map point corresponding to the traversed candidate point set;

[0021] Based on the descriptor distance, a new map point corresponding to the pose frame to be corrected is determined.

[0022] For example, the step of fine-correcting the pose of each frame after coarse correction based on the new map points and the second map point set includes:

[0023] Remove map points from the second map point set that do not belong to the pose frame to be corrected, and obtain the second map point set after removal;

[0024] Traverse each frame after coarse correction, and perform feature point matching between the traversed coarse correction frames and the second set of map points after removal and the new map points to obtain a fine map point set;

[0025] Based on the fine map point set, the pose of the coarsely corrected frames is finely corrected.

[0026] For example, obtaining the repositioned pose of the pose frame to be corrected after repositioning by the cloud includes:

[0027] At every first number of frames, a second number of frames corresponding to the first number are sent to the cloud so that the cloud can reposition the pose of the second number of frames, wherein the second number of frames includes pose frames to be corrected.

[0028] Receive the repositioned pose sent by the cloud after the cloud has repositioned the pose frame to be corrected.

[0029] For example, before sending a second number of frames corresponding to the first number to the cloud at each first number of frames interval, the process includes:

[0030] Obtain the actual trajectory and the estimated trajectory;

[0031] Based on the actual trajectory and the estimated trajectory, the first quantity is calculated.

[0032] Furthermore, to achieve the above objectives, this application also provides an image pose correction device, the device comprising:

[0033] The acquisition module is used to acquire the pose of the pose frame to be corrected after repositioning by the cloud; the pose frame to be corrected is uploaded to the cloud by the user terminal;

[0034] The coarse correction module is used to coarsely correct the poses of the pose frame to be corrected, the current frame, and the transition frame between the pose frame to be corrected and the current frame based on the pose; the current frame is the frame played when the user terminal receives the pose transmitted back from the cloud.

[0035] The fine correction module is used to perform fine correction on the pose of each frame after coarse correction.

[0036] In addition, to achieve the above objectives, this application also provides an image pose correction device, which includes a memory, a processor, and an image pose correction program stored in the memory and executable on the processor. When the image pose correction program is executed by the processor, it implements the steps of the image pose correction method described above.

[0037] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing an image pose correction program, which, when executed by a processor, implements the steps of the image pose correction method described above.

[0038] Compared to existing technologies where the pose is updated immediately after successful real-time repositioning on the user end, causing a jump in the position information of the current frame on the user end, this application obtains the repositioned pose of the pose frame to be corrected after repositioning by the cloud; the pose frame to be corrected is uploaded to the cloud by the user end; coarse correction is performed on the poses of the pose frame to be corrected, the current frame, and the transition frame based on the repositioned pose; the current frame is the frame played when the user end receives the pose returned from the cloud, and the transition frame is the frame between the pose frame to be corrected and the current frame; fine correction is performed on the poses of each frame after coarse correction. This application achieves real-time repositioning without performing it on the user end. Instead, it performs repositioning on the pose frame to be corrected via the cloud. After obtaining the repositioned pose from the cloud, it uses the pose of the pose frame to be corrected, the current frame, and the transition frame to perform fine correction on the pose of each frame after coarse correction, instead of only correcting the current frame. This makes the trajectory curve formed by the poses of each frame after fine correction smoother, thereby avoiding abrupt changes in position information when updating the pose. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the first embodiment of the image pose correction method of this application;

[0040] Figure 2 This is a schematic diagram of the actual trajectory and the estimated trajectory involved in the first embodiment of the image pose correction method of this application;

[0041] Figure 3 This is a schematic diagram of the coarse correction process in the first embodiment of the image pose correction method of this application;

[0042] Figure 4 This is a schematic diagram of the hardware operating environment involved in the embodiments of this application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0045] This application provides an image pose correction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image pose correction method of this application.

[0046] This application provides embodiments of an image pose correction method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order. The image pose correction method can be applied to user terminals (e.g., mobile phones, tablets, AR glasses, etc.). For ease of description, the execution entity descriptions of the various steps of the image pose correction method are omitted below. The image pose correction method includes:

[0047] Step S110: Obtain the repositioned pose of the pose frame to be corrected after repositioning from the cloud; the pose frame to be corrected is uploaded to the cloud by the user terminal.

[0048] For example, in this embodiment, the relocalization of the pose frame to be corrected is performed via the cloud (server). SLAM is a system with extremely high real-time requirements, but the user end may be a mobile phone, tablet, AR glasses, or other devices with limited computing power. To ensure the real-time performance of the entire system, only feature points and descriptors with low computational cost and fast computation speed can be used for relocalization, resulting in low success rate and accuracy. The computing power of the cloud is far superior to that of these devices. Therefore, by performing relocalization on the cloud server, the powerful computing power and GPU (Graphics Processing Unit) hardware advantages can be utilized to employ the most effective and robust feature extraction methods to extract feature points with strong computational performance, greatly improving the success rate and accuracy of relocalization. The accuracy of the relocalized pose obtained in this way can be comparable to the accuracy of the relocalized pose in the real trajectory collected by high-precision equipment.

[0049] For example, obtaining the repositioned pose of the pose frame to be corrected after repositioning by the cloud includes:

[0050] Step a: At every first number of frames, send a second number of frames corresponding to the first number to the cloud, so that the cloud can reposition the pose of the second number of frames, wherein the second number of frames includes the pose frames to be corrected.

[0051] Step b: Receive the repositioned pose sent by the cloud after the cloud has repositioned the pose frame to be corrected.

[0052] For the process of relocalizing the second number of frames in the cloud, taking the relocalization of the pose frame to be corrected as an example, it includes the following steps: extracting feature points from the pose frame to be corrected using superpoint; performing NetVLAD matching between the pose frame to be corrected and the map keyframes stored in the cloud to obtain a matching image set; performing feature matching based on the extracted feature points and the matched image set to obtain a matching result; and performing PNP to solve the pose based on the matching result to obtain the relocalized pose.

[0053] After obtaining the repositioned pose, the error correction amount is calculated by comparing the repositioned pose with the estimated pose calculated by the user terminal. Specifically, after n (second number) frames of images are processed by n service threads in the cloud, n repositioned poses are obtained. The translation information (translation amount) in the repositioned poses is extracted and stored in set X; the translation information (translation amount) is extracted from the estimated poses of the n frames of images from the user terminal and stored in set Y, and the error correction amount T is calculated. Δ The process of (R, t) is as follows:

[0054] R = U * S * VH, t = m y -R*m x

[0055] Where R is the rotation amount, t is the translation amount, U and VH are matrices composed of the eigenvectors after performing SVD (Singular Value Decomposition) on the covariance matrix of XY, S is a 3x3 identity matrix, and m x m y It is the set of averages obtained by averaging each translation individually.

[0056]

[0057]

[0058] It is understood that the error correction amount is calculated using multiple frames of images, specifically images preceding the frame whose pose needs to be corrected. For the pose correction of the current frame, the smoother the trajectory curve, the smaller the error correction amount; conversely, the less smooth the trajectory curve, the larger the error correction amount. The error correction amount is the cause of sudden shifts in "position." The magnitude of the error correction amount is positively correlated with the degree of the shift; that is, the larger the error correction amount, the more pronounced the shift; and the smaller the error correction amount, the less pronounced the shift. Therefore, this embodiment corrects the transition frames to make the trajectory curve smoother, thereby avoiding jumps in position information when updating the pose of the current frame.

[0059] For example, before sending a second number of frames corresponding to the first number to the cloud at each first number of frames interval, the process includes:

[0060] Step c: Obtain the actual trajectory and the estimated trajectory;

[0061] Step d: Calculate the first quantity based on the actual trajectory and the estimated trajectory.

[0062] Reference Figure 2 , Figure 2 This diagram illustrates the true and estimated trajectories. 201 represents the true trajectory, and 202 represents the estimated trajectory. The difference in the ordinate between the true and estimated trajectories represents the error. The trajectory is obtained by concatenating the poses of multiple frames. It should be noted that because the pose of each subsequent frame incorporates the error from the previous frame during calculation, the gap between the estimated and true trajectories will increase over time, meaning the error will grow larger.

[0063] For example, the actual trajectory is obtained from a pre-built map, which is constructed using data collected by a high-precision device (such as LiDAR). For instance, in a scenic area, before a user with a terminal arrives at the scenic area, data for building a map is collected from the scenic area using LiDAR. After the map is built, the actual trajectory is obtained from the map.

[0064] For example, the estimated trajectory is obtained by SLAM estimation on the user end, and the pose that constitutes the estimated trajectory reflects the camera pose when the user end takes the image.

[0065] It should be noted that the factors contributing to the error include both the data acquisition process of the user-end sensors and the computation process of the user-end SLAM algorithm. The first quantity is related to these error contributing factors; since different user-end sensors and computational parameters differ, the first quantity will vary for different user-ends.

[0066] For example, the pose includes position and orientation, and the first quantity is calculated by the error in position and the error in orientation.

[0067] For errors in orientation, the orientation is determined by the formula... The calculation yields, where Q i P i ∈SE(3), Q i P represents the true pose. i Δ represents the pose estimated by SLAM, and △ represents the time interval.

[0068] For positional errors, the position is represented by translation, where the translation is the amount by which the current pose position is shifted relative to the camera space origin. The translation is calculated using the following formula:

[0069]

[0070] Where, trans represents the calculation of the translation amount R:

[0071]

[0072] This gives us the first number, which is to request cloud relocation once every first number of frames.

[0073] For example, the second quantity is determined by the first quantity, specifically the smaller value between the number of relocation threads deployed in the cloud and the first quantity. That is, when the first quantity is greater than the number of relocation threads, frames of the number of relocation threads are uploaded to avoid queuing in cloud computing, thus affecting the speed of cloud relocation; when the first quantity is less than or equal to the number of relocation threads, frames of the first quantity are uploaded, which can be understood as not causing queuing in cloud computing, and therefore does not affect the speed of cloud relocation.

[0074] Step S120: Based on the repositioning pose, coarsely correct the poses of the pose frame to be corrected, the current frame, and the transition frame; the current frame is the frame played when the user receives the pose transmitted back from the cloud, and the transition frame is the frame between the pose frame to be corrected and the current frame.

[0075] It should be noted that repositioning the pose of the frame to be corrected via the cloud requires transmitting the frame to the cloud, performing calculations on the cloud, and then retrieving the repositioned pose from the cloud. This process adds two network transmissions, causing the entire repositioning process to be non-real-time; that is, the pose returned from the cloud does not correspond to the image frame currently displayed on the user's device. For example, refer to... Figure 3 , Figure 3 This is a schematic diagram of the coarse correction process. Cloud repositioning is triggered in frame 6. After receiving the repositioning pose returned from the cloud, the user terminal has already displayed up to frame 10.

[0076] Assume the error correction amount after relocalization calculation of the pose frame to be corrected (the i-th frame) in the cloud is T. Δ After the cloud computation is completed, the user terminal has currently run (played) up to frame j. Let the local tracking pose (SLAM-estimated pose) within this range be denoted as ij.

[0077] Tcw k k = i, i+1...j

[0078] The pose Tcw of the current k-th frame after coarse correction k (coarse) = T Δ *Tcw kAssuming the cloud-based relocation accuracy is high enough, after coarse correction using error correction, it can be considered that the cumulative error of the entire user-side local tracking system (SLAM system) before the i-th frame has been eliminated, while the cumulative error between the remaining ij-th frames has not been completely eliminated. For example, refer to... Figure 3 Where 301 is the true trajectory, 302 is the estimated trajectory, and when i is the 6th frame, the error is 303, that is, the error correction amount is 303. After the cloud calculation is completed, the user terminal has run to the 10th frame when j is the error correction amount. The error correction amount is directly applied to the 7th to 10th frames to obtain a series of coarsely corrected poses. Ideally, it can be considered that the accumulated error before the 6th frame has been eliminated.

[0079] Step S130: Perform fine correction on the pose of each frame after coarse correction.

[0080] For example, the pose obtained after coarse correction generally still has a certain error compared to the true value, so fine correction is needed to further reduce this error. The fine correction of the pose of each frame after coarse correction includes:

[0081] Step e: Obtain the first map point set matched by the cloud from the pose frame to be corrected, and obtain the second map point set matched by the user terminal from the pose frame to be corrected.

[0082] Before matching, feature points of the pose frame to be corrected need to be extracted. It should be noted that these feature points are 2D points.

[0083] It should be noted that, due to the abundant computing power in the cloud, unlike the user end which needs to limit the number of feature points extracted per frame (usually 500 or 1000 points), the number of feature points extracted in the cloud is unlimited. Therefore, the number of feature points extracted in the cloud can be far greater than that extracted by the user end. This means that, based on using more feature points and a stronger matching algorithm, the number of map points that can be successfully matched in the cloud will also be greater than that on the user end. That is, the first map point set P... c The number of map points in the first map set is greater than that in the second map point set P. u The number of map points in the map. It should be noted that the map points are 3D points.

[0084] The matching process involves matching the extracted feature points with map points in the constructed map.

[0085] Step f: Determine a new map point corresponding to the pose frame to be corrected from the first map point set, wherein the new map point is a map point not present in the second map point set;

[0086] For example, determining the new map points corresponding to the pose frame to be corrected from the first map point set includes:

[0087] Step f1: Take the difference between the first map point set and the second map point set to obtain the third map point set;

[0088] Take the points belonging to the first map set P c But it does not belong to the second map point set P u The map points (i.e., the difference set is the new map points found through cloud matching) are used to obtain the third map point set, denoted as P. m .

[0089] Step f2: Project the third map point set onto the pose frame to be corrected to obtain the feature point set;

[0090] Let the feature point set be p m The projection process is achieved using the following formula:

[0091] p m =KT iw (Cloud)P m

[0092] Where K represents the camera intrinsics of the user's camera, which describe the relationship between points of a 3D object and points in the image, and T... iw (Cloud) represents the repositioning pose of the i-th frame obtained through cloud computing.

[0093] Step f3: Determine the candidate point set for each first feature point in the feature point set;

[0094] Take the feature point set p m A point p in the feature point set p m The matching process for finding matching feature points is as follows: Within a radius r centered on the coordinates of point p, we search for feature points. If no feature points exist within this radius, point p fails to match; if feature points exist within this radius, all feature points within this radius are recorded as a candidate point set. Here, r is a variable that can be set as needed.

[0095] It is understandable that for a feature point set p m Each first feature point in the dataset may have a corresponding set of candidate points.

[0096] Step f4: Based on the map points corresponding to the candidate point set, determine the new map points corresponding to the pose frame to be corrected.

[0097] For example, determining the new map points corresponding to the pose frame to be corrected based on the map points corresponding to the candidate point set includes:

[0098] Step f41: Traverse each candidate point set;

[0099] Step f42: Calculate the descriptor distance between each second feature point in the traversed candidate point set and the map point corresponding to the traversed candidate point set;

[0100] Calculate the descriptor distance between the feature points in the candidate point set and the map point P corresponding to the first feature point p in the traversed candidate point set, and take the maximum value dmax and the minimum value dmin of the descriptor distance.

[0101] Step f43: Based on the descriptor distance, determine the new map points corresponding to the pose frame to be corrected;

[0102] If dmax > dmin * 1.5, then map point P is determined to have successfully matched with a feature point whose descriptor distance is dmin; if dmax ≤ dmin * 1.5, then map point P is determined to have failed to match with a feature point whose descriptor distance is dmin. The successfully matched map point P is added to the newly added map point set P. add That is, P add The center point is the new map point.

[0103] Step g: Based on the new map points and the second map point set, perform fine correction on the pose of each frame after coarse correction;

[0104] For example, the step of fine-correcting the pose of each frame after coarse correction based on the new map points and the second map point set includes:

[0105] Step g1: Remove map points in the second map point set that do not belong to the pose frame to be corrected, and obtain the second map point set after removal.

[0106] Pose T obtained using cloud repositioning iw (Cloud) Perform interior point verification on feature points that are successfully matched on the user end, and record the map points in the second map point set corresponding to the outer points that fail the interior point verification as the map point set P to be removed. out The difference between the second set of map points and the set of map points to be removed is the second set of map points after removal. It should be noted that the set of map points to be removed, P... out The map points in the image are those that do not belong to the pose frame to be corrected.

[0107] Step g2: Traverse each frame after coarse correction, and perform feature point matching between the traversed coarse correction frames and the new map points and the second set of removed map points to obtain the fine map point set.

[0108] Taking the i-th frame as an example, the fine map point set P of each frame after coarse correction correct,i It is calculated using the following formula:

[0109] P correct,i =P u ∪P add -P out

[0110] Step g3: Based on the fine map point set, perform fine correction on the pose of the coarsely corrected frames that have been traversed.

[0111] The fine map point set P of the i-th frame (the pose frame to be corrected) correct,i Match the feature points of each frame of the coarsely corrected frame j (the current frame) with the feature points of each frame. Let k be the frame to be finely corrected (i.e., a frame in the transition frames), and P be the point set P. correct,i For any map point P in the i-th frame, the feature point corresponding to P is p. i P is considered a successful match if both of the following conditions are met:

[0112] 1. Feature point p in the i-th frame i It is continuously tracked up to the k-th frame, and denoted as p i The feature point matched in the k-th frame is p k ;

[0113] 2. Project map point P onto the k-th frame, and the resulting projected point is the same as p. k Match successful.

[0114] Remove the map points that failed to match from P. correct,i After removing points from the middle, we obtain the fine map point set P belonging to the k-th frame. correct,k .

[0115] Tcw with coarse correction for repositioning pose k (coarse) is the initial value, P correct,k The map points in the image are projected onto the k-th frame, and the optimized fine-corrected pose Tcw,k is obtained by minimizing the following cost function.

[0116]

[0117] It is understandable that fine correction can be performed using the optimized fine-corrected pose Tcw,k.

[0118] Compared to existing technologies where the pose is updated immediately after successful real-time repositioning on the user end, causing a jump in the position information of the current frame on the user end, this application obtains the repositioned pose of the pose frame to be corrected after repositioning by the cloud; the pose frame to be corrected is uploaded to the cloud by the user end; coarse correction is performed on the poses of the pose frame to be corrected, the current frame, and the transition frame based on the repositioned pose; the current frame is the frame played when the user end receives the pose returned from the cloud, and the transition frame is the frame between the pose frame to be corrected and the current frame; fine correction is performed on the poses of each frame after coarse correction. This application achieves real-time repositioning without performing it on the user end. Instead, it performs repositioning on the pose frame to be corrected via the cloud. After obtaining the repositioned pose from the cloud, it uses the pose of the pose frame to be corrected, the current frame, and the transition frame to perform fine correction on the pose of each frame after coarse correction, instead of only correcting the current frame. This makes the trajectory curve formed by the poses of each frame after fine correction smoother, thereby avoiding abrupt changes in position information when updating the pose.

[0119] In addition, this application also provides an image pose correction device, the device comprising:

[0120] The acquisition module is used to acquire the pose of the pose frame to be corrected after repositioning by the cloud; the pose frame to be corrected is uploaded to the cloud by the user terminal;

[0121] The coarse correction module is used to coarsely correct the poses of the pose frame to be corrected, the current frame, and the transition frame between the pose frame to be corrected and the current frame based on the pose; the current frame is the frame played when the user terminal receives the pose transmitted back from the cloud.

[0122] The fine correction module is used to perform fine correction on the pose of each frame after coarse correction.

[0123] For example, the fine correction module is specifically used for:

[0124] Obtain the first map point set matched by the cloud from the pose frame to be corrected, and obtain the second map point set matched by the user terminal from the pose frame to be corrected;

[0125] Determine a new map point corresponding to the pose frame to be corrected from the first map point set, wherein the new map point is a map point not present in the second map point set;

[0126] Based on the new map points and the second map point set, the pose of each frame after coarse correction is finely corrected.

[0127] For example, the fine correction module is also used for:

[0128] The third map point set is obtained by taking the difference between the first map point set and the second map point set;

[0129] Project the third map point set onto the pose frame to be corrected to obtain the feature point set;

[0130] Determine the candidate point set for each first feature point in the feature point set;

[0131] Based on the map points corresponding to the candidate point set, new map points corresponding to the pose frame to be corrected are determined.

[0132] For example, the fine correction module is also used for:

[0133] Iterate through each candidate point set;

[0134] Calculate the descriptor distance between each second feature point in the traversed candidate point set and the map point corresponding to the traversed candidate point set;

[0135] Based on the descriptor distance, a new map point corresponding to the pose frame to be corrected is determined.

[0136] For example, the fine correction module is also used for:

[0137] Remove map points from the second map point set that do not belong to the pose frame to be corrected, and obtain the second map point set after removal;

[0138] Traverse each frame after coarse correction, and perform feature point matching between the traversed coarse correction frames and the second set of map points after removal and the new map points to obtain a fine map point set;

[0139] Based on the fine map point set, the pose of the coarsely corrected frames is finely corrected.

[0140] For example, the acquisition module is specifically used for:

[0141] At every first number of frames, a second number of frames corresponding to the first number are sent to the cloud so that the cloud can reposition the pose of the second number of frames, wherein the second number of frames includes pose frames to be corrected.

[0142] Receive the repositioned pose sent by the cloud after the cloud has repositioned the pose frame to be corrected.

[0143] For example, the acquisition module is further configured to:

[0144] Obtain the actual trajectory and the estimated trajectory;

[0145] Based on the actual trajectory and the estimated trajectory, the first quantity is calculated.

[0146] The specific implementation of the image pose correction device in this application is basically the same as the embodiments of the image pose correction method described above, and will not be repeated here.

[0147] In addition, this application also provides an image pose correction device. For example... Figure 4 As shown, Figure 4 This is a schematic diagram of the hardware operating environment involved in the embodiments of this application.

[0148] In one possible implementation, Figure 4 This can be a schematic diagram of the hardware operating environment of the image pose correction device.

[0149] like Figure 4 As shown, the image pose correction device may include a processor 401, a communication interface 402, a memory 403, and a communication bus 404. The processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. The memory 403 is used to store computer programs. When the processor 401 executes the program stored in the memory 403, it implements the steps of the image pose correction method.

[0150] The communication bus 404 mentioned in the aforementioned image pose correction device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 404 can be divided into an address bus, a data bus, and a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0151] The communication interface 402 is used for communication between the above-mentioned image pose correction device and other devices.

[0152] The memory 403 may include random access memory (RMD) or non-volatile memory (NM), such as at least one disk storage device. Optionally, the memory 403 may also be at least one storage device located remotely from the aforementioned processor 401.

[0153] The processor 401 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0154] The specific implementation of the image pose correction device in this application is basically the same as the embodiments of the image pose correction method described above, and will not be repeated here.

[0155] Furthermore, embodiments of this application also propose a computer-readable storage medium storing an image pose correction program, which, when executed by a processor, implements the steps of the image pose correction method described above.

[0156] The specific implementation of the computer-readable storage medium in this application is basically the same as the embodiments of the above-described image pose correction method, and will not be repeated here.

[0157] In addition, to achieve the above objectives, this application also provides a computer program product, comprising: a computer program, which, when executed by a processor, implements the steps of the image pose correction method as described above.

[0158] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described image pose correction method, and will not be repeated here.

[0159] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0160] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes: several instructions to cause a terminal device (which may be a mobile phone, computer, server, device, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0162] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image pose correction method, characterized in that, The method includes: The repositioned pose of the pose frame to be corrected is obtained after repositioning from the cloud; the pose frame to be corrected is uploaded from the user terminal to the cloud. Based on the repositioning pose, the poses of the pose frame to be corrected, the current frame, and the transition frame are coarsely corrected. The current frame is the frame played when the user receives the pose transmitted back from the cloud. The transition frame is the frame between the pose frame to be corrected and the current frame. The step of coarsely correcting the poses of the frame to be corrected, the current frame, and the transition frame based on the repositioning pose includes: Extract the first translation information from the repositioning pose, and extract the second translation information from the images of the user terminal in a preset second number of frames; The first translation information and the second translation information are processed to obtain the error correction amount; Based on the error correction amount, the repositioning pose is coarsely corrected for the pose of the frame to be corrected, the current frame, and the transition frame. Fine correction is performed on the pose of each frame after coarse correction.

2. The method as described in claim 1, characterized in that, The fine correction of the pose of each frame after coarse correction includes: Obtain the first map point set matched by the cloud from the pose frame to be corrected, and obtain the second map point set matched by the user terminal from the pose frame to be corrected; Determine a new map point corresponding to the pose frame to be corrected from the first map point set, wherein the new map point is a map point not present in the second map point set; Based on the new map points and the second map point set, the pose of each frame after coarse correction is finely corrected.

3. The method as described in claim 2, characterized in that, The step of determining the new map points corresponding to the pose frame to be corrected from the first map point set includes: The third map point set is obtained by taking the difference between the first map point set and the second map point set; Project the third map point set onto the pose frame to be corrected to obtain the feature point set; Determine the candidate point set for each first feature point in the feature point set; Based on the map points corresponding to the candidate point set, new map points corresponding to the pose frame to be corrected are determined.

4. The method as described in claim 3, characterized in that, The step of determining new map points corresponding to the pose frame to be corrected based on the map points corresponding to the candidate point set includes: Iterate through each candidate point set; Calculate the descriptor distance between each second feature point in the traversed candidate point set and the map point corresponding to the traversed candidate point set; Based on the descriptor distance, a new map point corresponding to the pose frame to be corrected is determined.

5. The method as described in claim 2, characterized in that, The step of fine-correcting the pose of each frame after coarse correction based on the new map points and the second map point set includes: Remove map points from the second map point set that do not belong to the pose frame to be corrected, and obtain the second map point set after removal; Traverse each frame after coarse correction, and perform feature point matching between the traversed coarse correction frames and the second set of map points after removal and the new map points to obtain a fine map point set; Based on the fine map point set, the pose of the coarsely corrected frames is finely corrected.

6. The method as described in claim 1, characterized in that, The process of obtaining the repositioned pose of the pose frame to be corrected after repositioning from the cloud includes: At every first number of frames, a second number of frames corresponding to the first number are sent to the cloud so that the cloud can reposition the pose of the second number of frames, wherein the second number of frames includes pose frames to be corrected. Receive the repositioned pose sent by the cloud after the cloud has repositioned the pose frame to be corrected.

7. The method as described in claim 6, characterized in that, Before sending a second number of frames corresponding to the first number to the cloud at each interval of a first number of frames, the process includes: Obtain the actual trajectory and the estimated trajectory; Based on the actual trajectory and the estimated trajectory, the first quantity is calculated.

8. An image pose correction device, characterized in that, The device includes: The acquisition module is used to acquire the pose of the pose frame to be corrected after repositioning by the cloud; the pose frame to be corrected is uploaded to the cloud by the user terminal; The coarse correction module is used to coarsely correct the poses of the pose frame to be corrected, the current frame, and the transition frame between the pose frame to be corrected and the current frame based on the pose; the current frame is the frame played when the user terminal receives the pose transmitted back from the cloud. The fine correction module is used to perform fine correction on the pose of each frame after coarse correction. The coarse correction module is further configured to: extract first translation information from the repositioning pose and extract second translation information from the images of a preset second number of frames on the user terminal; process the first translation information and the second translation information to obtain an error correction amount; and, based on the error correction amount, coarsely correct the pose of the repositioning pose for the pose frame to be corrected, the current frame, and the transition frame.

9. An image pose correction device, characterized in that, The image pose correction device includes a memory, a processor, and an image pose correction program stored in the memory and executable on the processor. When the image pose correction program is executed by the processor, it implements the steps of the image pose correction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image pose correction program, which, when executed by a processor, implements the steps of the image pose correction method as described in any one of claims 1 to 7.

Images