Method, device, equipment, medium and program product for determining image pose data
By optimizing the video segmentation and keyframe association relationship of the to-process video, the problem of inaccurate positioning of image pose data is solved, the accuracy and efficiency of high-precision maps are improved, and the high-accurate pose data determination is achieved.
Patent Information
- Application Number
- CN202411154821.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The method for determining image pose data in the prior art is insufficient in accuracy, especially in the process of high-precision map production, inaccurate positioning of image pose data leads to global positioning instability, affecting the accuracy of high-precision map.
By segmenting the video to be processed, pose optimization and iteration is performed by sub-videos, pose processing and optimization are used to utilize the correlation between keyframes to reduce error accumulation and improve the consistency and accuracy of pose data.
It improves the accuracy and efficiency of image pose data, ensures the high accuracy of high-precision maps, and reduces the requirements of computing resources and processing time.
Smart Images

Figure CN118674786B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of high-precision map technology, and in particular to a method, device, equipment, medium and program product for determining image posture data. Background Art
[0002] With the rapid development of autonomous driving technology, the importance of high-precision maps has become increasingly prominent. In the process of making high-precision maps, the precise positioning of image pose data is the key to improving the accuracy of high-precision maps.
[0003] At present, the method for determining image pose data is mainly based on all the collected image data, extracting the visual feature points in the image data one by one, and applying the Simultaneous Localization and Mapping (SLAM) algorithm or the Structure from Motion (SFM) algorithm to determine the image pose.
[0004] However, the above-mentioned method for determining image pose data has the problem of insufficient accuracy. Summary of the invention
[0005] The present application provides a method, device, equipment, medium and program product for determining image pose data, so as to solve the problem of insufficient accuracy in determining image pose data.
[0006] In a first aspect, the present application provides a method for determining image pose data, comprising:
[0007] For at least two sub-videos obtained by segmenting the video to be processed, at least one posture optimization iteration is performed until an iteration termination condition is satisfied, and the posture data of the image in the video to be processed is obtained based on each sub-video after the posture optimization; wherein each posture optimization iteration at least includes:
[0008] Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video, wherein the key frame in each sub-video is an image in the sub-video that meets a preset image feature condition;
[0009] Based on the optimized key frames in each sub-video, the pose of each sub-video is optimized.
[0010] In a possible design of the first aspect, the iteration termination condition includes at least one of the following:
[0011] The number of iterations reaches the preset number of iterations;
[0012] The reprojection error value of the video to be processed is less than or equal to a preset error value.
[0013] In a possible design of the first aspect, the preset image feature condition includes at least one of the following conditions:
[0014] The number of feature points in the image is greater than or equal to a preset number;
[0015] The curvature value of the image is greater than or equal to the preset curvature;
[0016] There are at least two images in the common viewing area, and the at least two images are respectively in different video segments;
[0017] Selected images are spaced at preset geographic length intervals.
[0018] In a possible design of the first aspect, based on the association relationship between the key frames in different sub-videos, performing posture processing on the key frames in each sub-video to obtain the optimized key frames in each sub-video includes:
[0019] Taking each sub-video as a target video, respectively, and determining a target key frame set corresponding to each target video based on key frames in the target video and non-target videos; the target key frame set corresponding to each target video includes a first key frame and a second key frame, the first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a co-viewing area with the first key frame; the non-target video is a sub-video other than the target video in at least two sub-videos;
[0020] The pose of the first key frame and the second key frame in each target key frame set is optimized.
[0021] In a possible design of the first aspect, determining a target key frame set corresponding to each target video based on key frames in the target video and the non-target video includes:
[0022] Determine a key frame in the non-target video whose spatial distance with the first key frame is less than or equal to a preset distance threshold as a candidate key frame;
[0023] A candidate key frame that has a common viewing area with the first key frame is determined as the second key frame.
[0024] In a possible design of the first aspect, performing pose optimization on the first key frame and the second key frame in each target key frame set includes:
[0025] For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point based on the first pose data of the first key frame, the second pose data of the second key frame, the first pixel data corresponding to the physical point in the first key frame, and the second pixel data corresponding to the physical point in the second key frame;
[0026] Back-projecting the three-dimensional position data to the first key frame to obtain third pixel data, and back-projecting the three-dimensional position data to the second key frame to obtain fourth pixel data;
[0027] Based on the difference between the first pixel data and the third pixel data, and the difference between the second pixel data and the fourth pixel data corresponding to each physical point, the first posture data and the second posture data are optimized to obtain an optimized first key frame and an optimized second key frame.
[0028] In a possible design of the first aspect, the step of optimizing the posture of each sub-video based on the optimized key frame in each sub-video includes:
[0029] According to the pose data of the optimized key frames in each sub-video, pose optimization processing is performed on the ordinary frames in each sub-video to obtain optimized ordinary frames, where the ordinary frames are images in the sub-video excluding the key frames.
[0030] In a possible design of the first aspect, the at least two sub-videos obtained by segmenting the video to be processed are subjected to at least one pose optimization iteration until an iteration termination condition is satisfied, and the method further includes:
[0031] A preset posture determination algorithm is used to optimize the posture data of each video frame in each sub-video.
[0032] In a possible design of the first aspect, the at least two sub-videos obtained by segmenting the video to be processed are subjected to at least one pose optimization iteration until an iteration termination condition is satisfied, and the method further includes:
[0033] Based on the trajectory information when the video to be processed is collected, the video to be processed is divided into the at least two sub-videos.
[0034] In a second aspect, the present application provides a method for determining image pose data, comprising:
[0035] Perform at least one posture optimization iteration on the video to be processed until an iteration termination condition is met, and obtain the posture data of the image in the video to be processed based on the video to be processed after the posture optimization; wherein each posture optimization iteration includes:
[0036] Performing posture optimization on the key frames in the video to be processed to obtain optimized key frames;
[0037] Based on the optimized key frames, optimizing the common frames in the video to be processed;
[0038] The key frame is an image in the video to be processed that meets a preset image feature condition, and the common frame is an image in the video to be processed except the key frame.
[0039] In a possible design of the second aspect, the iteration termination condition includes at least one of the following:
[0040] The number of iterations reaches the preset number of iterations;
[0041] The reprojection error value of the video to be processed is less than or equal to a preset error value.
[0042] In a possible design of the second aspect, the preset image feature condition includes at least one of the following conditions:
[0043] The number of feature points in the image is greater than or equal to a preset number;
[0044] The curvature value of the image is greater than or equal to the preset curvature;
[0045] Selected images are spaced at preset geographic length intervals.
[0046] In a third aspect, the present application provides a device for determining image pose data, comprising:
[0047] A processing module, used for performing at least one posture optimization iteration on at least two sub-videos obtained by segmenting the video to be processed until an iteration termination condition is satisfied;
[0048] An acquisition module, used for acquiring the pose data of the image in the video to be processed based on each sub-video after the pose is optimized;
[0049] Wherein, the processing module is used at least in each posture optimization iteration to:
[0050] Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video, wherein the key frame in each sub-video is an image in the sub-video that meets a preset image feature condition;
[0051] Based on the optimized key frames in each sub-video, the pose of each sub-video is optimized.
[0052] In a fourth aspect, the present application provides a device for determining image pose data, comprising:
[0053] A processing module, used for performing at least one posture optimization iterative process on the video to be processed until an iteration termination condition is met;
[0054] An acquisition module, used for acquiring the posture data of the image in the video to be processed based on the video to be processed after the posture optimization;
[0055] Wherein, the processing module is used in each posture optimization iteration to:
[0056] Performing posture optimization on the key frames in the video to be processed to obtain optimized key frames;
[0057] Based on the optimized key frames, ordinary frames in the video to be processed are optimized; wherein the key frames are images in the video to be processed that meet preset image feature conditions, and the ordinary frames are images in the video to be processed except the key frames.
[0058] In a fifth aspect, the present application provides a computer device, comprising: a transceiver, a processor, and a memory communicatively connected to the processor;
[0059] The memory stores computer-executable instructions;
[0060] The processor executes the computer-executable instructions stored in the memory to implement the method for determining image pose data as described in any one of the first aspect or the second aspect.
[0061] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method for determining image pose data as described in any one of the first aspect or the second aspect.
[0062] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for determining image pose data as described in any one of the first aspect or the second aspect.
[0063] The method, device, equipment, medium and program product for determining image pose data provided by the present application relate to the field of high-precision map technology. The method determines the pose data of the image by performing at least one pose optimization iteration on each sub-video in the video to be processed. Each pose iteration optimization includes at least: performing pose processing on the key frames based on the association relationship between the key frames in different sub-videos, and adjusting the pose of each sub-video according to the optimized key frames. Optimizing the pose using the association relationship between key frames can make full use of relative information, reduce the error accumulation of pose data, and improve the consistency of pose data. At the same time, the iterative optimization method can improve the accuracy and efficiency of image pose data, thereby ensuring the high accuracy of the final pose data. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0065] Figure 1 A schematic diagram of an application scenario of a method for determining image pose data provided in an embodiment of the present application;
[0066] Figure 2 A flowchart of a method for determining image pose data according to an embodiment of the present application;
[0067] Figure 3 A schematic diagram of a flow chart of a second embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0068] Figure 4 A flowchart of a third embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0069] Figure 5 A flowchart of a fourth embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0070] Figure 6 A flowchart of a fifth embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0071] Figure 7 A flowchart of a sixth embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0072] Figure 8 A flowchart of a seventh embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0073] Fig. 9 A flowchart of an eighth embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0074] Fig.10 A flowchart of a ninth embodiment of a method for determining image pose data provided in an embodiment of the present application;
[0075] Fig.11 A flowchart of a method for determining image pose data according to an embodiment of the present application;
[0076] Fig.12 A schematic diagram of a framework of a method for determining image pose data provided in an embodiment of the present application;
[0077] Fig.13A flowchart of a key frame selection strategy provided in an embodiment of the present application;
[0078] Fig.14 A factor graph for solving an image pose data optimization problem provided in an embodiment of the present application;
[0079] Fig.15 A schematic diagram of the structure of a first embodiment of a device for determining image pose data provided in an embodiment of the present application;
[0080] Fig.16 A schematic diagram of the structure of a second embodiment of a device for determining image pose data provided in an embodiment of the present application;
[0081] Fig.17 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.
[0082] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0083] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0084] In the field of autonomous driving technology, the importance of high-precision maps is becoming increasingly prominent. In the process of applying automated technology solutions to produce high-precision maps, the current mainstream production process includes image data acquisition, automated mapping technology processing, high-precision map generation and review, etc. Among them, in the process of applying automated mapping technology processing, the precise determination of image pose data is a key factor in improving the accuracy of high-precision maps.
[0085] At present, the method for determining image pose data mainly relies on all image data collected by image sensors and global navigation and positioning systems. Specifically, for processing systems with high real-time computing requirements, the Simultaneous Localization and Mapping (SLAM) algorithm is usually used for processing. For processing systems with relatively low real-time computing requirements, the Structure from Motion (SFM) algorithm is usually used for processing. In these processing flows, it is first necessary to extract the visual feature points of all images and match the visual feature point descriptors. Then, using the matching relationship of the descriptors, the motion recovery structure problem is constructed using stereo geometry methods. Finally, the image pose data is solved by the bundle adjustment method. This series of steps constitutes the basic process for determining image pose data in the prior art.
[0086] However, the main difficulties faced by existing methods for determining image pose data include large-scale computing problems caused by the huge scale of image data and the problem of insufficient positioning accuracy of image pose data. Specifically, in the process of collecting image data, multiple collections are usually performed within a small geographical range (such as an intersection) to ensure the integrity of the image data. However, this multi-pass collection method will result in an extremely large amount of image data. For example, in an area of 500 meters × 500 meters, the number of collected images after preprocessing usually reaches 10,000 to 100,000. This huge amount of image data will bring huge computational challenges. At the same time, large-scale image data can easily cause the problem of unstable global positioning, resulting in the inability of some images to achieve visual alignment globally. Therefore, how to improve the positioning accuracy of image pose data and effectively process large-scale image data has become a key problem that current technology urgently needs to solve.
[0087] In view of the above problems, the inventors found in the process of studying the method for determining the image pose data that the problem of inaccurate positioning of the image pose data is usually caused by the huge amount of image data and the participation of all images in the optimization. Specifically, large-scale image data easily leads to the problem of unstable global positioning, resulting in the inability of some images to achieve visual alignment globally, thereby affecting the accuracy of the image pose data. Based on this, the inventors involve pre-dividing the collected image data into multiple parts according to a certain image segmentation method before optimizing the image pose data, and then iterating and associating the pose optimization processing of these segmented parts to obtain the image pose data. Specifically, in the embodiment of the present application, the video to be processed is divided into multiple sub-videos, and each sub-video is subjected to at least one pose optimization iteration to determine the pose data of the image. In each iteration, the optimization process includes: performing pose processing on the key frames based on the association relationship between the key frames in different sub-videos, and adjusting the pose of each sub-video according to the optimized key frames. This pose iterative optimization method can reduce the amount of image data processed each time, thereby reducing the demand for computing resources and processing time. At the same time, it helps to improve the stability of global positioning and reduce the situation where images cannot be visually aligned globally, thereby improving the positioning accuracy of image pose data.
[0088] Figure 1 A schematic diagram of an application scenario of a method for determining image pose data provided in an embodiment of the present application. Figure 1 As shown, the application scenario of the solution provided by the present application includes a computer device 100. The computer device 100 is deployed with a system 1001 for determining image pose data.
[0089] It is worth noting that the computer device 100 can be a single terminal device, such as a laptop, a desktop computer, or a vehicle-mounted device. The computer 100 can also be a combination of the above-mentioned multiple terminal devices. In addition, the computer device 100 can also be an independent server, a cloud server cluster, or other types of devices. The present application does not specifically limit the specific form and type of the computer device 100.
[0090] Specifically, the image pose data determination system 1001 in the computer device 100 divides the video to be processed into at least two sub-videos, and performs at least one pose optimization iteration on these sub-videos until the iteration termination condition is met. Then, based on each optimized sub-video, the image pose data determination system 1001 obtains the pose data of the image in the video to be processed.
[0091] It should be noted that the above scenario is only an example of an application scenario provided by an embodiment of the present application. The embodiment of the present application does not limit the actual form of the various devices included in the scenario. In the specific application of the solution, it can be set according to actual needs.
[0092] For ease of description, the following describes the implementation of the method by taking the execution subject as a computer device capable of executing the method for determining image pose data as an example. It can be understood that the execution subject of the method as a computer device is only an exemplary description and should not be understood as a limitation of the method.
[0093] The technical solutions shown in the present application are described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.
[0094] Figure 2 The flowchart of the first embodiment of the method for determining image pose data provided in the present application is as follows. Figure 2 As shown, the process of the method for determining the image pose data may include step S201 and step S202, specifically:
[0095] S201: Perform at least one pose optimization iteration on at least two sub-videos obtained by segmenting the video to be processed until an iteration termination condition is satisfied.
[0096] In this step, given that the accuracy of the image pose data directly affects the accuracy of the high-precision map, this application focuses on processing and optimizing the image pose data, aiming to improve its accuracy, thereby improving the accuracy of the high-precision map.
[0097] In order to reduce the computational scale and improve processing efficiency, the present application divides the video to be processed into at least two sub-videos before processing and optimizing the image pose data. It is worth noting that the specific division method of dividing the video to be processed into at least two sub-videos can refer to step S901, which will not be repeated here.
[0098] Specifically, the video to be processed is the original video data obtained from the acquisition device, including a continuous image sequence taken at different times and the initial posture data corresponding to each image. Among them, the acquisition device includes but is not limited to any one or any combination of a global navigation positioning system, an inertial navigation system, and other sensors that can record geographic location information and motion trajectory. Acquiring the video to be processed through a variety of acquisition devices can ensure the comprehensiveness and diversity of the image posture data, thereby providing rich basic data for subsequent processing and optimization. The initial posture data corresponding to each image includes the position coordinates and posture angle of the acquisition device that captured the image in three-dimensional space, wherein the posture angle includes any one or any combination of pitch angle, yaw angle and roll angle.
[0099] Furthermore, in order to improve the accuracy of the image pose data, at least one pose optimization iteration is performed on at least two sub-videos obtained by segmentation until an iteration termination condition is met. The iteration termination condition is used to determine whether the optimization process has converged to a stable pose result, thereby stopping further iterations.
[0100] It is worth noting that each time the pose optimization iteration is performed, at least Figure 3 The process of the method for determining image pose data is shown. Figure 3 The flowchart of the second embodiment of the method for determining image pose data provided in the present application is as follows. Figure 3 As shown, the process of the method for determining the image pose data may include:
[0101] S301: Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video. The key frames in each sub-video are images in the sub-video that meet preset image feature conditions.
[0102] In this step, the process of each pose optimization iteration is mainly described.
[0103] Specifically, each sub-video is composed of a series of continuous frames, which contain feature point information of varying degrees, such as one or any combination of corner points, edges and textures. In order to improve processing efficiency and the accuracy of image pose data, the present application pre-determines key frames that meet preset image feature conditions from each sub-video, and based on the association between key frames in different sub-videos, performs pose processing on the key frames in each sub-video to obtain optimized key frames in each sub-video. The specific implementation method can refer to steps S401 to S402, which will not be repeated here.
[0104] In a possible implementation manner, the preset image feature condition includes at least one of the following conditions:
[0105] The number of feature points in the image is greater than or equal to a preset number;
[0106] The curvature value of the image is greater than or equal to the preset curvature;
[0107] There are at least two images in the common viewing area, and the at least two images are in different sub-videos;
[0108] Selected images are spaced at preset geographic length intervals.
[0109] Specifically, when determining the key frame from each sub-video, it is based on whether each frame image in each sub-video meets the preset image feature conditions. If a frame image meets any one of the preset image feature conditions, it is determined as a key frame.
[0110] The preset image feature conditions are mainly set based on a reference factor or any combination of reference factors among the expressiveness of the image, the mutation characteristics of the image content, the correlation of the image across video segments, and the uniformity of the image in spatial distribution.
[0111] Specifically, the expressive power of an image is mainly measured by the number of feature points in the image. Among them, a feature point is a point with significant features in the image, such as any one or combination of corner points, edge points, etc. Optionally, the method for determining the number of feature points in an image includes but is not limited to any one or combination of a scale-invariant feature transformation algorithm and an accelerated robust feature algorithm. The scale-invariant feature transformation algorithm detects local extreme points as feature points by constructing a Gaussian pyramid and calculating a differential Gaussian image, thereby determining the number of feature points in each frame of the image. The accelerated robust feature algorithm uses integral images to accelerate calculations, and detects feature points at different scales through the determinant of the Hessian matrix, thereby determining the number of feature points in each frame of the image. It is worth noting that the present application does not specifically limit the method for determining the number of feature points in an image.
[0112] By detecting the number of feature points in each frame, the feature richness of each frame can be determined. If the number of feature points in a frame is greater than or equal to the preset number, the frame will be considered to contain enough feature information and can be determined as a key frame. It is worth noting that the preset number is determined according to the actual situation, and this application does not make a specific limitation on the preset number.
[0113] For example, if the preset number of feature points is 50, then images containing 50 or more feature points will be determined as key frames.
[0114] The mutation characteristics of image content are mainly measured by the curvature value of the image. The curvature value of the image is a measurement method to describe the rate of change of pixels in the image, reflecting the degree of change in the local area of the image. High curvature values usually correspond to edges, corners or other significant feature areas in the image, which are often where the image content mutates.
[0115] Optionally, the calculation method of the curvature value of the image includes but is not limited to any one or combination of a gradient-based calculation method and a second-order derivative-based calculation method. Among them, the gradient-based calculation method calculates the gradient vector of each pixel in the image, and then uses the rate of change of these gradient vectors to calculate the curvature value. The calculation method based on the second-order derivative directly calculates the curvature value by calculating the second-order derivative of the image, such as the eigenvalue of the Laplace operator or the Hessian matrix. It is worth noting that the present application does not specifically limit the calculation method of the curvature value of the image.
[0116] By calculating the curvature value of the image, the area with drastic changes in the image can be identified, thereby determining whether the current image can be used as a key frame image. Specifically, if the curvature value of a frame image is greater than or equal to a preset curvature, the frame image is considered to have a high degree of detail and can be determined as a key frame.
[0117] For example, if the preset curvature is 0.5, then the image with a curvature value greater than or equal to 0.5 will be determined as a key frame. It is worth noting that the preset curvature is determined according to actual conditions, and this application does not make any specific restrictions on the preset curvature.
[0118] The correlation of images across video segments is mainly determined by judging whether there is a common view area between at least two images in different sub-videos, wherein the common view area refers to the part of at least two images in different sub-videos containing the same scene or object.
[0119] In a possible implementation, a three-dimensional point cloud can be used to detect whether there is a common view area between at least two images in different sub-videos. Specifically, the image in each sub-video is firstly three-dimensionally reconstructed to generate corresponding three-dimensional point cloud data, wherein the three-dimensional point cloud data refers to a data set composed of a large number of three-dimensional coordinate points. Then, by comparing the three-dimensional point cloud data generated in different sub-videos and analyzing their overlapping parts, it is determined whether there is a common view area. This method utilizes the rich information of the three-dimensional point cloud data and can accurately detect the common view area between images.
[0120] In another possible implementation, a feature point matching algorithm can be used to detect whether there is a common view area between at least two images in different sub-videos. Specifically, a feature point detection algorithm is first used to detect feature points of each image in each sub-video, wherein the feature point detection algorithm includes but is not limited to any one or combination of a scale-invariant feature transformation algorithm and an accelerated robust feature algorithm. After the feature points of each image are detected, a feature point matching algorithm is used to find images with the same feature points in other sub-videos, wherein the feature point matching algorithm includes but is not limited to any one or combination of a nearest neighbor matching algorithm based on Euclidean distance and a binary descriptor matching algorithm based on Hamming distance. Specifically, the nearest neighbor matching algorithm based on Euclidean distance calculates the Euclidean distance between two feature point descriptors and takes the feature point pair with the smallest distance as a matching pair, thereby achieving feature point matching. The binary descriptor matching algorithm based on Hamming distance calculates the Hamming distance between two binary feature point descriptors and takes the feature point pair with the smallest Hamming distance as a matching pair, thereby achieving feature point matching. It is worth noting that the present application does not specifically limit the feature point detection algorithm and the feature point matching algorithm.
[0121] If a preset threshold number of matching feature points is found in the images of different sub-videos, it can be determined that there is a common view area between these images, thereby determining the correlation between them and using them as key frames. It is worth noting that the preset threshold number is determined according to actual conditions, and this application does not specifically limit the preset threshold number.
[0122] For example, if the preset threshold number is 50, then an image pair containing 50 or more matching feature points will be considered to have a common view area and will be determined as a key frame.
[0123] In a possible implementation, when determining key frames based on the correlation of images across video segments, there is a limitation on the spatial distance range. Specifically, when determining key frames based on the correlation of images across video segments, a predefined distance range is set based on the center position of the frame to be matched. Only frames within the distance range will be considered as candidate frames for the frame to be matched.
[0124] This distance range limitation helps to improve the efficiency and accuracy of key frame determination. First, by limiting the distance range, the number of images that need to be processed can be reduced, thereby reducing computational complexity and processing time. Second, the distance range limitation ensures that the candidate frame has a high spatial correlation with the frame to be matched, thereby improving the accuracy and reliability of matching. Finally, the distance range limitation can ensure that key frames can still be selected quickly and accurately in complex scenes, providing a more reliable basis for subsequent image pose data calculations.
[0125] The uniformity of the spatial distribution of images is mainly achieved by setting a preset geographic length. Specifically, in order to ensure effective coverage of each area in each sub-video, images can be selected as key frames at intervals of the preset geographic length. It is worth noting that the preset geographic length is determined according to actual conditions, and this application does not specifically limit the preset geographic length.
[0126] For example, if the preset geographic length is 200 meters, then a frame will be selected as a key frame every 200 meters in each sub-video. Based on the uniformity of the image distribution in space, it can be ensured that the selected key frames are comprehensive and can better represent the actual situation of the area to be 3D reconstructed.
[0127] Optionally, the preset image feature condition may also be set according to any one or a combination of contents such as the complexity of image texture, motion vector, and image entropy value.
[0128] Specifically, the complexity of image texture refers to the details and variation of the texture in the image. Images with higher complexity usually contain more feature information, which can be measured by calculating the gradient of the image. Therefore, frames whose texture complexity exceeds a preset complexity threshold can be determined as key frames.
[0129] Motion vectors describe the direction and speed of movement of an object or scene in an image, and are calculated by analyzing the pixel displacement between consecutive frames. Frames with more significant changes in motion vectors usually correspond to important scene changes. Therefore, frames with motion vectors exceeding a preset motion vector threshold can be identified as key frames.
[0130] Image entropy is an indicator to measure the amount of information and complexity of an image, reflecting the randomness and uncertainty of the grayscale distribution of the image. Images with higher entropy values usually contain more information and details. Therefore, frames with entropy values exceeding the preset entropy value can be determined as key frames.
[0131] By applying the above preset image feature conditions, key frames can be accurately identified from each sub-video. This not only ensures that key frames can be accurately selected under different scenes and conditions, but also lays a solid foundation for the determination of subsequent image pose data.
[0132] After the key frame is determined based on the above preset image feature conditions, the other images except the key frame are determined as ordinary frames. This method ensures that the determined key frame has a high degree of representativeness and information content, and also further simplifies the determination process of ordinary frames.
[0133] The method for determining image pose data provided in this embodiment identifies and determines key frames by setting image feature conditions. Specifically, determining key frames based on preset image feature conditions such as the number of feature points, curvature values, common viewing areas, and geographic length intervals can significantly improve the accuracy and reliability of key frame selection, while simplifying the selection process of ordinary frames. This method not only ensures that the selected key frames contain rich feature information and detail changes, but also effectively captures the correlation between different sub-videos and the coverage of geographical locations, thereby providing a more stable and accurate benchmark in subsequent image processing and pose data calculations. Through this systematic key frame determination method, errors can be reduced, processing efficiency can be improved, and the final generated image pose data can be ensured to have high accuracy and high reliability.
[0134] After the key frames are determined in each sub-video by using the preset image feature conditions, the association relationship between the key frames in different sub-videos needs to be established next.
[0135] It is worth noting that the association between key frames in different sub-videos is mainly determined by the common view area. By identifying the image areas containing the same or similar features in different key frames, the association between key frames can be effectively established. After the association is established, the key frames in each sub-video are processed based on the association between the key frames.
[0136] In a possible implementation, a preset posture determination algorithm may be used to perform posture processing on the key frames in each sub-video, wherein the preset posture determination algorithm includes but is not limited to any one or a combination of an SFM algorithm and a SLAM algorithm.
[0137] The SFM algorithm first determines the common view area between key frames in different sub-videos, then determines the key frame group to be optimized based on the common view area, and then constructs a global optimization problem based on the key frame group to be optimized. The goal of this global optimization problem is to minimize the reprojection error value by adjusting the pose of each key frame. The reprojection error value refers to the distance between the corresponding feature point in the actual image after the three-dimensional position data is projected back to the image plane through the pose data. Through iterative optimization, the SFM algorithm continuously adjusts the pose of the key frame to reduce the reprojection error value, thereby obtaining accurate key frame pose data.
[0138] The SLAM algorithm first extracts feature points in each key frame, then builds the correspondence between adjacent key frames based on the extracted feature points, and then constructs an optimization problem based on the correspondence. By adjusting the pose of each key frame, the reprojection error value is minimized, thereby obtaining the key frame pose data.
[0139] Preferably, the preset posture determination algorithm is the SFM algorithm. The SFM algorithm has higher accuracy than the SLAM algorithm. Specifically, the SFM algorithm is usually run in an offline environment and can perform more complex and sophisticated optimization to obtain higher three-dimensional reconstruction accuracy. At the same time, the SFM algorithm can process a large number of multi-view images without real-time calculation, so more computing resources can be used for optimization to further improve the accuracy of posture data.
[0140] By adopting the preset posture determination algorithm, optimized key frames in each sub-video can be obtained. These optimized key frames maintain good correlation and consistency between different sub-videos.
[0141] S302: Optimize the position and posture of each sub-video based on the optimized key frame in each sub-video.
[0142] In this step, after the key frames in each sub-video are optimized based on the association relationship, the posture of each sub-video is optimized based on the optimized key frames in each sub-video. When optimizing the posture of each sub-video, a preset posture determination algorithm can be used. The specific implementation process can refer to step S701, which will not be repeated here.
[0143] S202: Acquire pose data of images in the video to be processed based on each sub-video after pose optimization.
[0144] In this step, according to step S201, after performing at least one posture optimization iteration on at least two sub-videos until the iteration termination condition is met, the posture data of the image in the video to be processed is obtained based on each optimized sub-video.
[0145] Specifically, the pose optimization results of each sub-video provide accurate pose information. Since each sub-video is obtained by segmenting the video to be processed, the pose data of the optimized sub-video can be used to derive the pose of each image in the video to be processed. By applying this optimized pose information to the video to be processed, accurate alignment and pose determination of the image can be achieved, thereby providing effective data support for the three-dimensional construction of high-precision maps.
[0146] The method for determining image pose data provided in the embodiment of the present application significantly improves the accuracy of image pose data while reducing the scale of calculation by performing at least one pose optimization iteration on at least two sub-videos. The core of pose optimization iteration is to gradually approach the optimal solution to ensure the accuracy of pose data. Since key frames contain the most representative feature information, prioritizing key frames can effectively reduce the amount of calculation. At the same time, optimization based on the correlation between key frames in different sub-videos can ensure the consistency and correlation of key frames on a global scale, thereby further improving the accuracy and reliability of pose data. This method not only improves computing efficiency, but also provides a solid data foundation for the three-dimensional construction of high-precision maps.
[0147] In a possible implementation, the iteration termination condition includes at least one of the following:
[0148] The number of iterations reaches the preset number of iterations;
[0149] The reprojection error value of the video to be processed is less than or equal to the preset error value.
[0150] The preset number of iterations is used to prevent the image pose data optimization process from falling into an infinite loop, thereby ensuring that the processing process is completed within a reasonable time. By setting a reasonable preset number of iterations, it is possible to avoid the waste of computing resources and excessive extension of processing time caused by the algorithm used to optimize the image pose data in some cases due to slow convergence or failure to converge.
[0151] The reprojection error value is an important indicator to measure the accuracy of the optimized image pose data. The reprojection error value refers to the distance between the three-dimensional position data and the corresponding feature points in the actual image after the optimized pose data is projected back to the image plane. The preset error value is a threshold used to determine whether the optimization process has achieved sufficient accuracy. When the reprojection error value of the video to be processed is less than or equal to the preset error value, it means that the optimized pose data is accurate enough and the iteration process can be terminated. This iterative termination condition based on the error threshold can ensure the accuracy of the optimization results while avoiding unnecessary computational overhead.
[0152] It is worth noting that the preset number of iterations and the preset error value are determined according to actual conditions and actual experience, and this application does not specifically limit the preset number of iterations and the preset error value. At the same time, the number of iterations and the reprojection error value can be used alone or in combination.
[0153] It is worth noting that the reprojection error value of the video to be processed refers to the sum of the reprojection error values of all images in all the videos to be processed. Specifically, first, for each feature point in each frame of the image, its reprojection error value is calculated. Then, the reprojection error values of all feature points in each frame of the image are accumulated to obtain the total reprojection error value of the frame of the image. Finally, the total reprojection error values of all frame images are accumulated to obtain the reprojection error value of the entire video to be processed. In this way, the accuracy of the image pose data in the video to be processed can be quantified, thereby providing a basis for subsequent optimization and processing.
[0154] If the iteration termination condition is that the number of iterations reaches the preset number of iterations, then in each iteration process, the algorithm for optimizing the image pose data will check whether the current number of iterations has reached the preset number of iterations. If the number of iterations has not reached the preset number of iterations, the algorithm for optimizing the image pose data will continue to be applied for the next round of iterations, and the above steps will be repeated until the number of iterations reaches the preset number of iterations.
[0155] If the iteration termination condition is that the reprojection error value of the video to be processed is less than or equal to the preset error value, then in each iteration, the algorithm for optimizing the image pose data will calculate the reprojection error value of the current video to be processed and compare it with the preset error value. If the calculated reprojection error value of the video to be processed is greater than the preset error value, the algorithm for optimizing the image pose data will continue to be applied for the next round of iteration, and the above steps will be repeated until the reprojection error value of the video to be processed is less than or equal to the preset error value.
[0156] In actual operation, the optimization process of image pose data usually considers these two iteration termination conditions at the same time. When any one of the conditions is met, the iteration process is terminated. For example, if during the iteration process, the reprojection error value of the video to be processed has dropped below the preset error value, even if the number of iterations has not reached the preset maximum number of iterations, the iteration can be terminated in advance to save computing resources and time. Conversely, if the number of iterations has reached the preset maximum number of iterations, but the reprojection error value of the video to be processed has not reached the preset error value, the iteration process will also be terminated to prevent an infinite loop.
[0157] By setting the iteration termination conditions, we can ensure the efficiency of the image pose data optimization process and the accuracy and reliability of the optimization results, providing a solid foundation for the subsequent construction of high-precision maps.
[0158] Optionally, the iteration termination condition may also include other factors, such as the usage of computing resources, processing time limit, etc. These conditions may be used alone or in combination to flexibly control the termination of the iteration process.
[0159] The method for determining image pose data provided in this embodiment can effectively prevent the algorithm for optimizing image pose data from falling into an infinite loop by setting an iteration termination condition, thereby saving computing resources and time. At the same time, the iteration termination condition can also avoid overfitting, ensuring that the algorithm for optimizing image pose data converges to the optimal solution within a reasonable time range, thereby optimizing the execution effect of the algorithm and improving the accuracy of pose data.
[0160] Next, the specific implementation process of step S301 will be discussed in detail. Figure 4 The flowchart of the third embodiment of the method for determining image pose data provided in the present application is as follows. Figure 4 As shown, on the basis of the above embodiment, based on the association relationship between the key frames in different sub-videos, the key frames in each sub-video are processed for posture to obtain the optimized key frames in each sub-video. The process of the method for determining the image posture data may include:
[0161] S401: Take each sub-video as a target video, and determine a target key frame set corresponding to each target video based on key frames in the target video and non-target videos; the target key frame set corresponding to each target video includes a first key frame and a second key frame, the first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a common viewing area with the first key frame; the non-target video is a sub-video other than the target video in at least two sub-videos.
[0162] In this step, when performing posture processing on the key frames in each sub-video based on the association between the key frames in different sub-videos, it is necessary to predetermine the target key frame set for each key frame in each sub-video, that is, to determine the common viewing area between the key frames.
[0163] Specifically, when determining the target key frame set of each key frame in each sub-video, each sub-video is selected as the target video one by one, and the other sub-videos are used as non-target videos. Based on the key frames in the target video and the non-target videos, the target key frame set corresponding to the target video is determined.
[0164] Each target key frame set includes two types of key frames: a first key frame and a second key frame. The first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a common view area with the first key frame. Based on a three-dimensional point cloud method or any one or combination of feature point matching algorithms, it can be identified whether there is a common view area between key frames in different sub-videos. The specific implementation process can refer to step S301, which will not be repeated here.
[0165] For example, the video to be processed is divided into three sub-videos, namely sub-video A, sub-video B and sub-video C. Among them, when sub-video A is selected as the target video, sub-video B and sub-video C are used as non-target videos. When sub-video B is selected as the target video, sub-video A and sub-video C are used as non-target videos. When sub-video C is selected as the target video, sub-video A and sub-video B are used as non-target videos. If sub-video A includes key frame A1, key frame A2 and key frame A3, sub-video B includes key frame B1 and key frame B2, and sub-video C includes key frame C1, key frame C2 and key frame C3. The following takes sub-video A as the target video as an example to describe in detail the determination of the first key frame and the second key frame.
[0166] Specifically, based on the feature point matching algorithm, it is identified that there is a common view area between key frame A1 and key frame B1 and between key frame A1 and key frame C1, then key frame A1 is determined as the first key frame, and key frame B1 and key frame C1 are determined as the second key frame. Therefore, the target key frame set of sub-video A includes key frame A1, key frame B1 and key frame C1.
[0167] Based on the above processing method, a target key frame set can be determined for each sub-video. Each target key frame set contains the key frame in the target video (the first key frame) and the key frame in the non-target video that has a co-viewing area with it (the second key frame). These target key frame sets will be used for subsequent pose processing to achieve pose estimation and optimization based on the association relationship between key frames in different sub-videos.
[0168] S402: Optimize the position and posture of the first key frame and the second key frame in each target key frame set.
[0169] In this step, according to step S401, after the target key frame set corresponding to each target video is determined, the first key frame and the second key frame in each target key frame set are then subjected to posture optimization.
[0170] Among them, posture optimization is to minimize the reprojection error value by adjusting the posture data of each frame of the image (including the position coordinates and posture angle of the acquisition device that acquires the image in three-dimensional space), thereby improving the overall consistency and accuracy of the multi-view video.
[0171] Specifically, according to the target key frame set corresponding to each target video determined in step S401, feature points matched by the common viewing area between the key frames can be obtained, thereby efficiently updating the pose data of each frame of the image.
[0172] In a possible implementation, before performing pose optimization on the first key frame and the second key frame in each target key frame set, an error function is first constructed. The error function is used to quantify the pose difference between the first key frame and the second key frame. For each pair of matching feature points in the first key frame and the second key frame, their reprojection error values under different viewing angles can be calculated. The reprojection error value refers to the distance between the three-dimensional position data and the corresponding feature point in the actual image after the optimized pose data is projected back to the image plane. The error function can be expressed as the sum of the reprojection errors of all matching feature points.
[0173] Next, the preset posture determination algorithm is used to minimize the error function, that is, minimize the reprojection error value of the video to be processed. Among them, the preset posture determination algorithm includes but is not limited to any one or combination of the SFM algorithm, SLAM algorithm, etc. These algorithms iteratively adjust the posture data of each frame image to gradually reduce the reprojection error value of the video to be processed, thereby finding the optimal posture data. Specifically, the preset posture determination algorithm calculates the gradient of the error function based on the current posture data, and updates the posture data along the direction of the gradient until the reprojection error value of the video to be processed converges to a smaller value and / or the number of iterations reaches a preset number of iterations.
[0174] For example, there is a first key frame D1 in the target video D, and a second key frame E2 in the non-target video E. The feature point matching algorithm identifies that there is a common view area between D1 and E2, and there are multiple matching feature points between D1 and E2. In the pose optimization process, an error function can be constructed, which includes the reprojection error of all matching feature points. Then, the preset pose determination algorithm is used to iteratively adjust the pose data of D1 and E2 so that the reprojection error value is gradually reduced, thereby finding the optimal pose data.
[0175] Through the above process, the pose of the first key frame and the second key frame in each target key frame set can be effectively optimized to improve the overall consistency and accuracy of the multi-view video.
[0176] The method for determining image pose data provided in an embodiment of the present application optimizes the key frames in each sub-video using a predetermined target key frame set when performing pose processing. By considering the common viewing area between the first key frame in the target video and the second key frame in the non-target video, the pose of the key frame can be adjusted and optimized more accurately. This cross-video key frame association optimization method helps to reduce calculation errors and improve the accuracy of pose optimization. At the same time, through this method, it can be ensured that the key frame pose in each sub-video not only depends on its own feature information, but also combines the relevant feature information in other sub-videos, thereby further improving the overall accuracy and reliability of the pose data.
[0177] Next, the specific implementation process of step S401 will be discussed in detail. Figure 5 The flowchart of the fourth embodiment of the method for determining image pose data provided in the present application is as follows. Figure 5 As shown, based on any of the above embodiments, based on the key frames in the target video and the non-target video, a target key frame set corresponding to each target video is determined, and the process of the method for determining the image pose data may include:
[0178] S501: Determine a key frame in a non-target video whose spatial distance to the first key frame is less than or equal to a preset distance threshold as a candidate key frame.
[0179] In this step, in order to reduce the calculation scale and improve processing efficiency, when determining the target key frame set corresponding to the target video in combination with the non-target video, the candidate key frames are pre-determined by screening out the key frames in the non-target video whose spatial distance to the first key frame in the target video is less than or equal to a preset distance threshold.
[0180] The preset distance threshold is a parameter for filtering key frames, indicating the maximum allowable distance between two key frames in space. The preset distance threshold can be adjusted according to specific application scenarios and computing resources, and this application does not make specific restrictions on this.
[0181] Specifically, first, for the first key frame in each target video, the spatial distance between it and all key frames in the non-target video is calculated. The spatial distance can be calculated by various methods, such as any one or a combination of Euclidean distance, Manhattan distance, etc. It is worth noting that the present application does not specifically limit the method for calculating the spatial distance.
[0182] Next, the key frames in the non-target video whose spatial distance to the first key frame is less than or equal to the preset distance threshold are determined as candidate key frames. In this way, the key frames in the non-target video that are spatially close to the first key frame in the target video can be screened out, thereby reducing the computational scale of subsequent processing.
[0183] For example, the preset distance threshold is 6, there is a first key frame F1 in the target video F, and there are multiple key frames G1, G2, G3, etc. in the non-target video G. By calculating the spatial distances between F1 and key frames G1, G2, G3, etc., it is assumed that the distance between F1 and G1 is 5, the distance between F1 and G2 is 8, and the distance between F1 and G3 is 3. Therefore, G1 and G3 are determined as candidate key frames.
[0184] This screening method can significantly reduce the computational scale and complexity of subsequent processing, because only the candidate key frames need to be further matched and optimized, without processing all the key frames in the non-target video. This not only improves processing efficiency, but also improves matching accuracy to a certain extent, because key frames that are closer in space are more likely to contain similar feature points.
[0185] S502: Determine a candidate key frame that has a common viewing area with the first key frame as a second key frame.
[0186] In this step, according to step S501, after the candidate key frames of the first key frame in each target video are determined based on the spatial distance, a candidate key frame having a common viewing area with the first key frame in the target video is further determined as the second key frame.
[0187] In order to determine whether the candidate key frame screened out in step S501 has a common viewing area with the first key frame, any one or combination of a three-dimensional point cloud method or a feature point matching algorithm may be used for identification and determination.
[0188] In a possible implementation, for each candidate key frame, a second key frame can be determined by means of a three-dimensional point cloud. Specifically, each candidate key frame is first three-dimensionally reconstructed to generate corresponding point cloud data. Then, these point cloud data are compared with the point cloud data of the first key frame, and the overlapping parts therebetween are calculated. If the number of overlapping point clouds between the candidate key frame and the first key frame is greater than or equal to a first preset threshold number, it is considered that there is a common view area between them. At this point, the candidate key frame is determined as the second key frame. It is worth noting that the first preset threshold number is determined based on actual conditions, and this application does not make any specific restrictions on this.
[0189] In another possible implementation, for each candidate key frame, a feature point matching algorithm may be used to determine the key frame. If the number of matching feature points between the candidate key frame and the first key frame is greater than or equal to the second preset threshold number, it is considered that there is a common view area between them. At this time, the candidate key frame is determined as the second key frame. It is worth noting that the second preset threshold number is determined according to actual conditions, and this application does not make specific restrictions on this.
[0190] For example, the preset threshold number is 10, there is a first key frame H1 in the target video H, and candidate key frames I1, I2 and I3 are screened out in the non-target video I through step S501. Next, the feature point matching algorithm is used to match H1 with I1, I2 and I3 respectively. If the matching result shows that there are 10 matching feature points between H1 and I1, 15 matching feature points between H1 and I2, and only 2 matching feature points between H1 and I3. According to the number of matching feature points and the preset threshold number, it can be determined that there is a common viewing area between H1 and I1 and I2, but there is no common viewing area between H1 and I3. Therefore, I1 and I2 are determined to be the second key frames.
[0191] In this way, we can further filter out candidate keyframes that have a common view area with the first keyframe, ensuring that these keyframes have a high correlation in both space and features. This not only improves the matching accuracy, but also provides more reliable keyframe pairs for subsequent pose optimization.
[0192] Compared with the full image matching method, the method for determining image pose data provided by the embodiment of the present application can effectively screen out representative key frames by selecting candidate key frames for each first key frame based on spatial distance restrictions, providing a more reliable basis for subsequent image processing and pose data calculation. At the same time, further determining the second key frame that has a common view area with the first key frame can ensure that these key frames have a high correlation in space and features, thereby improving the accuracy and reliability of matching. In addition, since only the selected candidate key frames need to be processed, and there is no need to process all the key frames in the non-target video, this improves the efficiency of pose data processing while reducing the scale of calculation.
[0193] Next, the specific implementation process of step S402 will be discussed in detail. Figure 6 The flowchart of the fifth embodiment of the method for determining image pose data provided in the present application is as follows. Figure 6 As shown, based on any one of the above embodiments, the first key frame and the second key frame in each target key frame set are subjected to posture optimization, and the process of the method for determining the image posture data may include:
[0194] S601: For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point based on the first pose data of the first key frame, the second pose data of the second key frame, the first pixel data corresponding to the physical point in the first key frame, and the second pixel data corresponding to the physical point in the second key frame.
[0195] In this step, after the first key frame and the second key frame in each target key frame are determined to be completed, the three-dimensional position data of each physical point in the common viewing area is determined based on the posture data and the pixel data.
[0196] Among them, a physical point refers to a specific point in the real world or a specific position on the surface of an object. The physical point has a certain position coordinate in three-dimensional space and can be represented by pixel coordinates in different images. Three-dimensional position data refers to the specific position coordinates of a physical point determined in three-dimensional space by posture data and pixel data. The position coordinates are usually composed of three values, which respectively represent the position of the physical point in the X, Y and Z axis directions.
[0197] Specifically, first, for each physical point in the common viewing area in the first key frame and the second key frame, first pixel data of the physical point in the first key frame and second pixel data of the physical point in the second key frame are obtained, wherein the pixel data is the pixel coordinate of the physical point in the image.
[0198] Next, based on the first pose data of the first key frame and the second pose data of the second key frame, the three-dimensional position data of each physical point is determined using triangulation method in combination with the pixel data of each physical point in the first key frame and the second key frame. The triangulation method calculates the position of the physical point in three-dimensional space by using the pose data and pixel data under two different perspectives.
[0199] For example, there is a first key frame J1 in the target video J, and a second key frame K1 in the non-target video K. There is a common view area between the two key frames. The common view area contains a physical point P. The pixel coordinates of the physical point P in the first key frame J1 are (x1, y1), and the pixel coordinates in the second key frame K1 are (x2, y2). At the same time, the pose data of the first key frame J1 is T1, and the pose data of the second key frame K1 is T2. Through triangulation, the position coordinates of the physical point P in three-dimensional space are calculated to be (X, Y, Z).
[0200] S602: Back-projecting the three-dimensional position data to the first key frame to obtain third pixel data, and back-projecting the three-dimensional position data to the second key frame to obtain fourth pixel data.
[0201] In this step, according to step S601, after the three-dimensional position data of each physical point is determined, the three-dimensional position data are respectively back-projected to the first key frame to obtain the third pixel data, and back-projected to the second key frame to obtain the fourth pixel data.
[0202] Among them, back-projection refers to the process of mapping the three-dimensional position data in the three-dimensional space back to the two-dimensional image plane, that is, converting the three-dimensional position data into pixel data in the image.
[0203] Specifically, the back-projection process converts the 3D position data into pixel data in the image based on the pose data of the key frame. However, due to factors such as pose data error, triangulation error, image resolution and pixel accuracy, the back-projection result has certain errors.
[0204] Among them, the pose data error refers to the error in the pose data of the key frame when calculating the 3D position data, which will directly affect the accuracy of the 3D position data. Similarly, in the back-projection process, if the pose data used is inaccurate, it will lead to deviations in the back-projection results.
[0205] Triangulation error refers to the error in the triangulation method itself when calculating three-dimensional position data. In particular, when the viewing angle between key frames is small, the accuracy of triangulation will decrease.
[0206] Image resolution and pixel accuracy refer to the resolution and pixel accuracy of the image, which will also affect the accuracy of the calculation and back-projection of the 3D position data. Lower image resolution will result in larger pixel errors, thus affecting the accuracy of the 3D position data.
[0207] S603: Based on the difference between the first pixel data and the third pixel data, and the difference between the second pixel data and the fourth pixel data corresponding to each physical point, the first pose data and the second pose data are optimized to obtain an optimized first key frame and an optimized second key frame.
[0208] In this step, after obtaining the third pixel data and the fourth pixel data according to step S602, the difference between the first pixel data of each physical point in the first key frame and the third pixel data obtained by back projection, and the difference between the second pixel data in the second key frame and the fourth pixel data obtained by back projection are calculated.
[0209] Among them, the difference is used to quantify the deviation between the actual pixel position of the same physical point in the image and the pixel position obtained by back-projection. For example, the position of the first pixel data of physical point A in the first key frame is (x3, y3), and the position in the third pixel data obtained by back-projection is (x4, y4), then their difference can be expressed as the Euclidean distance between (x3, y3) and (x4, y4). Similarly, the position of the second pixel data of physical point A in the second key frame is (x5, y5), and the position in the fourth pixel data obtained by back-projection is (x6, y6), then their difference can be expressed as the Euclidean distance between (x5, y5) and (x6, y6). It is worth noting that in addition to the Euclidean distance, the difference can also be expressed as the Manhattan distance and the Hamming distance. The way of expressing the difference can be determined according to the actual situation to more accurately reflect the accuracy of the current posture data. This application does not make specific restrictions on this.
[0210] Based on the obtained difference, a preset pose determination algorithm can be used to adjust the pose data of the first key frame and the second key frame. The goal of the preset pose determination algorithm is to minimize the pixel data difference of each physical point in different key frames, that is, to minimize the reprojection error value, thereby optimizing the first pose data and the second pose data, and obtaining the optimized first key frame and the optimized second key frame. Specifically, the preset pose determination algorithm calculates the gradient of the error function based on the current first pose data and the second pose data, and updates the first pose data and the second pose data along the direction of the gradient until the reprojection error value of the video to be processed converges to a smaller value and / or the number of iterations reaches a preset number of iterations. The error function can be expressed as the sum of the reprojection error values of the video to be processed.
[0211] The method for determining image pose data provided in the embodiment of the present application determines the three-dimensional position data of physical points by utilizing the pose data and pixel data of multi-perspective key frames, projects these three-dimensional position data back to each key frame to generate new pixel data, calculates the difference between the newly generated pixel data and the original pixel data, quantifies the deviation of the current pose data, and optimizes it using a preset pose determination algorithm based on these difference data, thereby significantly improving the accuracy of the key frame pose data and reducing the deviation of the three-dimensional position data.
[0212] Next, the specific implementation process of step S302 will be discussed in detail. Figure 7 The flowchart of the sixth embodiment of the method for determining image pose data provided in the present application is as follows. Figure 7 As shown, based on any of the above embodiments, each sub-video is subjected to posture optimization based on the optimized key frames in each sub-video. The process of the method for determining the image posture data may include:
[0213] S701: Performing posture optimization processing on ordinary frames in each sub-video according to the posture data of the optimized key frames in each sub-video to obtain optimized ordinary frames, where ordinary frames are images in the sub-video excluding the key frames.
[0214] In this step, after optimizing the key frames of each sub-video based on the association relationship, the posture optimization processing is performed on the ordinary frames in each sub-video according to the posture data of the optimized key frames in each sub-video to obtain the optimized ordinary frames. Among them, the ordinary frames are images in the sub-video except the key frames.
[0215] After optimizing the key frames of each sub-video based on the association relationship, the pose data of these optimized key frames will be used as a reference for optimizing the pose of ordinary frames. The pose data of key frames has been optimized and has high accuracy and reliability, so it can provide effective guidance for the pose optimization of ordinary frames.
[0216] Specifically, by finding the same or similar image feature points between the ordinary frame and the key frame, the pose data of the ordinary frame is optimized. By matching these feature points, the pose of the ordinary frame can be adjusted more accurately to make it more consistent with the pose data of the optimized key frame. For example, any one or combination of feature extraction algorithms such as scale-invariant feature transformation or fast robust feature can be used to find the corresponding feature points between the ordinary frame and the key frame, and then the pose data of the ordinary frame is optimized through the matching relationship of these feature points. When optimizing the pose data of the ordinary frame, the pose data of the ordinary frame is used as a variable to construct an error function, which is used to represent the sum of the reprojection error values of the video to be processed. Specifically, the preset pose determination algorithm calculates the gradient of the error function based on the pose data of the current ordinary frame, and updates the pose data of the ordinary frame along the direction of the gradient until the reprojection error value of the video to be processed converges to a smaller value and / or the number of iterations reaches a preset number of iterations.
[0217] After obtaining the optimized common frame, whether to continue to perform iterative optimization between the key frame and the common frame is determined according to whether the iteration termination condition is met.
[0218] In a possible implementation, when performing posture optimization processing on ordinary frames in each sub-video, posture iterative optimization can be performed in combination with key frames in each sub-video. The following is an example of posture optimization of key frames and ordinary frames in a sub-video. Specifically, based on the optimized key frames, the posture of ordinary frames is optimized, and after the posture optimization of ordinary frames is completed, the posture of key frames is optimized, and this process is continuously iterated until the iteration termination condition is met, thereby further reducing the posture error and improving the accuracy of the posture data.
[0219] The method for determining image pose data provided in the embodiment of the present application uses the pose data of the optimized key frame as a reference benchmark to accurately adjust the pose data of the ordinary frame, thereby significantly improving the accuracy and consistency of the pose data of the ordinary frame. At the same time, by ensuring that the pose data between the key frame and the ordinary frame are highly consistent, a solid data foundation is provided for the construction of high-precision maps.
[0220] Figure 8 The flowchart of the seventh embodiment of the method for determining image pose data provided in the present application is as follows. Figure 8 As shown, based on any one of the above embodiments, at least two sub-videos obtained by segmenting the video to be processed are subjected to at least one posture optimization iteration until the iteration termination condition is met, and the process of the method for determining the image posture data further includes:
[0221] S801: Using a preset posture determination algorithm, perform posture data optimization processing on each video frame in each sub-video.
[0222] In this step, in order to ensure the accuracy of subsequent key frame selection, the present application adopts a preset posture determination algorithm to optimize the posture data of each video frame in each sub-video.
[0223] Specifically, the initial pose data of each video frame can be obtained in a variety of ways. For example, an inertial measurement unit can be used to record the motion trajectory and posture changes of the acquisition device during the shooting process, or the geographic location information can be obtained through a positioning system such as global navigation. The initial pose data of each video frame provides basic data for the subsequent pose determination algorithm. Based on the initial pose data of each video frame, the pose determination algorithm processes each frame image in each video frame to generate processed image pose data. These processed images will be used as a benchmark for the subsequent recognition of key frames and ordinary frames.
[0224] Among them, the preset posture determination algorithm includes but is not limited to any one or combination of SLAM algorithm, SFM algorithm, etc., and the specific implementation method is not repeated here.
[0225] Optionally, before using a preset posture determination algorithm to optimize the posture data of each video frame in each sub-video, a preprocessing operation may be performed on each sub-video. For example, the preprocessing includes but is not limited to deleting severely blurred images, denoising images, correcting uneven lighting, and adjusting image resolution, etc., any one or a combination of operations to avoid affecting the accuracy of the image posture data.
[0226] The method for determining image pose data provided in the embodiment of the present application optimizes the pose data of each video frame in each sub-video through a preset pose determination algorithm, which can effectively reduce the error caused by inaccurate initial pose data of the video frame, thereby improving the accuracy of key frame selection and further improving the accuracy of image pose data.
[0227] Fig. 9 The flowchart of the eighth embodiment of the method for determining image pose data provided in the present application is as follows. Fig. 9 As shown, based on any one of the above embodiments, at least two sub-videos obtained by segmenting the video to be processed are subjected to at least one posture optimization iteration until the iteration termination condition is met, and the process of the method for determining the image posture data further includes:
[0228] S901: Based on the trajectory information when the video to be processed is collected, the video to be processed is divided into at least two sub-videos.
[0229] In this step, before performing at least one pose optimization iteration on the at least two sub-videos obtained by segmenting the video to be processed until the iteration termination condition is met, it is necessary to pre-segment the video to be processed into at least two sub-videos, so as to efficiently manage and analyze the video data in the subsequent processing process.
[0230] The segmentation method of the video to be processed depends on the trajectory information when the video to be processed is collected, and the geographical length of each sub-video may be consistent or inconsistent.
[0231] In a possible implementation, the video to be processed is segmented according to a predefined geographic length interval, in which case the geographic length of each sub-video remains consistent. Specifically, by calculating the trajectory information of each frame of the video during the video shooting process, when the accumulated geographic length reaches the predefined geographic length interval, the video is automatically segmented into a new sub-video. For example, a sub-video is segmented every 100 meters or 500 meters.
[0232] In another possible implementation, the video to be processed is segmented according to a predefined time interval. Specifically, by recording the timestamp of each frame, when the accumulated time reaches the predefined time interval, the video is automatically segmented into a new sub-video. For example, a sub-video is segmented every 5 minutes or 10 minutes.
[0233] In another possible implementation, the video to be processed is segmented based on scene changes. Specifically, the scene changes in the video are detected by an image processing algorithm, and when the scene changes reach a predefined threshold, the video is automatically segmented into a new sub-video.
[0234] Through the above-mentioned multiple segmentation methods, the video to be processed can be flexibly segmented into at least two sub-videos. These segmentation methods can be based not only on geographical length intervals, but also on various factors such as time intervals and scene changes. Each segmentation method has its own unique advantages, and the specific method to be selected can be determined based on actual application requirements and video acquisition environment. Through reasonable segmentation methods, video data can be better managed and analyzed, providing a reliable data foundation for subsequent processing processes such as posture optimization and three-dimensional reconstruction.
[0235] The method for determining image pose data provided in the embodiment of the present application divides the video to be processed into at least two sub-videos based on the trajectory information when the video to be processed is collected and a reasonable segmentation method, thereby reducing the calculation scale and improving the processing efficiency. At the same time, by dividing the video to be processed into smaller sub-videos, more sophisticated management and analysis can be achieved, thereby optimizing the subsequent processing flow and ensuring that the image pose data processing process is more efficient and reliable.
[0236] Fig.10The flowchart of the ninth embodiment of the method for determining image pose data provided in the present application is as follows. Fig.10 As shown, the process of the method for determining the image pose data includes:
[0237] S1001: Perform at least one pose optimization iteration on the video to be processed until an iteration termination condition is met.
[0238] In this step, the case where the video to be processed satisfies specific conditions and does not need to be segmented to generate at least two sub-videos is mainly considered. The specific conditions include but are not limited to any one or a combination of the geographical length, time length, etc. corresponding to the video to be processed.
[0239] In this case, the pose optimization iteration can be directly performed on the entire video to be processed.
[0240] Specifically, by performing at least one iterative pose optimization process on the video to be processed, the optimal solution is gradually approached until the iteration termination condition is met, thereby ensuring that the pose data of each frame in the video to be processed reaches the required accuracy and consistency.
[0241] In a possible implementation, the process of performing at least one pose optimization iteration on the processed video is as follows: Fig.11 shown. Fig.11 The flowchart of the embodiment 10 of the method for determining image pose data provided in the present application is as follows. Fig.11 As shown, the process of the method for determining the image pose data includes:
[0242] S1101: Optimize the position and posture of key frames in the video to be processed to obtain optimized key frames.
[0243] In this step, when performing pose optimization iterations on the video to be processed, the key frames in the video to be processed are firstly pose optimized to obtain optimized key frames.
[0244] Among them, the key frame is an image in the video to be processed that meets the preset image feature conditions.
[0245] In a possible implementation manner, the preset image feature condition includes at least one of the following conditions:
[0246] The number of feature points in the image is greater than or equal to a preset number;
[0247] The curvature value of the image is greater than or equal to the preset curvature;
[0248] Selected images are spaced at preset geographic length intervals.
[0249] Specifically, the preset image feature conditions are mainly set based on a reference factor or any combination of reference factors among the expressiveness of the image, the mutation characteristics of the image content, and the uniformity of the image in spatial distribution.
[0250] The expressiveness of the image is expressed by the number of feature points in the image, the mutation characteristics of the image content are expressed by the curvature value of the image, and the uniformity of the image distribution in space is achieved by presetting the geographical length interval. For specific content explanations, please refer to the description of the preset image feature conditions in step S301, which will not be repeated here.
[0251] The method for determining image pose data provided in this embodiment can improve the accuracy of key frame selection by identifying and determining key frames based on preset image feature conditions, while simplifying the selection process of ordinary frames. This not only significantly improves the overall processing efficiency, but also enhances the consistency of pose data, providing a solid data foundation for high-precision map construction.
[0252] After the key frame is determined based on the preset image feature conditions, the determined key frame is subjected to posture optimization. The purpose of posture optimization is to improve the accuracy and consistency of the posture data of the key frame.
[0253] In a possible implementation, when performing posture optimization on key frames in a processed video, firstly, significant feature points are extracted from the key frames by a feature detection algorithm, wherein the feature detection algorithm includes but is not limited to any one or a combination of a scale-invariant feature transformation algorithm and an accelerated robust feature algorithm. Then, a feature point matching algorithm is used to find corresponding feature points between different key frames, and an association relationship is established between the key frames, wherein the establishment of the association relationship can be based on whether the number of matched feature points reaches a preset threshold number. Finally, based on the association relationship, a preset posture determination algorithm is used to optimize the posture of the key frames.
[0254] Optionally, in order to further improve the accuracy of the pose data and the precision of key frame determination, the present application uses a preset pose determination algorithm to optimize the initial pose data of all frames in the processed video before performing pose optimization on the key frames in the processed video. Through this preprocessing step, a more accurate initial pose data foundation can be provided for subsequent key frame identification and optimization, thereby further improving the accuracy of key frame selection and the effect of pose optimization.
[0255] S1102: Based on the optimized key frames, optimize the common frames in the video to be processed.
[0256] In this step, ordinary frames in the video to be processed are optimized according to the optimized key frames obtained in step S1101.
[0257] Among them, the common frames are images other than the key frames in the video to be processed.
[0258] Specifically, the optimized keyframes provide a more accurate reference point or benchmark. By comparing and correcting with these optimized keyframes, the pose of the ordinary frame can be adjusted to make it more consistent and coordinated with the pose of the keyframe.
[0259] First, the difference in pose data between the common frame and its adjacent optimized key frame is determined. Then, the pose of the common frame is adjusted according to the determined pose data difference to ensure that the pose of the common frame is consistent with the pose of the key frame.
[0260] The method for determining image pose data provided in this embodiment can significantly improve the accuracy of the pose data of the entire video to be processed by iteratively optimizing the key frames and ordinary frames in the video to be processed. Specifically, the optimization of the key frames provides a more accurate reference point, making the subsequent optimization of ordinary frames more effective, thereby reducing pose errors. At the same time, the iterative optimization method improves processing efficiency and reduces the amount of calculation for independent optimization of each frame.
[0261] S1002: Acquire posture data of images in the video to be processed based on the video to be processed after posture optimization.
[0262] In this step, according to step S1001, after performing at least one posture optimization iterative process on the video to be processed until the iteration termination condition is met, the posture data of the image in the video to be processed is obtained based on the video to be processed after the posture optimization.
[0263] Specifically, the pose-optimized video results provide accurate pose information. Based on the pose-optimized video, precise alignment and pose determination of images in the video can be achieved, thereby providing effective data support for the three-dimensional construction of high-precision maps.
[0264] In a possible implementation, the iteration termination condition includes at least one of the following:
[0265] The number of iterations reaches the preset number of iterations;
[0266] The reprojection error value of the video to be processed is less than or equal to the preset error value.
[0267] Among them, stopping the iterative optimization when the number of iterations reaches the preset number of iterations helps to avoid the algorithm for pose optimization from falling into an infinite loop and ensure the effective use of computing resources. Stopping the iterative optimization when the reprojection error value of the video to be processed is less than or equal to the preset error value helps to ensure that the pose data has reached the required accuracy, thereby ensuring the accuracy and reliability of the optimization results.
[0268] By setting the iteration termination condition, the iteration process can be effectively controlled to ensure that high-precision pose data is obtained within a reasonable computing time.
[0269] Specifically, the pose of the key frames in the video to be processed is first optimized to obtain the optimized key frames, and then the pose of the ordinary frames is optimized based on these optimized key frames, and this process is continuously iterated to gradually reduce the pose error and improve the accuracy of the pose data until the iteration termination condition is met. After the iteration termination condition is met, the pose data of all images in the video to be processed are finally obtained by comprehensively analyzing and calculating the pose data of the optimized key frames and ordinary frames, thereby ensuring that the pose data of each frame has been corrected and optimized for multiple times, thereby improving the accuracy and consistency of the pose data of the video to be processed.
[0270] The method for determining image pose data provided in the embodiment of the present application obtains the pose data of the image in the video to be processed by performing at least one pose optimization iteration on the video to be processed until the iteration termination condition is met. The method ensures that the pose data of each frame of the image is corrected and optimized multiple times, significantly improving the accuracy and consistency of the pose data while improving the processing efficiency.
[0271] Fig.12 A schematic diagram of a framework of a method for determining image pose data provided in an embodiment of the present application. Fig.12 As shown, the framework of the method for determining image pose data includes an image data acquisition module 1201, a data cutting module 1202, a single-segment optimization module 1203, a key frame judgment module 1204 and a multi-pass association optimization module 1205.
[0272] Among them, the image data acquisition module 1201 is used to obtain trajectory data 12011. Specifically, the image data acquisition module 1201 can extract trajectory data from various data sources, including but not limited to any one or combination of global navigation positioning data, inertial measurement unit data, visual odometer data, etc. These trajectory data provide the movement path and position information of the object in space, which can provide important basic data for subsequent image processing and posture data calculation. By integrating and analyzing these trajectory data, the shooting position and angle of the image can be determined more accurately, thereby improving the accuracy and reliability of the overall processing.
[0273] The data segmentation module 1202 is used to segment the trajectory data 12011 acquired by the image data acquisition module 1201, so as to obtain a plurality of single-trip data, including single-trip 12021, single-trip 12022, single-trip 12023 and single-trip 12024. Each single-trip data includes multiple frames of images. The specific implementation of this module can refer to step S901, which will not be repeated here.
[0274] The single-segment optimization module 1203 is used to optimize multiple single-trip data obtained by the data segmentation module 1202, including single-trip pose optimization 12031 (vision + inertial measurement unit), single-trip pose optimization 12032 (vision + inertial measurement unit), single-trip pose optimization 12033 (vision + inertial measurement unit), single-trip pose optimization 12034 (vision + inertial measurement unit) and single-trip pose optimization 12035 (vision + inertial measurement unit). Specifically, the single-segment optimization module 1203 performs fine pose optimization processing on each single-trip data segment by combining visual information and inertial measurement unit data. By performing independent pose optimization on each single-trip data segment, the cumulative error can be effectively reduced, and the accuracy and consistency of the overall pose data can be improved, thereby providing a more solid foundation for the subsequent determination of key frames. The specific implementation of this module can refer to step S801, which will not be repeated here.
[0275] The key frame judgment module 1204 is used to perform key frame judgment 12041, and the judgment conditions based on it include but are not limited to any one or combination of trajectory curvature, distance from the previous key frame, and feature richness. Specifically, the key frame judgment module 1204 identifies significant change points in the motion path by analyzing the trajectory curvature, and these change points usually correspond to important turning points or feature points in the scene. In addition, the key frame judgment module 1204 also considers the distance between the current frame and the previous key frame image to reduce the calculation scale. Finally, the key frame judgment module 1204 also judges the feature richness of the image, such as the number and distribution of feature points, to ensure that the selected key frame contains enough detail information. By combining these judgment conditions, the key frame judgment module 1204 can accurately select representative key frames, provide a reliable benchmark for subsequent posture data calculations, and improve accuracy and efficiency. The specific implementation of this module can refer to step S301, which will not be repeated here.
[0276] The single-pass multi-pass association optimization module 1205 includes a visual feature point constraint 12051 and a relative pose constraint 12052. Specifically, the visual feature point constraint 12051 ensures the consistency of these feature points in the global coordinate system by matching and associating the same visual feature points between different single-pass data, thereby improving the coherence and accuracy of the overall trajectory. The relative pose constraint 12052 ensures the correct alignment and connection of each single-pass data in space by analyzing and optimizing the relative pose relationship between different single-pass data. This multi-level association optimization can not only effectively reduce the cumulative error, but also improve the accuracy of the image pose data. By comprehensively applying the visual feature point constraint 12051 and the relative pose constraint 12052, the single-pass multi-pass association optimization module 1205 can achieve global optimization of multiple single-pass data, ensuring that the pose data finally generated has high precision and high consistency. The specific implementation of this module can refer to steps S201 to S202, which will not be repeated here.
[0277] The present application scheme greatly reduces the scale of calculation by adopting the key frame idea of single pass first and then multiple passes, and adopts the divide and conquer idea, thereby greatly reducing the time required for offline calculation. At the same time, in terms of global image data alignment and improving positioning accuracy, the single-pass and multiple-pass iterative optimization calculation scheme proposed in this application can also effectively solve the problem of misalignment of multiple-pass image data. By fixing the key frame and optimizing the ordinary frame in the single-pass optimization stage, and then fixing the ordinary frame and optimizing the key frame in the multi-pass optimization stage, the posture error can be gradually reduced, ensuring the consistency and coherence of the posture between different single-pass data, and improving the alignment accuracy and positioning accuracy of the global image data.
[0278] Fig.13 A flowchart of a key frame selection strategy provided in an embodiment of the present application. Fig.13 As shown, the process of the key frame selection strategy may include:
[0279] S1301: Single trip optimization.
[0280] In this step, before determining the key frame, each single-pass data is first optimized. By combining visual information and inertial measurement unit data, a preset pose determination algorithm is applied to perform fine pose optimization on the single-pass data to ensure that the pose information of each single-pass data is more accurate and reliable.
[0281] It is worth noting that each sub-video is a single-pass data, and a single-pass data includes multiple frames of images. Therefore, multiple sub-videos constitute multiple-pass data. The specific implementation of this step can refer to step S801, which will not be repeated here.
[0282] S1302: SFM initial point cloud.
[0283] In this step, after the single-pass data is optimized according to step S1301, an initial point cloud is generated according to the SFM algorithm. Specifically, the feature points in each frame image are analyzed and the three-dimensional position data of these feature points are calculated, thereby generating an initial three-dimensional point cloud.
[0284] S1303: Point cloud clustering.
[0285] In this step, after applying the SFM algorithm to the initial point cloud in step S1302, clustering is performed on the generated initial point cloud. Specifically, the feature points in the point cloud are grouped by a preset clustering algorithm to identify different objects or scene parts.
[0286] S1304: Determine candidate key frames according to the spatial distance.
[0287] In this step, according to step S1303, after the point cloud clustering is completed, a candidate key frame is determined for the first key frame according to the spatial distance. The specific implementation of this step can refer to step S501, which will not be repeated here.
[0288] In a possible implementation, when determining a candidate key frame for the first key frame, first draw a circle with the position of the feature point in the first key frame as the center and a preset length as the radius. Then, the key frame falling within the drawn circle is determined as the candidate key frame of the first key frame. This method ensures that the selected candidate key frame has a high spatial correlation with the first key frame through the constraint of the spatial position, thereby improving the accuracy and reliability of the matching.
[0289] It is worth noting that the preset length is determined based on actual conditions and practical experience, and this application does not make any specific limitations on this.
[0290] S1305: Determine a second key frame according to the common viewing area.
[0291] In this step, according to step S1304, after determining the candidate key frame based on the spatial distance, the second key frame is further determined from the candidate key frames according to the common view area. The specific implementation of this step can refer to step S502, which will not be repeated here.
[0292] S1306: Execute multiple passes of image matching.
[0293] In this step, according to step S1305, after the second key frame is determined according to the common view area, multiple passes of image matching are performed, that is, a target key frame set is constructed. The specific implementation of this step can refer to step S401, which will not be repeated here.
[0294] Fig.14A factor graph for solving the image pose data optimization problem provided in the embodiment of the present application. Fig.14 As shown in FIG. 1 , the factor graph is mainly used to represent the process of single-pass optimization and multi-pass optimization loop iteration. The following will take two sub-videos as examples for explanation.
[0295] If only single-pass optimization is considered, there will be a problem of visual misalignment between passes. If only multi-pass optimization based on key frames is considered, although the key frames are aligned, there will still be alignment errors in the ordinary frames. In order to solve this problem, the present application proposes a method of single-pass optimization and multi-pass optimization cyclic iteration, and constructs a global optimization problem based on the bundle adjustment method. In the global optimization process, the bundle adjustment method iteratively optimizes the pose data to minimize the sum of the reprojection error values, thereby ensuring image alignment within a single-pass data and alignment between multi-pass data, and ultimately achieving accurate alignment and high-precision positioning of global image data.
[0296] Single-pass optimization and multi-pass optimization interact through loop iteration. In each iteration, multi-pass optimization is first performed to fix the normal frame and optimize the key frame, and then single-pass optimization is performed to fix the key frame and optimize the normal frame. Through this alternating optimization method, the pose error can be gradually reduced and the accuracy and consistency of the pose data can be improved.
[0297] Specifically, in the single-pass optimization stage, based on the relative pose relationship between the ordinary frame and the key frame in each single-pass data, the pose of the ordinary frame is adjusted through a preset pose determination algorithm to keep it consistent with the fixed key frame. In the multi-pass optimization stage, based on the key frame relationship between different single-pass data, the pose of the key frame is adjusted through a preset pose determination algorithm to keep it consistent with the fixed ordinary frame.
[0298] like Fig.14 As shown, according to the preset image feature conditions, the corresponding key frame in the first sub-video is key frame 1, the corresponding key frame in the second sub-video is key frame 2, the corresponding ordinary frames in the first sub-video are ordinary frames 1 and ordinary frames 2, the corresponding ordinary frames in the second sub-video are ordinary frames 3 and ordinary frames 4, the feature point in the first sub-video is feature point 1, and the feature point in the second sub-video is feature point 2.
[0299] In the single-pass optimization stage, key frame 1 and key frame 2 are fixed, that is, key frame 1 and key frame 2 are not optimized, but the preset pose determination algorithm is applied to optimize ordinary frame 1 and ordinary frame 2, ordinary frame 3 and ordinary frame 4 respectively. In this way, the pose data of ordinary frames can be optimized while keeping the key frames stable, thereby reducing the cumulative error and improving the pose accuracy of single-pass data.
[0300] In the multi-pass optimization stage, ordinary frames 1, 2, 3 and 4 are fixed, that is, ordinary frames 1, 2, 3 and 4 are not optimized, but the preset pose determination algorithm is applied to optimize key frames 1 and 2. In this way, the pose data of the key frames can be optimized while keeping the ordinary frames stable, thereby ensuring the pose consistency between different single-pass data.
[0301] Fig.15 This is a schematic diagram of the structure of a first embodiment of a device for determining image posture data provided by the present application. Fig.15 As shown, the apparatus 1500 for determining image pose data includes:
[0302] The processing module 1501 is used to perform at least one posture optimization iteration on at least two sub-videos obtained by segmenting the video to be processed until an iteration termination condition is satisfied;
[0303] An acquisition module 1502 is used to acquire the pose data of the image in the video to be processed based on each sub-video after the pose is optimized;
[0304] The processing module 1501 is used at least in each pose optimization iteration to:
[0305] Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video, where the key frames in each sub-video are images in the sub-video that meet preset image feature conditions;
[0306] Based on the optimized key frames in each sub-video, the pose of each sub-video is optimized.
[0307] Optionally, the iteration termination condition includes at least one of the following:
[0308] The number of iterations reaches the preset number of iterations;
[0309] The reprojection error value of the video to be processed is less than or equal to the preset error value.
[0310] Optionally, the preset image feature condition includes at least one of the following conditions:
[0311] The number of feature points in the image is greater than or equal to a preset number;
[0312] The curvature value of the image is greater than or equal to the preset curvature;
[0313] There are at least two images in the common viewing area, and the at least two images are in different sub-videos;
[0314] Selected images are spaced at preset geographic length intervals.
[0315] Optionally, the processing module 1501 is further configured to:
[0316] Each sub-video is used as a target video, and based on the key frames in the target video and the non-target video, a target key frame set corresponding to each target video is determined; the target key frame set corresponding to each target video includes a first key frame and a second key frame, the first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a common viewing area with the first key frame; the non-target video is a sub-video other than the target video in at least two sub-videos;
[0317] The pose of the first key frame and the second key frame in each target key frame set is optimized.
[0318] Optionally, the processing module 1501 is further configured to:
[0319] A key frame in the non-target video whose spatial distance with the first key frame is less than or equal to a preset distance threshold is determined as a candidate key frame;
[0320] A candidate key frame having a common viewing area with the first key frame is determined as the second key frame.
[0321] Optionally, the processing module 1501 is further configured to:
[0322] For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point based on the first pose data of the first key frame, the second pose data of the second key frame, the first pixel data corresponding to the physical point in the first key frame, and the second pixel data corresponding to the physical point in the second key frame;
[0323] Back-projecting the three-dimensional position data to the first key frame to obtain third pixel data, and back-projecting the three-dimensional position data to the second key frame to obtain fourth pixel data;
[0324] Based on the difference between the first pixel data and the third pixel data, and the difference between the second pixel data and the fourth pixel data corresponding to each physical point, the first pose data and the second pose data are optimized to obtain an optimized first key frame and an optimized second key frame.
[0325] Optionally, the processing module 1501 is further configured to:
[0326] According to the pose data of the optimized key frames in each sub-video, the pose optimization processing is performed on the ordinary frames in each sub-video to obtain the optimized ordinary frames, where the ordinary frames are images in the sub-video excluding the key frames.
[0327] Optionally, the processing module 1501 is further configured to:
[0328] A preset posture determination algorithm is used to optimize the posture data of each video frame in each sub-video.
[0329] Optionally, the processing module 1501 is further configured to:
[0330] Based on the trajectory information when the video to be processed is collected, the video to be processed is divided into at least two sub-videos.
[0331] The device for determining image pose data provided in the embodiment of the present application can be used to execute the method for determining image pose data in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.
[0332] Fig.16 This is a schematic diagram of the structure of a second embodiment of a device for determining image posture data provided by the present application. Fig.16 As shown, the apparatus 1600 for determining image pose data includes:
[0333] The processing module 1601 is used to perform at least one posture optimization iterative process on the video to be processed until an iteration termination condition is met;
[0334] An acquisition module 1602 is used to acquire the pose data of the image in the video to be processed based on the video to be processed after the pose is optimized;
[0335] The processing module 1601 is used in each pose optimization iteration to:
[0336] Perform pose optimization on the key frames in the video to be processed to obtain optimized key frames;
[0337] Based on the optimized key frames, ordinary frames in the video to be processed are optimized; wherein the key frames are images in the video to be processed that meet preset image feature conditions, and the ordinary frames are images in the video to be processed except the key frames.
[0338] Optionally, the iteration termination condition includes at least one of the following:
[0339] The number of iterations reaches the preset number of iterations;
[0340] The reprojection error value of the video to be processed is less than or equal to the preset error value.
[0341] Optionally, the preset image feature condition includes at least one of the following conditions:
[0342] The number of feature points in the image is greater than or equal to a preset number;
[0343] The curvature value of the image is greater than or equal to the preset curvature;
[0344] Selected images are spaced at preset geographic length intervals.
[0345] The device for determining image pose data provided in the embodiment of the present application can be used to execute the method for determining image pose data in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.
[0346] Fig.17 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.17 As shown, the computer device 1700 may specifically include a transceiver 1701, a processor 1702 and a memory 1703. The transceiver 1701 is used to implement data exchange with other devices, and the memory 1703 stores computer execution instructions; the processor 1702 executes the computer execution instructions stored in the memory 1703 to implement the method for determining the image pose data in the above embodiment.
[0347] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method for determining image pose data in the above embodiment.
[0348] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for determining image pose data provided by any one of the above embodiments.
[0349] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0350] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for determining image pose data, It is characterized in that include: For at least two sub-videos obtained by segmenting the video to be processed, at least one posture optimization iteration is performed until an iteration termination condition is satisfied, and the posture data of the image in the video to be processed is obtained based on each sub-video after the posture optimization; wherein each posture optimization iteration at least includes: Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video, wherein the key frame in each sub-video is an image in the sub-video that meets a preset image feature condition; Based on the optimized key frames in each sub-video, the pose of each sub-video is optimized; The step of performing posture processing on the key frames in each sub-video based on the association relationship between the key frames in different sub-videos to obtain the optimized key frames in each sub-video includes: Taking each sub-video as a target video, respectively, and determining a target key frame set corresponding to each target video based on key frames in the target video and non-target videos; the target key frame set corresponding to each target video includes a first key frame and a second key frame, the first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a co-viewing area with the first key frame; the non-target video is a sub-video other than the target video in at least two sub-videos; For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point; Calculating a reprojection error value of the physical point; wherein the reprojection error value refers to the distance between the three-dimensional position data of the physical point and the corresponding physical point in the first key frame after back-projecting the three-dimensional position data of the physical point to the first key frame, and the distance between the three-dimensional position data of the physical point and the corresponding physical point in the second key frame after back-projecting the three-dimensional position data of the physical point to the second key frame; The first key frame and the second key frame are optimized based on the reprojection error value.
2. The method according to claim 1, It is characterized in that The iteration termination condition includes at least one of the following: The number of iterations reaches the preset number of iterations; The reprojection error value of the video to be processed is less than or equal to a preset error value.
3. The method according to claim 1, It is characterized in that The preset image feature condition includes at least one of the following conditions: The number of feature points in the image is greater than or equal to a preset number; The curvature value of the image is greater than or equal to the preset curvature; There are at least two images in the common viewing area, and the at least two images are respectively in different sub-videos; Selected images at intervals of a preset geographic length.
4. The method according to claim 1, It is characterized in that Determining a target key frame set corresponding to each target video based on the key frames in the target video and the non-target video includes: Determine a key frame in the non-target video whose spatial distance with the first key frame is less than or equal to a preset distance threshold as a candidate key frame; A candidate key frame that has a common viewing area with the first key frame is determined as the second key frame.
5. The method according to claim 1, It is characterized in that The step of determining the three-dimensional position data of each physical point in the common viewing area in the first key frame and the second key frame includes: For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point based on the first pose data of the first key frame, the second pose data of the second key frame, the first pixel data corresponding to the physical point in the first key frame, and the second pixel data corresponding to the physical point in the second key frame; The calculating the reprojection error value of the physical point and optimizing the first key frame and the second key frame based on the reprojection error value includes: Back-projecting the three-dimensional position data to the first key frame to obtain third pixel data, and back-projecting the three-dimensional position data to the second key frame to obtain fourth pixel data; Based on the difference between the first pixel data and the third pixel data, and the difference between the second pixel data and the fourth pixel data corresponding to each physical point, the first posture data and the second posture data are optimized to obtain an optimized first key frame and an optimized second key frame.
6. The method according to claim 1, It is characterized in that The step of optimizing the posture of each sub-video based on the optimized key frame in each sub-video includes: According to the pose data of the optimized key frames in each sub-video, pose optimization processing is performed on the ordinary frames in each sub-video to obtain optimized ordinary frames, where the ordinary frames are images in the sub-video excluding the key frames.
7. The method according to any one of claims 1 to 6, It is characterized in that The at least two sub-videos obtained by segmenting the processed video are subjected to at least one posture optimization iteration until an iteration termination condition is satisfied, and the method further comprises: A preset posture determination algorithm is used to optimize the posture data of each video frame in each sub-video.
8. The method according to any one of claims 1 to 6, It is characterized in that The at least two sub-videos obtained by segmenting the processed video are subjected to at least one posture optimization iteration until an iteration termination condition is satisfied, and the method further comprises: Based on the trajectory information when the video to be processed is collected, the video to be processed is divided into the at least two sub-videos.
9. The method according to claim 1, It is characterized in that include: Perform at least one posture optimization iteration on the video to be processed until an iteration termination condition is met, and obtain the posture data of the image in the video to be processed based on the video to be processed after the posture optimization; wherein each posture optimization iteration includes: Performing posture optimization on the key frames in the video to be processed to obtain optimized key frames; Based on the optimized key frames, optimizing the common frames in the video to be processed; The key frame is an image in the video to be processed that meets a preset image feature condition, and the common frame is an image in the video to be processed except the key frame.
10. The method according to claim 9, It is characterized in that The iteration termination condition includes at least one of the following: The number of iterations reaches the preset number of iterations; The reprojection error value of the video to be processed is less than or equal to a preset error value.
11. The method according to claim 9, It is characterized in that The preset image feature condition includes at least one of the following conditions: The number of feature points in the image is greater than or equal to a preset number; The curvature value of the image is greater than or equal to the preset curvature; Selected images are spaced at preset geographic length intervals.
12. A device for determining image pose data, It is characterized in that include: A processing module, used for performing at least one posture optimization iteration on at least two sub-videos obtained by segmenting the video to be processed until an iteration termination condition is satisfied; An acquisition module, used for acquiring the pose data of the image in the video to be processed based on each sub-video after the pose is optimized; Wherein, the processing module is used at least in each posture optimization iteration to: Based on the correlation between key frames in different sub-videos, the key frames in each sub-video are processed in position and posture to obtain optimized key frames in each sub-video, wherein the key frame in each sub-video is an image in the sub-video that meets a preset image feature condition; Based on the optimized key frames in each sub-video, the pose of each sub-video is optimized; The processing module is specifically used to take each sub-video as a target video, and determine a target key frame set corresponding to each target video based on key frames in the target video and non-target videos; the target key frame set corresponding to each target video includes a first key frame and a second key frame, the first key frame is a key frame in the target video, and the second key frame is a key frame in the non-target video that has a common viewing area with the first key frame; the non-target video is a sub-video other than the target video in at least two sub-videos; For each physical point in the common viewing area in the first key frame and the second key frame, determine the three-dimensional position data of the physical point; Calculating a reprojection error value of the physical point; wherein the reprojection error value refers to the distance between the three-dimensional position data of the physical point and the corresponding physical point in the first key frame after back-projecting the three-dimensional position data of the physical point to the first key frame, and the distance between the three-dimensional position data of the physical point and the corresponding physical point in the second key frame after back-projecting the three-dimensional position data of the physical point to the second key frame; The first key frame and the second key frame are optimized based on the reprojection error value.
13. The device according to claim 12, It is characterized in that include: A processing module, used for performing at least one posture optimization iterative process on the video to be processed until an iteration termination condition is met; An acquisition module, used for acquiring the posture data of the image in the video to be processed based on the video to be processed after the posture optimization; Wherein, the processing module is used in each posture optimization iteration to: Performing posture optimization on the key frames in the video to be processed to obtain optimized key frames; Based on the optimized key frames, ordinary frames in the video to be processed are optimized; wherein the key frames are images in the video to be processed that meet preset image feature conditions, and the ordinary frames are images in the video to be processed except the key frames.
14. A computer device, It is characterized in that include: A transceiver, a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for determining image pose data according to any one of claims 1 to 11.
15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for determining image pose data according to any one of claims 1 to 11.
16. A computer program product, It is characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for determining image pose data according to any one of claims 1 to 11.
Citation Information
Patent Citations
Image automatic labeling method and device based on visual SLAM, and storage medium
CN113362363A
Key frame pose optimization visual SLAM method based on time delay feature regression, storage medium and equipment
CN115937011A
Image rendering method and device, electronic equipment and storage medium
CN117079172A
Dynamic scene self-positioning and online high-precision three-dimensional reconstruction method
CN117115343A
Image rendering method and device, electronic equipment and storage medium
CN117788659A
