Three-dimensional scene hole filling method, device, equipment and storage medium

Abstract

The application discloses a three-dimensional scene hole filling method and device, equipment and storage medium, and relates to the technical field of three-dimensional reconstruction. The technical scheme provided by the application comprises the following steps: in the case that the overall three-dimensional map obtained through three-dimensional reconstruction does not satisfy a preset completeness condition, obtaining a recovery frame and a missing frame from a plurality of image frames used for reconstructing the overall three-dimensional map; based on the positions of the missing frame and each recovery frame, adding the missing frame and the recovery frame in the corresponding neighborhood into a to-be-solved group; determining the matching results between each image frame in the to-be-solved group, and dividing two image frames matched according to the matching results into the same connected group; performing three-dimensional reconstruction according to the image frames in the connected group to obtain a local three-dimensional map, aligning the local three-dimensional map with the overall three-dimensional map, and obtaining a target three-dimensional map. Through the above technical means, the missing frame and the corresponding pose are used to complete the holes in the three-dimensional map, and the reconstruction efficiency and success rate of the three-dimensional scene are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional scene hole filling method, device, equipment and storage medium. BACKGROUND

[0002] When the unmanned equipment carries out high-altitude mapping, it can navigate according to a pre-planned flight route, take images of a region to be mapped, determine the shooting poses of the images through a real-time three-dimensional reconstruction algorithm, and construct a three-dimensional map of the region to be mapped according to the images and the corresponding shooting poses. If the region to be mapped has a region with serious occlusion or complex texture, the three-dimensional map is likely to be incomplete.

[0003] In the prior art, all regions of the region to be mapped are reconstructed through an offline method to reconstruct a complete three-dimensional map. However, it takes a long time to reconstruct a complete three-dimensional map, and the reconstruction efficiency is low. Moreover, in the case of using the same images, the reconstruction cannot guarantee success, and in the case of reconstruction failure, it may be necessary to take aerial photographs by the unmanned equipment again, which seriously affects the reconstruction efficiency. SUMMARY

[0004] The present application provides a three-dimensional scene hole filling method, device, equipment and storage medium, which recovers the poses of missing frames through the constraint relationship between the missing frames and the recovered frames, fills the holes in the three-dimensional map through the missing frames and the corresponding poses, does not need to reconstruct a complete three-dimensional map, avoids the case of reconstruction failure, effectively improves the reconstruction efficiency and success rate of the three-dimensional scene, and solves the problem of low reconstruction efficiency of the three-dimensional scene in the prior art.

[0005] In a first aspect, the present application provides a three-dimensional scene hole filling method, comprising:

[0006] In the case where the overall three-dimensional map obtained through three-dimensional reconstruction does not satisfy a preset completeness condition, recovering frames and missing frames are obtained from a plurality of image frames used to reconstruct the overall three-dimensional map;

[0007] Based on the positions of the missing frames and each of the recovered frames, the missing frames and the recovered frames in the corresponding neighborhood are added to a to-be-solved group;

[0008] The matching results between each image frame in the to-be-solved group are determined, and two image frames that match each other are divided into the same connected group according to the matching results;

[0009] The image frames in the connected group are used for three-dimensional reconstruction to obtain a local three-dimensional map, and the local three-dimensional map is aligned with the overall three-dimensional map to obtain a target three-dimensional map.

[0010] In a second aspect, the present application provides a three-dimensional scene hole filling device, comprising:

[0011] a scene reconstruction module configured to acquire a recovery frame and a missing frame from a plurality of image frames used to reconstruct an overall three-dimensional map obtained through three-dimensional reconstruction, in a case where the overall three-dimensional map does not satisfy a preset completeness condition;

[0012] a to-be-solved group generation module configured to add the missing frame and recovery frames in a corresponding neighborhood to a to-be-solved group based on positions of the missing frame and the recovery frames;

[0013] a connected group generation module configured to determine a matching result between image frames in the to-be-solved group, and divide two image frames that match into a same connected group according to the matching result;

[0014] a scene hole filling module configured to perform three-dimensional reconstruction on image frames in the connected group, to obtain a local three-dimensional map, and to align the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map.

[0015] In a third aspect, the present application provides a three-dimensional scene hole filling device, comprising:

[0016] one or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional scene hole filling method according to the first aspect.

[0017] In a fourth aspect, the present application provides a storage medium containing computer executable instructions, when the computer executable instructions are executed by a computer processor, the computer executable instructions are used to execute the three-dimensional scene hole filling method according to the first aspect.

[0018] In the present application, the lost frames and the recovered frames are obtained, the lost frames and the recovered frames are added into a to-be-solved group based on the positions of the lost frames and the recovered frames, and the lost frames and the recovered frames in the corresponding neighborhood are added into the to-be-solved group, so that the lost frames and the recovered frames having the correlation are quickly searched out from the to-be-solved group. According to the matching results between the image frames in the to-be-solved group, it is determined that the two image frames having the matching result have the correlation, and the image frames having the correlation are put into the same connected group, so as to construct the local three-dimensional map associated with the whole three-dimensional map by the image frames in the connected group. After the local three-dimensional map is aligned with the whole three-dimensional map, the local three-dimensional map can be used to fill the holes of the whole three-dimensional map, and a highly complete target three-dimensional map is obtained, and the hole filling operation of the whole three-dimensional map is realized. Through the above technical means, the lost frames and the recovered frames having the correlation are searched out from the image frames, and the local three-dimensional map associated with the whole three-dimensional map is constructed based on the lost frames and the recovered frames having the correlation, so that the local three-dimensional map can be aligned into the whole three-dimensional map, the holes of the whole three-dimensional map are filled, and the completeness of the three-dimensional scene is improved. When the hole filling operation is performed on the hole scene, only a small amount of computing resources and waiting time are required to construct the local three-dimensional map, the complete three-dimensional map does not need to be reconstructed, and the situation that the reconstruction fails is avoided, the reconstruction efficiency and the success rate of the three-dimensional scene are effectively improved, and the problem that the reconstruction efficiency of the three-dimensional scene is low in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a three-dimensional scene hole filling method provided by an embodiment of the present application;

[0020] Figure 2 is a top view schematic diagram of a whole three-dimensional map provided by an embodiment of the present application;

[0021] Figure 3 is a flowchart of adding the lost frames and the recovered frames in the corresponding neighborhood into a to-be-solved group provided by an embodiment of the present application;

[0022] Figure 4 is a flowchart of determining a matched image pair in a to-be-solved group provided by an embodiment of the present application;

[0023] Figure 5 is a flowchart of obtaining a third image frame of a lost frame provided by an embodiment of the present application;

[0024] Figure 6 is a flowchart of determining a matching result of an image pair provided by an embodiment of the present application;

[0025] Figure 7 is a flowchart of dividing a connected group according to a matching result provided by an embodiment of the present application;

[0026] Figure 8is a schematic diagram of a tree structure provided by an embodiment of the present application;

[0027] Figure 9 is a flowchart of constructing a local three-dimensional map through an effective connected group provided by an embodiment of the present application;

[0028] Figure 10 is a flowchart of determining a relative transformation parameter provided by an embodiment of the present application;

[0029] Figure 11 is a structural schematic diagram of a three-dimensional scene hole filling device provided by an embodiment of the present application;

[0030] Figure 12 is a structural schematic diagram of a three-dimensional scene hole filling device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes specific embodiments of the present application in conjunction with the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, but not all contents. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted by flowcharts. Although the flowcharts describe each operation (or step) as a sequential process, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0033] The three-dimensional scene hole filling method provided in this embodiment can be executed by a three-dimensional scene hole filling device. The three-dimensional scene hole filling device can be implemented in a software and / or hardware manner, and can be composed of two or more physical entities or one physical entity. For example, the three-dimensional scene hole filling device can be an unmanned device for collecting image frames, and can also be a remote control device for controlling the unmanned device or a server wirelessly connected to the unmanned device. The unmanned device refers to a device such as a drone that can automatically execute a preset task.

[0034] The three-dimensional scene hole filling device is installed with at least one type of operating system. The three-dimensional scene hole filling device can install at least one application program based on the operating system. The application program can be an application program provided by the operating system, or an application program downloaded from a third-party device or server. In this embodiment, the three-dimensional scene hole filling device has at least an application program for executing the three-dimensional scene hole filling method.

[0035] For ease of understanding, the server is taken as an example to describe the subject of executing the three-dimensional scene hole filling method in this embodiment.

[0036] In one embodiment, when the unmanned device navigates according to a pre-planned flight route, the unmanned device captures images of the area to be surveyed according to a preset shooting strategy, and sends the images collected in real time to the server. The server determines the shooting pose of the image captured by the unmanned device through a real-time three-dimensional reconstruction algorithm, and constructs a three-dimensional map of the area to be surveyed according to the image and the corresponding shooting pose. In the real-time three-dimensional reconstruction algorithm, due to the constraints of image resolution and the number of feature points, if the occlusion of some scenes in the image is too serious or the texture is not ideal, the shooting pose of the image cannot be determined, and the three-dimensional scene reconstruction is incomplete. The prior art generally uses an offline method to reconstruct the three-dimensional scene of the entire area to be surveyed based on all the images of the area to be surveyed. However, due to the constraints of image resolution and the number of feature points, the new mapping cannot guarantee the success of the mapping. If the new mapping fails, the unmanned device will re-shoot the images of the area to be surveyed until the mapping is successful. In the prior art, whether it is new mapping or re-shooting images for new mapping, a lot of time is needed, the reconstruction efficiency is low, and the success rate of reconstruction cannot be guaranteed.

[0037] To solve the problem of low three-dimensional scene reconstruction efficiency in the prior art, the embodiment provides a three-dimensional scene hole filling method.

[0038] Figure 1 A flowchart of the three-dimensional scene hole filling method provided in the embodiment of the application is given. Referring to FIG. 1, the three-dimensional scene hole filling method specifically includes the following steps. Figure 1

[0039] ​S110, in the case that the overall three-dimensional map obtained by three-dimensional reconstruction does not satisfy a preset completeness condition, obtaining a recovery frame and a missing frame from a plurality of image frames used to reconstruct the overall three-dimensional map.

[0040] In this embodiment, the plurality of image frames refer to all image frames captured by the unmanned device on the to-be-mapped region when the unmanned device sails according to the pre-planned flight route. The overall three-dimensional map refers to a three-dimensional map of the to-be-mapped region constructed by the server based on a real-time three-dimensional reconstruction algorithm and the plurality of image frames. For example, each time the unmanned device collects an image frame and transmits it to the server, the server can determine the shooting pose of the unmanned device when capturing the image frame based on the real-time three-dimensional reconstruction algorithm, and construct a three-dimensional map of the corresponding scene in the image frame based on the image frame and the corresponding shooting pose. After constructing the three-dimensional map of the corresponding scene in the last image frame captured by the unmanned device, the three-dimensional maps of the corresponding scenes of the respective image frames are integrated to obtain the overall three-dimensional map.

[0041] Due to the resolution of the image and the number of feature matching, the server can not be able to determine the shooting pose of the image frame involving the occluded scene or the scene with non-ideal texture. If the server cannot determine the shooting pose of a certain image frame, it continues to determine the shooting pose of the next image frame. The image frame for which the shooting pose is not determined cannot construct a three-dimensional map of the corresponding scene, so part of the three-dimensional map corresponding to the region in the overall three-dimensional map is missing, that is, there are holes in the overall three-dimensional map in some regions. When there are more holes in the overall three-dimensional map, the completeness of the overall three-dimensional map is lower, and the overall three-dimensional map cannot completely describe the three-dimensional scene of the to-be-mapped region; when there are fewer holes in the overall three-dimensional map, the completeness of the overall three-dimensional map is higher, and the overall three-dimensional map can completely describe the three-dimensional scene of the to-be-mapped region.

[0042] In this embodiment, the preset completeness condition refers to a condition that the overall three-dimensional map can completely describe the three-dimensional scene of the to-be-mapped region. For example, when the overall three-dimensional map satisfies the preset completeness condition, it is determined that the overall three-dimensional map is successfully constructed, and the overall three-dimensional map is taken as the target three-dimensional map of the to-be-mapped region; when the overall three-dimensional map does not satisfy the preset completeness condition, it is determined that the overall three-dimensional map is not successfully constructed, and a hole filling operation is performed on the hole region in the overall three-dimensional map to fill the holes in the overall three-dimensional map, so as to obtain the target three-dimensional map of the to-be-mapped region.

[0043] In an embodiment, the preset completeness condition is used to represent that the ratio of the number of recovery frames in the plurality of image frames to the number of image frames in the plurality of image frames is greater than or equal to a preset ratio threshold. The recovery frame refers to an image frame for which the shooting pose is determined. The image frames in the plurality of image frames are sorted according to the shooting order. For example, if the unmanned device captures N image frames, the N image frames are determined as {I1, I2,... IN} according to the shooting order of the image frames. The recovery frame refers to an image frame for which the shooting pose is determined.i ,..., I N}, I i is the i-th image frame in the N image frames. In an embodiment, when the server determines the shooting pose of the image frame I i , the image frame I i is marked as a recovery frame P i , to indicate that the i-th image frame in the N image frames is a recovery frame. The number M of recovery frames in the N image frames is counted. M / N is the ratio of the number of recovery frames to the number of image frames. In another embodiment, when the shooting pose of the image frame I i cannot be determined, the image frame I i is marked as a lost frame L i . The number K of lost frames in the N image frames is counted, and the number M of recovery frames in the N image frames is determined based on the number K of lost frames. In this embodiment, whenever an image frame determines the shooting pose, a three-dimensional map of the corresponding scene can be constructed. The more the number of recovery frames in the multiple image frames, the more the scenes for which three-dimensional maps are constructed, and the higher the completeness of the overall three-dimensional map. Therefore, the embodiment sets a preset ratio threshold, which can be the minimum ratio when the overall three-dimensional map can completely describe the three-dimensional scene of the to-be-mapped area. Accordingly, when the ratio is less than the preset ratio threshold, it indicates that the overall three-dimensional map cannot completely describe the three-dimensional scene of the to-be-mapped area, i.e., it can be determined that the overall three-dimensional map does not meet the preset completeness condition. When the ratio is greater than or equal to the preset ratio threshold, it indicates that the overall three-dimensional map can completely describe the three-dimensional scene of the to-be-mapped area, i.e., it can be determined that the overall three-dimensional map meets the preset completeness condition.

[0044] Further, if each time the shooting pose of an image frame is determined, the image frame is marked as a recovery frame, after it is determined that the overall three-dimensional map fails to be constructed, the image frames that are not marked as recovery frames can be marked as lost frames. If each time the shooting pose of an image frame is not determined, the image frame is marked as a lost frame, after it is determined that the overall three-dimensional map fails to be constructed, the image frames that are not marked as lost frames can be marked as recovery frames, to obtain the recovery frames and the lost frames in the multiple image frames.

[0045] In S120, based on the positions of the lost frames and the recovery frames, the lost frames and the recovery frames in the corresponding neighborhoods are added to the to-be-solved group.

[0046] Since the hollow region in the overall three-dimensional map is caused by the fact that the real-time three-dimensional reconstruction algorithm cannot determine the shooting pose of the lost frame, after it is determined that the overall three-dimensional map fails to be constructed, the shooting pose of the lost frame is determined, which can complete the hollow region in the overall three-dimensional map. Figure 2 is a top view of the overall three-dimensional map provided by an embodiment of the present application. As shown in Figure 2 , the overall three-dimensional map is divided into a plurality of regions, and the shooting poses of the image frames in each region are determined.Figure 11 There are multiple hollow regions 13 in the image. The scene within the hollow region 13 does not have a corresponding 3D point cloud, and the overall 3D image is not visible. Figure 11 The area excluding the hollow region 13 contains a 3D point cloud. The hollow region 13 can be considered as a scene unique to the corresponding lost frame, and the surrounding region 14 of the hollow region can be considered as the lost frame and the overall 3D map. Figure 11 In scenarios where frames overlap, there is a correlation between lost frames and certain recovered frames. Recovered frames correlated with lost frames can be used as constraints to gradually recover the shooting pose of the lost frames. Furthermore, since lost frames and their corresponding recovered frames in the neighborhood overlap, there is a high probability that they are correlated. Therefore, in this embodiment, lost frames and their corresponding recovered frames in the neighborhood are added to a group to be solved, in order to quickly search for correlated lost and recovered frames from the group.

[0047] In one embodiment, a circular region is defined as the neighborhood of the lost frame, centered on its location and with a preset distance as its radius. By calculating the distances between the lost frame and each recovered frame, recovered frames with distances less than the preset distance are determined to be located within the neighborhood of the lost frame. These recovered frames, along with the lost frame, are then added to the group to be solved.

[0048] In another embodiment, based on the positions of the lost frame and each recovered frame, the distance between the lost frame and each recovered frame is determined. Based on the distance between the lost frame and each recovered frame, a certain number of recovered frames closest to the lost frame are determined to be located within the neighborhood of the lost frame. These certain number of recovered frames and the lost frame are then added to the group to be solved. For example, Figure 3 This is a flowchart illustrating the process of adding lost frames and their corresponding recovered frames within a neighborhood to a group to be solved, as provided in an embodiment of this application. Figure 3 As shown, the step of adding the lost frame and the recovered frame in the corresponding neighborhood to the group to be solved specifically includes S1201-S1203:

[0049] S1201. Determine the coordinate distance between the lost frame and each recovered frame based on the geographical coordinates of the lost frame and each recovered frame.

[0050] For example, when the unmanned device captures an image frame, its RTK positioning system determines the device's geographic coordinates and uses these coordinates as the geographic coordinates of the currently captured image frame. The unmanned device sends the image frame and its corresponding geographic coordinates to a server so that the server can perform real-time 3D reconstruction based on the image frame and its corresponding geographic coordinates. Assume image frame I... i The geographic coordinates are (x i y i , z i ) T Then the lost frame L can be obtained.i Or restore frame P i The geographic coordinates are (x i y i , z i ) T .

[0051] Correspondingly, lost frame L i and recovery frame P j The coordinate distance s between ij for:

[0052]

[0053] Among them, (x j y j , z j ) T To recover frame P j The geographical coordinates.

[0054] S1202. Sort the recovered frames in ascending order according to the coordinate distance between the lost frame and each recovered frame.

[0055] S1203. Add the first preset number of recovered frames and the corresponding lost frames that are ranked first to the group to be solved.

[0056] For example, for any lost frame, based on the coordinate distance between the lost frame and each recovered frame, the recovered frames are sorted in ascending order according to their corresponding coordinate distances. The earlier a recovered frame appears in the sorted order, the closer it is to the lost frame. The closer a recovered frame is to the lost frame, the higher its overlap with the lost frame. Therefore, the first predetermined number of recovered frames at the top of the sorted order can be considered as recovered frames within the corresponding neighborhood of the lost frame. For example, recovered frames P1, P2, P3, ..., and Pj are related to the lost frame L. i The coordinate distances from smallest to largest are s 2j s 1j s 3j ... and s ij Assuming the first preset quantity is 2, then the recovered frame P1, recovered frame P2, and lost frame L will be recovered. i Add it to the group to be solved.

[0057] It should be noted that, since there may be correlations between lost frames, this embodiment adds all lost frames and the corresponding recovered frames in the neighborhood to a group to be solved, so as to completely search out the correlations between each image frame from the group to be solved.

[0058] S130. Determine the matching results between each image frame in the group to be solved, and classify the two matching image frames into the same connected group according to the matching results.

[0059] For example, feature matching is performed on any two image frames in the group to be solved to determine the number of feature matching pairs between the two image frames. When the number of feature matching pairs exceeds a preset first threshold, the two image frames are determined to be matched; otherwise, they are determined not to be matched. The first threshold is the minimum number of feature matching pairs required for two image frames to match, as set in this embodiment. It can be understood that if the number of feature matching pairs between two image frames exceeds the preset threshold, it indicates that the two image frames highly overlap, and the shooting pose of one image frame significantly constrains the shooting pose of the other image frame. Therefore, it can be determined that there is a correlation between the two matched image frames.

[0060] Since the heading of unmanned equipment is relatively fixed when navigating along a flight path, two image frames with close coordinate distances are more likely to match. Therefore, based on the coordinate distances between image frames, potentially matching image pairs can be filtered out. Then, it is determined whether the number of feature matching pairs between image pairs exceeds a preset threshold, in order to quickly search for matching image pairs in the group to be solved. In this embodiment, Figure 4 This is a flowchart illustrating the process of determining matching image pairs within a group to be solved, as provided in an embodiment of this application. Figure 4 As shown, the step of determining matching image pairs in the group to be solved specifically includes S1301-S1304:

[0061] S1301. Obtain the first image frame from the group to be solved, and take the remaining image frames in the group to be solved, excluding the first image frame, as the second image frame.

[0062] In this embodiment, the first image frame refers to any image frame in the group to be solved, and the second image frame refers to any image frame in the group to be solved other than the first image frame.

[0063] S1302. Based on the distance between the first image frame and each of the second image frames, determine a second preset number of third image frames from each of the second image frames.

[0064] For example, the coordinate distances between the first image frame and each of the second image frames are calculated based on their geographic coordinates. The second image frames are then sorted in ascending order of their coordinate distances to the first image frame, so that the earlier the sorted second image frame is, the closer it is to the first image frame. Assuming the second preset number is 8, the first 8 sorted second image frames are taken as the third image frames corresponding to the first image frame.

[0065] In one embodiment, to find the correlation between lost frames and gradually restore the shooting pose of each lost frame, when constructing image pairs corresponding to lost frames, an equal number of lost frames and recovered frames can be selected. This not only searches for the correlation between recovered frames and lost frames, but also searches for the correlation between lost frames themselves, strengthening the constraint relationship between lost frames and the overall 3D map and improving the success rate of lost frame recovery. For example, Figure 5 This is a flowchart illustrating the acquisition of the third image frame of a lost frame, provided in an embodiment of this application. For example... Figure 5 As shown, the step of obtaining the third image frame of the lost frame specifically includes S13021-13022:

[0066] S13021. In the case that the first image frame is a lost frame, sort each second image frame in ascending order according to the coordinate distance between the first image frame and each second image frame; wherein, the second image frame is a lost frame or a recovered frame in the group to be solved.

[0067] S13022, The third preset number of lost frames and the third preset number of recovered frames that are ranked first are determined as the third image frames.

[0068] For example, suppose the first image frame is a lost frame L. i Then calculate the lost frame L i The coordinate distances between the remaining lost and recovered frames in the unsolved group are sorted in ascending order of coordinate distance. If the second preset number is 8, then the third preset number is 4, meaning that 4 lost frames and 4 recovered frames are selected from the sequence of remaining lost and recovered frames in the unsolved group as the third image frame of lost frame Li. When 4 lost frames are selected, only recovered frames are selected, and so on, until 4 recovered frames are selected; when 4 recovered frames are selected, only lost frames are selected, and so on, until 4 lost frames are selected.

[0069] S1303. Combine each third image frame with its corresponding first image frame to form an image pair.

[0070] Assuming frame L is lost i A certain third image frame is the restored frame P. j Then frame L will be lost. i With recovery frame P j Composition of image pairs (L i P j ).

[0071] S1304. Perform feature matching on each image pair to determine the matching result for each image pair.

[0072] In this embodiment, the matching result refers to whether the image pairs match. For example, Figure 6 This is a flowchart illustrating the process of determining the matching results of image pairs according to embodiments of this application. For example... Figure 6 As shown, the steps for determining the matching result of the image pair specifically include S13041-S13043:

[0073] S13041. Upsample the image pair and extract feature points, match the feature points of the image pair, and determine the number of feature matching pairs for the image pair.

[0074] For example, due to the influence of scene and texture in the lost frames, conventional feature extraction algorithms are unlikely to obtain more effective feature points in the lost frames. Therefore, this embodiment upsamples the two image frames of the image pair before extracting feature points to extract more effective feature points, increase the number of feature matching pairs of the image pair, deeply explore the correlation between the two image frames, and improve the success rate of lost frame recovery.

[0075] S13042. If the number of feature matching pairs is greater than or equal to a preset first quantity threshold, determine that the two image frames of the image pair are matched.

[0076] S13043. If the number of feature matching pairs is less than the first quantity threshold, determine that the two image frames of the image pair do not match.

[0077] For example, suppose frame L is lost. i With recovery frame P j The image pairs (L) i P j The number of feature matching pairs is m, and the preset threshold is T. If m ≥ T, then the lost frame L is obtained. i With recovery frame P j The matching result is {(L i P j If m < T, then the lost frame L is obtained. i With recovery frame P j The matching result is {(L i P j The matching result is 1, where 0 indicates that the corresponding image pair matches, and 0 indicates that the corresponding image pair does not match.

[0078] In another embodiment, in the lost frame L i With recovery frame P j The image pairs (L) i P j The number of feature matching pairs is m, and this image pair is denoted as {(L)}. i P jThe image pair is binarized according to a preset threshold T to obtain the matching result of the image pair. For example, when m≥T, the matching result of the image pair {(L)} is obtained. i P j After binarization of {(L}, the matching result is {(L}). i P j ), 1}; In the case of m < T, for image pair {(L i P j After binarization of {(L}, the matching result is {(L}). i P j ), 0}.

[0079] Furthermore, the matching results of each image pair are iterated, and the two matching image frames are grouped into the same connected group. For example, if frame L is missing... i With recovery frame P j Match, recover frame P j With recovery frame P k Matching, even if frame L is lost i With recovery frame P k Mismatch will also result in the loss of frame L. i , recovery frame P j and recovery frame P k The images are stored in the same connected group. In this case, 3D reconstruction based on the image frames in the connected group can construct a local 3D map that is associated with the overall 3D map and covers the hole areas. This allows the local 3D map to be transformed into the overall 3D map based on the relationship between the local and overall 3D maps, thereby filling in the hole areas in the overall 3D map.

[0080] In one embodiment, to improve the efficiency of connected group partitioning, the unsolved group can be converted into a tree structure based on the matching results of each image pair in the unsolved group. A disjoint-set data structure is then used on the tree structure to quickly partition the unsolved group into at least one connected group. For example, Figure 7 This is a flowchart illustrating the process of dividing connected groups based on matching results, provided in an embodiment of this application. For example... Figure 7 As shown, the step of dividing the connected groups based on the matching results specifically includes S1305-S1306:

[0081] S1305. Take the image frames in the group to be solved as nodes of the tree structure, connect the nodes corresponding to two matching image frames, and construct the corresponding tree structure.

[0082] For example, one image frame from the group to be solved is filled into a node of the tree structure, and the matching result of each pair of images in the group to be solved is filled into the connectivity relationship of the corresponding node in the tree structure, thus constructing the tree structure corresponding to the group to be solved. For example, the lost frame L...i and recovery frame P j The missing frame L is represented by nodes i and j in the tree structure, respectively. i and recovery frame P j Lost frame L i and recovery frame P j The matching result {(L i P j ), 1} are filled into the connectivity relationships of the corresponding nodes in the tree structure so that node i and node j are connected.

[0083] S1306. Query the connected nodes in the tree structure and combine the image frames corresponding to each connected node into a connected group.

[0084] Figure 8 This is a schematic diagram of the tree structure provided in an embodiment of this application. For example... Figure 8 As shown, the tree structure 20 has several nodes 21. Each node 21 corresponds to an image frame in the group to be solved, and the connection 22 between two nodes 21 corresponds to the matching result between the two image frames. By querying the connected nodes 21 in the tree structure 20 using the disjoint-set data structure technique, it is found that any node 22 within the first dashed box 23 is connected to at least one other node 22 within the first dashed box, but not to any node 25 within the second dashed box 23 or the third dashed box 25. Therefore, the image frames corresponding to the nodes 22 within the first dashed box 23 are grouped into a first connected group. Similarly, the image frames corresponding to the nodes within the second dashed box 24 are grouped into a second connected group, and the image frames corresponding to the nodes within the third dashed box 25 are grouped into a third connected group.

[0085] S140. Perform 3D reconstruction based on the image frames in the connected group to obtain a local 3D map. Align the local 3D map with the global 3D map to obtain the target 3D map.

[0086] For example, the connected group contains lost frames and recovered frames. The shooting pose of the lost frames can be recovered based on the shooting pose of the recovered frames and the feature matching pairs between the lost and recovered frames. Based on each image frame in the connected group and its corresponding shooting pose, a local 3D map corresponding to each image frame in the connected group is constructed. By using the recovered frames that are identical in the local 3D map and the overall 3D map, the local 3D map can be aligned to the overall 3D map, completing the hole-filling operation.

[0087] Since not all connected groups can construct local 3D maps that can be aligned with the overall 3D map—for example, connected groups consisting entirely of lost frames lack recovered frames identical to the overall 3D map, making it difficult to align their constructed local 3D maps with the overall 3D map—it is advisable to first select valid connected groups before constructing the local 3D map to create local 3D maps that can be aligned with the overall 3D map. In one embodiment, Figure 9 This is a flowchart illustrating the construction of a local 3D map using an effective connected group, as provided in an embodiment of this application. Figure 9 As shown, the steps for constructing a local 3D map through an effective connected group specifically include S1401-S1403:

[0088] S1401. If the number of recovered frames in the connected group is greater than or equal to a preset second quantity threshold, perform three-dimensional reconstruction based on the image frames in the connected group.

[0089] In this embodiment, the more recovered frames in the connected group, the more overlapping scenes there are between the local 3D map constructed by the connected group and the overall 3D map, and the more accurately the local 3D map can be aligned to the overall 3D map. Therefore, the second quantity threshold can be regarded as the minimum number of recovered frames required for the local 3D map to be aligned to the overall 3D map. Generally, the second quantity threshold is set to 3. When the connected group contains at least 3 recovered frames, aligning the 3D scene constructed by the recovered frames in the connected group with the 3D scene constructed by the corresponding recovered frames in the overall 3D map can accurately obtain the relative transformation parameters between the local 3D map and the overall 3D map. These relative transformation parameters can then be used to transform the local 3D map into the overall 3D map, completing the hole-filling operation of the overall 3D map.

[0090] S1402. If the number of recovered frames in the connected group is less than the second quantity threshold, determine the number of shooting routes corresponding to the connected group based on the shooting route of each image frame in the connected group.

[0091] For example, when the connected group contains fewer than 3 recovery frames, the error between the 3D scene constructed by the recovery frames in the connected group and the 3D scene constructed by the corresponding recovery frames in the overall 3D map is large, so the two 3D scenes cannot be aligned.

[0092] Furthermore, since the overall 3D map is aligned to an absolute geographic coordinate system, if image frames in a connected group are captured along two or more flight paths, the image frames along the two flight paths form a plane. The 3D scene constructed based on these two flight paths can be well aligned to the overall 3D map. Therefore, when the number of recovered frames in a connected group is insufficient, the capture flight paths of each image frame in the connected group can be used to determine whether the local 3D map constructed from the connected group can be aligned to the overall 3D map.

[0093] In this embodiment, the straight-line flight path segment where the unmanned equipment was located when each image frame in the connected group was captured can be determined based on the geographical coordinates of each straight-line flight path segment in the flight path, and this straight-line flight path segment can be used as the shooting path for the corresponding image frame. The number of different shooting paths corresponding to each image frame in the connected group is counted to obtain the number of shooting paths.

[0094] S1403. When the number of shooting routes is greater than one, perform 3D reconstruction based on the image frames in the connected group.

[0095] For example, when there are two or more shooting routes in a connected group, a local 3D map can be constructed using each image frame in the connected group and its corresponding shooting pose, and then aligned with the overall 3D map. When there is only one shooting route in a connected group, since the rotation axis of the rotating route is not observable, the local 3D map constructed from this connected group cannot be aligned with the overall 3D map, and therefore the connected group is discarded.

[0096] In this embodiment, when the number of shooting routes in a connected group is greater than one, the local 3D map of the connected group is combined with the overall 3D map to obtain the target 3D map. It can be understood that for connected groups with more than one shooting route, when performing 3D reconstruction using geographic coordinates acquired by RTK, the coordinate system of the local 3D map has already been transformed to the absolute geographic coordinate system. Therefore, the local 3D map can be directly combined with the overall 3D map to achieve seamless integration with the overall 3D map.

[0097] In another embodiment, the relative transformation parameters of the local and global 3D maps can be determined based on the shooting poses of each recovered frame in the connected group within the local and global 3D maps. For example, Figure 10 This is a flowchart illustrating the determination of relative transformation parameters provided in an embodiment of this application. For example... Figure 10 As shown, the steps for determining the relative transformation parameters specifically include S1404-S1406:

[0098] S1404. Obtain the first shooting pose of the recovered frame in the local 3D map and the second shooting pose in the global 3D map.

[0099] For example, for a recovered frame in a connected group, the second shooting pose T of the recovered frame in the coordinate system of the overall 3D map is determined when constructing the overall 3D map. 2 When constructing a local 3D map, the recovery frame P will be determined. j The first shooting pose T in the coordinate system of the local 3D map 1 .

[0100] It is understandable that, for the recovered frame, the first shooting pose T of the recovered frame... 1 With the second shooting pose T 2 Relative transformation parameters between This refers to the relative transformation parameters between the coordinate system of the overall 3D map and the coordinate system of the local 3D map. Therefore, the relative transformation parameters between the local 3D map and the overall 3D map can be determined by the first and second shooting poses of each recovered frame.

[0101] S1405. Determine the relative transformation parameters between the first shooting pose and the corresponding second shooting pose, and transform the local 3D map using the relative transformation parameters.

[0102] For example, assuming the connected group has u recovered frames, construct the objective function F based on the first and second shooting poses of the u recovered frames:

[0103]

[0104] in, Let q be the second shooting pose of the q-th recovered frame out of u recovered frames. Let be the first shooting pose of the q-th recovered frame out of u recovered frames. By optimizing the objective function F, the relative transformation parameters are obtained.

[0105] Furthermore, by using relative transformation parameters, the local 3D map is transformed into the coordinate system of the global 3D map, and the lost frames in the connected group are transformed into the coordinate system of the global 3D map, so as to recover the shooting pose of the lost frames and obtain a local 3D map in the absolute geographic coordinate system.

[0106] S1406. Combine the converted local 3D map and the overall 3D map to obtain the target 3D map.

[0107] For example, by combining a local 3D map located in an absolute geographic coordinate system with the overall 3D map, a complete target 3D map of the area to be surveyed is obtained, thus completing the hole-filling operation of the overall 3D map.

[0108] In summary, the 3D scene hole-filling method provided in this application obtains lost frames that failed to reconstruct the scene and recovered frames that successfully reconstructed the scene. Based on the positions of the lost and recovered frames, the lost frames and their corresponding neighboring recovered frames are added to a group to be solved, so as to quickly search for related lost and recovered frames from the group to be solved. According to the matching results between each image frame in the group to be solved, it is determined that there is a relationship between two matching image frames, and the image frames with the relationship are added to the same connected group, so as to construct a local 3D map associated with the overall 3D map through the image frames in the connected group. After aligning the local 3D map with the overall 3D map, the local 3D map can be used to fill the holes in the overall 3D map, resulting in a highly complete target 3D map, realizing the hole-filling operation of the overall 3D map. Through the above technical means, lost and recovered frames with relationships are searched from image frames, and a local 3D map associated with the overall 3D map is constructed based on the lost and recovered frames with relationships, so that the local 3D map can be aligned with the overall 3D map, filling the holes in the overall 3D map and improving the integrity of the 3D scene. When filling holes in a scene, a local 3D map can be built with only a small amount of computing resources and waiting time, without having to rebuild the complete 3D map. This also avoids the situation where rebuilding the map fails, effectively improving the reconstruction efficiency and success rate of 3D scenes and solving the problem of low reconstruction efficiency of 3D scenes in existing technologies.

[0109] Based on the above embodiments, Figure 11 This is a schematic diagram of a three-dimensional scene hole-filling device provided in an embodiment of this application. (Reference) Figure 11 The three-dimensional scene hole-filling device provided in this embodiment specifically includes: a scene reconstruction module 31, a group to be solved generation module 32, a connected group generation module 33, and a scene hole-filling module 34.

[0110] The scene reconstruction module is configured to obtain recovered frames and lost frames from multiple image frames used to reconstruct the overall 3D map when the overall 3D map obtained by 3D reconstruction does not meet the preset completeness conditions.

[0111] The module for generating the group to be solved is configured to add the lost frame and the corresponding neighboring recovered frames into the group to be solved based on the positions of the lost frame and each recovered frame.

[0112] The connected group generation module is configured to determine the matching results between each image frame in the group to be solved, and to assign two matching image frames to the same connected group based on the matching results.

[0113] The scene hole-filling module is configured to perform 3D reconstruction based on image frames in a connected group to obtain a local 3D map, and then align the local 3D map with the overall 3D map to obtain the target 3D map.

[0114] Based on the above embodiments, the preset integrity condition is used to characterize that the ratio of the number of recovered frames in multiple image frames to the number of image frames in multiple image frames is greater than or equal to a preset ratio threshold.

[0115] Based on the above embodiments, the unsolved group generation module includes: a coordinate distance determination unit, configured to determine the coordinate distance between the lost frame and each recovered frame based on the geographical coordinates of the lost frame and the geographical coordinates of each recovered frame; a recovered frame sorting unit, configured to sort each recovered frame in ascending order based on the coordinate distance between the lost frame and each recovered frame; and an unsolved group generation unit, configured to add a first preset number of recovered frames and corresponding lost frames that are sorted first to the unsolved group.

[0116] Based on the above embodiments, the connected group generation module includes: a first image frame acquisition unit, configured to acquire a first image frame from the group to be solved, and use the remaining image frames in the group to be solved, excluding the first image frame, as second image frames; a third image frame determination unit, configured to determine a second preset number of third image frames from each of the second image frames based on the distance between the first image frame and each of the second image frames; wherein the first image frame and the second image frame are image frames in the group to be solved; an image pair composition unit, configured to compose each third image frame and the corresponding first image frame into an image pair; and a feature matching unit, configured to perform feature matching on each image pair and determine the matching result of each image pair.

[0117] Based on the above embodiments, the third image frame determination unit includes: a second image frame sorting subunit, configured to sort each second image frame in ascending order according to the coordinate distance between the first image frame and each second image frame when the first image frame is a lost frame; wherein the second image frame is a lost frame or a recovered frame in the group to be solved; and a third image frame determination subunit, configured to determine the third image frame as the third preset number of lost frames and the third preset number of recovered frames that are ranked first.

[0118] Based on the above embodiments, the feature matching unit includes: an upsampling matching subunit, configured to upsample the image pair and extract feature points, match the feature points of the image pair, and determine the number of feature matching pairs of the image pair; a first result determination subunit, configured to determine that the two image frames of the image pair match when the number of feature matching pairs is greater than or equal to a preset first quantity threshold; and a second result determination subunit, configured to determine that the two image frames of the image pair do not match when the number of feature matching pairs is less than the first quantity threshold.

[0119] Based on the above embodiments, the connected group generation module includes: a tree structure construction unit, configured to use image frames in the group to be solved as nodes of the tree structure, connect the nodes corresponding to two matching image frames, and construct the corresponding tree structure; and a connected group generation unit, configured to query the connected nodes in the tree structure and combine the image frames corresponding to each connected node into a connected group.

[0120] Based on the above embodiments, the scene hole-filling module includes: a first local reconstruction unit, configured to perform three-dimensional reconstruction based on image frames in the connected group when the number of recovered frames in the connected group is greater than or equal to a preset second quantity threshold.

[0121] Based on the above embodiments, the scene hole-filling module includes: a flight path number determination unit, configured to determine the number of shooting flight paths corresponding to the connected group based on the shooting flight path of each image frame in the connected group when the number of recovered frames in the connected group is less than a second number threshold; and a second local reconstruction unit, configured to perform three-dimensional reconstruction based on the image frames in the connected group when the number of shooting flight paths is greater than one.

[0122] Based on the above embodiments, the scene hole filling module includes: a first combination unit, configured to combine the local 3D map of the connected group with the overall 3D map when the number of shooting routes in the connected group is greater than one, to obtain the target 3D map.

[0123] Based on the above embodiments, the scene hole-filling module includes: a shooting pose acquisition unit, configured to acquire the first shooting pose of the recovered frame in the local 3D map and the second shooting pose in the overall 3D map; a map conversion unit, configured to determine the relative transformation parameters between the first shooting pose and the corresponding second shooting pose, and convert the local 3D map through the relative transformation parameters; and a second combination unit, configured to combine the converted local 3D map and the overall 3D map to obtain the target 3D map.

[0124] The 3D scene hole-filling device provided in this application obtains lost frames that failed to reconstruct the scene and recovered frames that successfully reconstructed the scene. Based on the positions of the lost and recovered frames, the lost frames and their corresponding neighboring recovered frames are added to a group to be solved, so as to quickly search for related lost and recovered frames from the group to be solved. According to the matching results between each image frame in the group to be solved, it is determined that there is a relationship between two matching image frames, and the image frames with the relationship are added to the same connected group, so as to construct a local 3D map associated with the overall 3D map through the image frames in the connected group. After aligning the local 3D map with the overall 3D map, the local 3D map can be used to fill the holes in the overall 3D map, resulting in a highly complete target 3D map, realizing the hole-filling operation of the overall 3D map. Through the above technical means, lost and recovered frames with relationships are searched from image frames, and a local 3D map associated with the overall 3D map is constructed based on the lost and recovered frames with relationships, so that the local 3D map can be aligned with the overall 3D map, filling the holes in the overall 3D map and improving the completeness of the 3D scene. When filling holes in a scene, a local 3D map can be built with only a small amount of computing resources and waiting time, without having to rebuild the complete 3D map. This also avoids the situation where rebuilding the map fails, effectively improving the reconstruction efficiency and success rate of 3D scenes and solving the problem of low reconstruction efficiency of 3D scenes in existing technologies.

[0125] The three-dimensional scene hole-filling device provided in this application embodiment can be used to execute the three-dimensional scene hole-filling method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0126] Figure 12 This is a structural schematic diagram of a three-dimensional scene hole-filling device provided in an embodiment of this application, with reference to... Figure 12 The 3D scene patching device includes a processor 41, a memory 42, a communication device 43, an input device 44, and an output device 45. The number of processors 41 and the number of memories 42 in the 3D scene patching device can be one or more. The processor 41, memory 42, communication device 43, input device 44, and output device 45 of the 3D scene patching device can be connected via a bus or other means.

[0127] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the three-dimensional scene hole-filling method in any embodiment of this application (e.g., scene reconstruction module 31, unsolved group generation module 32, connected group generation module 33, and scene hole-filling module 34 in the three-dimensional scene hole-filling device). The memory 42 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The communication device 43 is used for data transmission.

[0129] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned three-dimensional scene hole filling method.

[0130] Input device 44 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 45 may include display devices such as a display screen.

[0131] The three-dimensional scene hole-filling device provided above can be used to perform the three-dimensional scene hole-filling method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0132] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a three-dimensional scene hole-filling method. The three-dimensional scene hole-filling method includes: when the overall three-dimensional map obtained by three-dimensional reconstruction does not meet a preset integrity condition, obtaining restored frames and lost frames from multiple image frames used to reconstruct the overall three-dimensional map; adding the lost frames and the restored frames in their corresponding neighborhoods to a group to be solved based on the positions of the lost frames and each restored frame; determining the matching results between each image frame in the group to be solved, and dividing two matching image frames into the same connected group according to the matching results; performing three-dimensional reconstruction based on the image frames in the connected group to obtain a local three-dimensional map, and aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map.

[0133] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0134] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the above-mentioned three-dimensional scene hole-filling method, but can also execute related operations in the three-dimensional scene hole-filling method provided in any embodiment of this application.

[0135] The three-dimensional scene hole-filling device, storage medium, and three-dimensional scene hole-filling equipment provided in the above embodiments can execute the three-dimensional scene hole-filling method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the three-dimensional scene hole-filling method provided in any embodiment of this application.

[0136] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A method of inpainting a three-dimensional scene, the method comprising: The method comprises the steps of: In the case that the overall three-dimensional map obtained by three-dimensional reconstruction does not meet a preset completeness condition, obtaining recovery frames and missing frames from a plurality of image frames used for reconstructing the overall three-dimensional map; Based on the positions of the missing frames and each of the recovery frames, adding each of the missing frames and the recovery frames in a corresponding neighborhood into a to-be-solved group; Determining a matching result between each of the image frames in the to-be-solved group, and dividing two image frames matched according to the matching result into the same connected group; wherein the determination of the matching result between each of the image frames in the to-be-solved group specifically comprises: obtaining any one of the image frames in the to-be-solved group as a first image frame, and obtaining the remaining image frames in the to-be-solved group except the first image frame as second image frames; determining a second preset number of third image frames from each of the second image frames according to the distances between the first image frame and each of the second image frames, and if the first image frame is a missing frame, obtaining the same number of missing frames and recovery frames as corresponding third image frames; forming an image pair by each of the third image frames and the corresponding first image frame; performing feature matching on each of the image pairs to determine a matching result of each of the image pairs; Performing three-dimensional reconstruction according to the image frames in the connected group to obtain a local three-dimensional map, and aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map.

2. The 3-D scene hole filling method of claim 1, wherein, The preset completeness condition is used to represent that a ratio of the number of the recovery frames in the plurality of image frames to the number of the image frames in the plurality of image frames is greater than or equal to a preset ratio threshold.

3. The 3-D scene hole filling method of claim 1, wherein, The adding of the missing frames and the recovery frames in a corresponding neighborhood into a to-be-solved group based on the positions of the missing frames and each of the recovery frames comprises: Determining a coordinate distance between the missing frames and each of the recovery frames according to geographical coordinates of the missing frames and geographical coordinates of each of the recovery frames; According to the coordinate distance between the missing frames and each of the recovery frames, sorting each of the recovery frames in ascending order; Adding a first preset number of recovery frames and corresponding missing frames sorted in the front into the to-be-solved group.

4. The 3-D scene hole filling method of claim 1, wherein, The determination of a second preset number of third image frames from each of the second image frames according to the distances between the first image frame and each of the second image frames comprises: In the case that the first image frame is a missing frame, sorting each of the second image frames in ascending order according to the coordinate distance between the first image frame and each of the second image frames; wherein the second image frame is a missing frame or a recovery frame in the to-be-solved group; Determining a third preset number of missing frames and a third preset number of recovery frames sorted in the front as the third image frames.

5. The method of claim 1, wherein, The feature matching on each of the image pairs to determine a matching result of each of the image pairs comprises: Performing up-sampling on the image pair and extracting feature points, matching the feature points of the image pair, and determining a number of feature matching pairs of the image pair; In the case that the number of feature matching pairs is greater than or equal to a preset first number threshold, determining that the two image frames of the image pair are matched. In a case where the number of the feature matching pairs is less than the first number threshold, it is determined that the two image frames of the image pair are not matched.

6. The 3-D scene hole filling method of claim 1, wherein, The dividing the two image frames matched according to the matching result into the same connected group comprises: The image frames in the group to be solved are taken as nodes of a tree structure, and the nodes corresponding to the two image frames matched are connected, so as to construct a corresponding tree structure; The connected nodes in the tree structure are queried, and the image frames corresponding to each of the connected nodes are combined into one connected group.

7. The 3-D scene hole filling method of claim 1, wherein, The three-dimensional reconstruction according to the image frames in the connected group comprises: In a case where the number of the recovered frames in the connected group is greater than or equal to a preset second number threshold, the three-dimensional reconstruction is performed according to the image frames in the connected group.

8. The 3-D scene hole filling method of claim 7, wherein, The three-dimensional reconstruction according to the image frames in the connected group comprises: In a case where the number of the recovered frames in the connected group is less than the second number threshold, the number of the photographing routes corresponding to the connected group is determined according to the photographing routes of each of the image frames in the connected group; In a case where the number of the photographing routes is greater than one, the three-dimensional reconstruction is performed according to the image frames in the connected group.

9. The 3-D scene hole filling method of claim 8, wherein, The aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map comprises: In a case where the number of the photographing routes of the connected group is greater than one, the local three-dimensional map of the connected group is combined with the overall three-dimensional map to obtain a target three-dimensional map.

10. The 3-D scene hole filling method of claim 1, wherein, The aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map comprises: The first photographing pose of the recovered frame in the connected group in the local three-dimensional map and the second photographing pose of the recovered frame in the overall three-dimensional map are acquired; The relative transformation parameter between the first photographing pose and the corresponding second photographing pose is determined, and the local three-dimensional map is converted through the relative transformation parameter; The converted local three-dimensional map and the overall three-dimensional map are combined to obtain the target three-dimensional map.

11. A three-dimensional scene inpainting apparatus, characterized by: Comprise: The scene reconstruction module is configured to, in a case where the overall three-dimensional map obtained through the three-dimensional reconstruction does not satisfy a preset completeness condition, acquire a recovered frame and a missing frame from a plurality of image frames used to reconstruct the overall three-dimensional map; The group to be solved generation module is configured to, based on the position of the missing frame and each of the recovered frames, add the missing frame and the recovered frames in a corresponding neighborhood into a group to be solved. The three-dimensional reconstruction according to the image frames in the connected group comprises: In a case where the number of the recovered frames in the connected group is greater than or equal to a preset second number threshold, the three-dimensional reconstruction is performed according to the image frames in the connected group. The three-dimensional reconstruction according to the image frames in the connected group comprises: In a case where the number of the recovered frames in the connected group is less than the second number threshold, the number of the photographing routes corresponding to the connected group is determined according to the photographing routes of each of the image frames in the connected group; In a case where the number of the photographing routes is greater than one, the three-dimensional reconstruction is performed according to the image frames in the connected group. The aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map comprises: In a case where the number of the photographing routes of the connected group is greater than one, the local three-dimensional map of the connected group is combined with the overall three-dimensional map to obtain a target three-dimensional map. The aligning the local three-dimensional map with the overall three-dimensional map to obtain a target three-dimensional map comprises: The first photographing pose of the recovered frame in the connected group in the local three-dimensional map and the second photographing pose of the recovered frame in the overall three-dimensional map are acquired; The relative transformation parameter between the first photographing pose and the corresponding second photographing pose is determined, and the local three-dimensional map is converted through the relative transformation parameter; The converted local three-dimensional map and the overall three-dimensional map are combined to obtain the target three-dimensional map. Comprise: The scene reconstruction module is configured to, in a case where the overall three-dimensional map obtained through the three-dimensional reconstruction does not satisfy a preset completeness condition, acquire a recovered frame and a missing frame from a plurality of image frames used to reconstruct the overall three-dimensional map; The group to be solved generation module is configured to, based on the position of the missing frame and each of the recovered frames, add the missing frame and the recovered frames in a corresponding neighborhood into a group to be solved. The connected group generation module is configured to determine matching results between each image frame in the to-be-solved group, and divide two matched image frames into the same connected group according to the matching results; wherein the connected group generation module is specifically configured to obtain any one image frame in the to-be-solved group as a first image frame, and obtain the remaining image frames in the to-be-solved group except the first image frame as second image frames; determine a second preset number of third image frames from each second image frame according to the distance between the first image frame and each second image frame, and if the first image frame is a lost frame, obtain the same number of lost frames and recovered frames to determine corresponding third image frames; group each third image frame and the corresponding first image frame into an image pair; and perform feature matching on each image pair to determine the matching result of each image pair. The scene hole filling module is configured to perform three-dimensional reconstruction according to the image frames in the connected group, obtain a local three-dimensional map, align the local three-dimensional map with the overall three-dimensional map, and obtain a target three-dimensional map.

12. A three-dimensional scene hole filling device, comprising: The method comprises: one or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional scene hole filling method according to any one of claims 1-10.

13. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to perform the three-dimensional scene hole filling method according to any one of claims 1-10.

Images