Three-dimensional reconstruction data generation method, medium and system

Abstract

The invention discloses a three-dimensional reconstruction data generation method, medium and system, and the method comprises the steps: obtaining a panoramic video, and carrying out the frame-by-frame extraction of the panoramic video, so as to obtain a corresponding first image sequence; extracting local feature points corresponding to each video frame in the first image sequence, and calculating feature point matching pairs between adjacent images according to the local feature points; filtering the first image sequence according to the feature point matching to obtain a second image sequence; and performing loopback marking on the video frames in the second image sequence to obtain three-dimensional reconstruction data. The three-dimensional reconstruction efficiency can be effectively improved; meanwhile, the success rate of three-dimensional reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction data generation method, medium and system. Background Art

[0002] Three-dimensional reconstruction is the process of processing input image or video data and reconstructing the structure in three-dimensional space based on the processed data.

[0003] In the related art, when generating three-dimensional reconstruction data, video frames in the input data are often directly extracted at fixed intervals to perform three-dimensional reconstruction based on the extracted video frames; the input data generated in this way contains a large number of redundant video frames, which reduces the efficiency of three-dimensional reconstruction; and the success rate of three-dimensional reconstruction is low. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one purpose of the present invention is to propose a method for generating three-dimensional reconstruction data, which can effectively improve the efficiency of three-dimensional reconstruction and at the same time improve the success rate of three-dimensional reconstruction.

[0005] In a first aspect, an embodiment of the present invention proposes a method for generating three-dimensional reconstruction data, comprising the following steps: acquiring a panoramic video, and extracting the panoramic video frame by frame to obtain a corresponding first image sequence; extracting local feature points corresponding to each video frame in the first image sequence, and calculating feature point matching pairs between adjacent images based on the local feature points; filtering the first image sequence based on the feature point matching to obtain a second image sequence; and loop-marking the video frames in the second image sequence to obtain three-dimensional reconstruction data.

[0006] According to the method for generating 3D reconstruction data of an embodiment of the present invention, first, a panoramic video is obtained, and the panoramic video is extracted frame by frame to obtain a corresponding first image sequence; then, local feature points corresponding to each video frame in the first image sequence are extracted, and feature point matching pairs between adjacent images are calculated based on the local feature points; then, the first image sequence is filtered based on the feature point matching to obtain a second image sequence; then, the video frames in the second image sequence are loop-marked to obtain 3D reconstruction data. This effectively improves the efficiency of 3D reconstruction and at the same time, improves the success rate of 3D reconstruction.

[0007] In some embodiments, redundant image filtering is performed on the first image sequence according to the following formula: ; in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a first preset threshold; If the filtering condition is met, then remove the frame.

[0008] In some embodiments, occlusion or motion blur image filtering is performed on the first image sequence according to the following formula: ; in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a second preset threshold; If the filtering condition is met, then remove the Frame and Video frames between frames.

[0009] In some embodiments, the second image sequence includes ordinary video frames and reference images. After filtering the first image sequence according to the feature point matching to obtain the second image sequence, it also includes: calculating the rotation matrix and translation vector of all adjacent images in the second image sequence; for any ordinary video frame, calculating the rotation matrix of the ordinary video frame relative to the reference image according to the rotation matrix of the adjacent image; converting the rotation matrix of the ordinary video frame relative to the reference image into Euler angles, and using the Euler angles as the yaw angle corresponding to the ordinary video frame; rotating the ordinary video frame according to the yaw angle to align the camera perspective of the second image sequence.

[0010] In some embodiments, loop marking is performed on video frames in the second image sequence, including: establishing a landmark point relationship graph based on the feature point matching relationship between the video frames in the second image sequence, so as to characterize the common view relationship of the video frames in the second image sequence through the landmark point relationship graph; clustering is performed based on the landmark point relationship graph to obtain multiple image categories; for each video frame in the second image sequence, calculating the distance between the current video frame and each of the image categories, and taking the image category with the smallest distance as the target image category; selecting corresponding video frames from the target image category to perform feature point matching with the current video frame, so as to determine the loop frame corresponding to the current video frame according to the matching results; and establishing an adjacent relationship between the current video frame and the loop frame.

[0011] In some embodiments, a landmark point relationship graph is established based on the feature point matching relationship between video frames in the second image sequence, including: for an image pair formed by any two video frames in the second image sequence, initializing landmark points based on the feature point pairs of the image pairs; for a feature point in any one video frame, determining whether the feature point matches feature points in multiple images; if so, merging the feature points that match the feature point into the same landmark point; and constructing a graph structure in which nodes represent video frames or landmark points, thereby representing the connection relationship between video frames and landmark points.

[0012] In some embodiments, clustering is performed based on the landmark point relationship graph to obtain multiple image categories, including: for any video frame in the second image sequence, calculating the global features corresponding to the current video frame; obtaining similar video frames corresponding to the current video frame in the second image sequence according to the global features; and clustering is performed based on the landmark point relationship graph and the similar video frames to obtain multiple image categories.

[0013] In some embodiments, the method further comprises: S201, setting the video frames in the second image sequence to an initial state; S202, traversing the second image sequence to determine whether there is a video frame in the second image sequence that is in an initial state; if yes, executing step S203; if no, executing step S211; S203, taking the video frame with the largest landmark length among the video frames in the initial state as the current video frame; S204, acquiring an adjacent image corresponding to the current video frame according to the landmark point relationship graph; S205, determining whether the number of adjacent images is 1; if yes, executing step S206; if no, executing step S207; S206, modifying the state of the current video frame to a reserved state, and returning to step S202; S207, pre-deleting the current video frame, updating the landmark point relationship graph according to the pre-deletion result, and calculating the average landmark point length of the updated landmark point relationship graph; S208, determining whether the average landmark length is greater than or equal to a preset length threshold; if yes, executing step S209; if no, executing step S210; S209, deleting the current video frame and returning to step S202; S210, changing the state of the current video frame to a reserved state, and returning to step S202; S211, taking the video frame in the reserved state as a key frame.

[0014] In a second aspect, an embodiment of the present invention proposes a computer-readable storage medium on which a three-dimensional reconstruction data generation program is stored. When the three-dimensional reconstruction data generation program is executed by a processor, the three-dimensional reconstruction data generation method as described above is implemented.

[0015] In a third aspect, an embodiment of the present invention proposes a three-dimensional reconstruction data generation system, comprising: an acquisition module, wherein the acquisition module is used to acquire a panoramic video and extract the panoramic video frame by frame to obtain a corresponding first image sequence; an extraction module, wherein the extraction module is used to extract local feature points corresponding to each video frame in the first image sequence, and calculate feature point matching pairs between adjacent images based on the local feature points; a filtering module, wherein the filtering module is used to filter the first image sequence based on the feature point matching to obtain a second image sequence; and a marking module, wherein the marking module is used to loop-mark the video frames in the second image sequence to obtain three-dimensional reconstruction data.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flow chart of a method for generating three-dimensional reconstruction data according to an embodiment of the present invention; Figure 2 is a schematic diagram of camera perspective alignment effect according to an embodiment of the present invention; Figure 3 is a schematic diagram of a key frame extraction process according to an embodiment of the present invention; Figure 4 4 is a block diagram of a system for generating three-dimensional reconstruction data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0019] The following describes a method for generating three-dimensional reconstruction data according to an embodiment of the present invention with reference to the accompanying drawings.

[0020] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for generating three-dimensional reconstruction data according to an embodiment of the present invention. Figure 1 As shown, the 3D reconstruction data generation method includes the following steps: S101, obtaining a panoramic video, and extracting the panoramic video frame by frame to obtain a corresponding first image sequence.

[0021] As an example, the acquired panoramic video is extracted frame by frame to obtain the corresponding first image sequence ,in, Indicates video frames, Indicates the number of video frames in the panoramic video. By extracting the panoramic video frame by frame, a first image sequence is obtained and named in sequence, such as Frame, and The next frame of the frame is Frames; that is, the naming of the images indicates the temporal relationship between video frames.

[0022] S102, extracting local feature points corresponding to each video frame in the first image sequence, and calculating feature point matching pairs between adjacent images based on the local feature points.

[0023] As an example, first, the local feature points corresponding to each video frame can be extracted by SIFT, SuperPoint and other methods; then, the feature point matching pairs between adjacent images are the same feature points in adjacent images; the feature point matching pairs between adjacent images can be achieved by mutual nearest neighbor search; for example, by superglue and other methods. To express the Frame and The number of matching pairs of feature points between frames.

[0024] S103: Filter the first image sequence according to feature point matching to obtain a second image sequence.

[0025] In some embodiments, redundant image filtering is performed on the first image sequence according to the following formula: ; in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a first preset threshold; If the filtering condition is met, then remove the frame.

[0026] As an example, first, set the redundancy judgment condition as: ; Preferably, the determination threshold (i.e., the first preset threshold) is set to 5%. If the redundant determination condition is met, it means that the first Frame and The number of feature matching points in the frames is similar; The feature points extracted from the frame can be used Frame is used to replace, therefore, the determination of Frame redundancy, for the For the case where the redundancy judgment condition is met, the first frame: ; For the case that does not meet the redundancy judgment conditions, keep the frame: .

[0027] In some embodiments, occlusion or motion blur image filtering is performed on the first image sequence according to the following formula: ; in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a second preset threshold; If the filtering condition is met, then remove the Frame and Video frames between frames.

[0028] As an example, first, the judgment condition of occlusion or motion blur image is set as: ; Preferably, the threshold corresponding to the judgment condition (i.e., the second preset threshold) is set to 90%. , , if there is a frame gap satisfy: ,express Much smaller than , meet the judgment conditions of occlusion or motion blurred image, filter, filter out the current frame arrive The video frames between if ,express and Similar, do not meet the filtering conditions, and are retained. Specifically, assuming ,and ; … ; ; It can be seen that the video frame , … The image is filtered if the judgment conditions of occlusion or motion blur are met.

[0029] In some embodiments, the second image sequence includes ordinary video frames and reference images. After filtering the first image sequence according to feature point matching to obtain the second image sequence, it also includes: calculating the rotation matrix and translation vector of all adjacent images in the second image sequence; for any ordinary video frame, calculating the rotation matrix of the ordinary video frame relative to the reference image according to the rotation matrix of the adjacent image; converting the rotation matrix of the ordinary video frame relative to the reference image into Euler angles, and using the Euler angles as the yaw angles corresponding to the ordinary video frame; rotating the ordinary video frame according to the yaw angle to align the camera perspective of the second image sequence.

[0030] That is to say, the camera perspectives of the video frames in the second image sequence are aligned, which can facilitate the retrieval and matching of subsequent images.

[0031] As an example, first, the rotation matrix and translation vector between all adjacent images in the second image sequence are calculated; it should be noted that the rotation matrix and translation vector can be calculated from the feature point matching pair by epipolar geometry, and the translation vector and rotation matrix are used to restore the rotation matrix and translation vector of the adjacent image; then, record for Relative to The rotation matrix represents the relative relationship between the orientations of the two video frames in three-dimensional space. for Relative to The translation vector represents the relative position relationship between the two video frames in the three-dimensional space; then, the rotation matrix of any common video frame relative to the reference image is calculated based on the rotation matrix and translation vector between adjacent images: , ; Then, the yaw angle of the ordinary video frame relative to the reference image is calculated according to the rotation matrix of the ordinary video frame relative to the reference image; Specifically, the rotation matrix of the ordinary video frame relative to the reference image is converted into Euler angles, and the rotation order is ryxz (i.e., rotation is performed in the order of internal rotation from y to z to x), and the yaw angle yaw of the Euler angle is taken to rotate the image according to the yaw angle to achieve camera perspective alignment. Among them, the perspective alignment effect is as follows Figure 2 shown.

[0032] S104: loop-close marking the video frames in the second image sequence to obtain three-dimensional reconstruction data.

[0033] In some embodiments, loop marking is performed on video frames in the second image sequence, including: establishing a landmark point relationship graph based on the feature point matching relationship between the video frames in the second image sequence, so as to characterize the co-viewing relationship of the video frames in the second image sequence through the landmark point relationship graph; clustering is performed based on the landmark point relationship graph to obtain multiple image categories; for each video frame in the second image sequence, the distance between the current video frame and each image category is calculated, and the image category with the smallest distance is taken as the target image category; the corresponding video frame is selected from the target image category to perform feature point matching with the current video frame, so as to determine the loop frame corresponding to the current video frame according to the matching result; and an adjacent relationship between the current video frame and the loop frame is established.

[0034] In some embodiments, a landmark point relationship graph is established based on the feature point matching relationship between video frames in the second image sequence, including: for an image pair formed by any two video frames in the second image sequence, initializing landmark points based on the feature point pairs of the image pairs; for a feature point in any one video frame, determining whether the feature point matches feature points in multiple images; if so, merging the feature points that match the feature point into the same landmark point; and constructing a graph structure in which nodes represent video frames or landmark points, thereby representing the connection relationship between video frames and landmark points.

[0035] As an example, first, the landmark point Track represents the landmark point in the three-dimensional space, which can be projected to the feature points in different images to represent the co-viewing relationship between different images. Since there are multiple feature points in a video frame, the definition For video frames No. feature points, define For video frames No. feature points, if and is the video frame and video frames A matching pair, then we correspond to the same landmark; if and , and is a matching pair, and the feature points , and If there is no other feature point threshold match, then they correspond to a landmark point with a length of 3, that is, the landmark point track_len=3, where track_len is the number of feature points corresponding to the landmark point. Then, using the definition of landmark points, the landmark point relationship graph of all video frames in the second image sequence can be constructed; it should be noted that the landmark point relationship graph includes the following information: the connection between the video frame and the landmark point and the feature point matching pairs of adjacent images.

[0036] Specifically, first, a landmark point is created for each feature point matching pair, and all image pairs are traversed; for each pair of matched feature points, check whether the feature point pair belongs to a landmark point; if they do not belong to any landmark point, create a new landmark point and add the feature point pair to the newly added landmark point; if one of the feature points of the feature point pair already belongs to a landmark point, the other feature point is also added to the landmark point; the data structure uses a hash table or a union-difference data structure; then, the landmark points are expanded, that is, the matched feature points are merged into the same landmark point to ensure that the landmark point contains all co-viewed feature points; specifically, all video frames are traversed, and for each feature point in the current video frame, check whether the feature point matches feature points in multiple images; if the feature point matches multiple feature points, all feature points that match the feature point are merged into the same landmark point, and the data structure uses a union-difference data structure to efficiently merge landmark points. Then, a landmark point relationship graph is constructed; that is, a graph structure describing the connection relationship between images and landmark points is constructed; specifically, a graph structure G is first initialized, in which nodes represent video frames and landmark points, and edges represent the connection relationship between video frames and landmark points; then, all landmark points are traversed, and for each landmark point, all feature points connected to it are found, and the video frames and landmark points are connected to form edges in the graph G; then, all image pairs are traversed, and if two images share the same landmark point, an edge is added to the graph G to represent the feature point matching relationship between the two images; a graph data structure is used to represent the landmark point relationship graph, in which nodes can be images and landmark points, and edges represent connection relationships.

[0037] As an example, suppose we have the following video frame sequence and feature point matching pairs: Image Feature points and images Feature points Matching; Image Feature points and images Feature points Matching; Image Feature points and images Feature points Match; then, initialize the landmarks and create landmarks , and and Add to Create waypoints , and and Add to Create waypoints , and and Add to Then, expand the landmark points and check whether there are feature points that need to be merged; assuming that and belong to the same landmark point, then and Merge; then, construct a landmark point relationship graph; image Connect to and ;image Connect to waypoint and ;image Connect to waypoint and .

[0038] In some embodiments, clustering is performed based on a landmark point relationship graph to obtain multiple image categories, including: for any video frame in the second image sequence, calculating a global feature corresponding to the current video frame; obtaining a similar video frame corresponding to the current video frame in the second image sequence based on the global feature; and clustering is performed based on the landmark point relationship graph and the similar video frames to obtain multiple image categories.

[0039] As an example, first, for any video frame, if there is an image frame that is far apart in time but has similar spatial position and orientation, then the image frame is the loop frame of the current video frame; it can be understood that the loop frame can provide some longer time interval constraints beyond adjacent frames, reducing the cumulative error problem caused by long-term movement.

[0040] Specifically, first, the global features of all video frames are calculated. The global features are the overall attributes of the image. Images with similar colors, textures, and shapes will also have similar global features. The global features can be calculated using methods such as BOW and Netvald. Then, the global features are used to calculate the K most similar images of each video frame in the second image sequence (excluding the K nearest neighbors of the current video frame, and the specific value of K can be selected as needed). Then, the K most similar images are clustered according to the landmark point relationship graph, and images with common landmark points are clustered into one category.

[0041] Assume that the value of K is 8, and the 8 most similar images are f1, f3, f4, f6, f20, f25, f27 and f45, and the landmarks connected to them are 1, 2, 3, 100, 102 and 350. First, initialize the image class name with the serial number of all landmarks connected to each image in the most similar K images. Each image can correspond to multiple class names (for example, f1 corresponds to class names 1 and 2); then, sort the class name of each image, and map all class names corresponding to the image (except the smallest class name) to the smallest class name corresponding to the image (such as f3, the corresponding class names 2 and 3 are mapped to class 1); then, merge all class name mappings until there is no mapping value for the corresponding class name; then, for the mapping of 3->2, the mapping value is 2. Since there is 2->1 in the class name mapping, merge to get 3->1, and its mapping value is 1. At this time. There is no mapping value of 1 in the class name mapping, and the merge ends. The merged class name mapping is used as the final class name, until each image corresponds to only one class name. At this time, this class name is the cluster to which the image belongs (for example, the class names 1, 2, 3 corresponding to f3 are modified to 1, 1, 1, so the cluster name of f3 is 1); then, calculate each cluster for the current frame Distance: ; in, represents the set of all images in each cluster, Representing a collection The number of images in Next, from the current frame From the cluster with the smallest distance, take out the five images with the highest similarity and match their feature points with the current frame respectively; then, take the image with the most matching points as the loop frame of the current frame, and establish the adjacent relationship between the current frame and the loop frame.

[0042] Specifically, first, add feature point matching pairs between the current frame and the loop frame; then, replace the loop frame The landmark point, that is, for and All matching pairs and ,delete The corresponding landmark point and connect it to The corresponding landmark point.

[0043] In some embodiments, Figure 3 As shown, the method also includes: S201, setting the video frames in the second image sequence to an initial state; S202, traversing the second image sequence to determine whether there is a video frame in the second image sequence that is in the initial state; if yes, executing step S203; if not, executing step S211; S203, taking the video frame with the largest landmark length among the video frames in the initial state as the current video frame; S204, acquiring an adjacent image corresponding to the current video frame according to the landmark point relationship graph; S205, determining whether the number of adjacent images is 1; if yes, executing step S206; if no, executing step S207; S206, changing the state of the current video frame to a reserved state, and returning to step S202; S207, pre-deleting the current video frame, updating the landmark point relationship graph according to the pre-deletion result, and calculating the average landmark point length of the updated landmark point relationship graph; S208, determining whether the average landmark length is greater than or equal to a preset length threshold; if yes, executing step S209; if no, executing step S210; S209, delete the current video frame and return to step S202; S210, changing the state of the current video frame to a reserved state, and returning to step S202; S211, taking the video frame in the reserved state as a key frame.

[0044] As an example, first, for the image frame , the number of local feature points is , Local feature points The length of the corresponding landmark point, image frame The length of the landmark is: ; It should be noted that the landmark length threshold TL can be set according to the mapping precision requirement. The larger the value of TL, the finer the mapping result.

[0045] Next, all images are set to an initialization state, which means that the image is unprocessed; then, the image fp with the largest landmark length and the adjacent images of fp are taken from the images in the initialization state. Adjacent images are images with matching pair relationships in the landmark point relationship graph, and the number is greater than one; then, if there is only one adjacent image, fp is marked as a retained state; if there are more than one adjacent images; then, the image with the largest landmark length is removed from the landmark point relationship graph, and the landmark point relationship graph is regenerated; that is, this step attempts to delete the image with the largest landmark length; if after deletion, the landmark point relationship graph can meet the set conditions, the image is discarded; if the landmark point relationship graph cannot meet the set conditions, the image is retained. Specifically, the image frames in the adjacent images are paired to perform image feature point matching. Define the connection condition: for any matching pair ,if , then the image Satisfy the conditions for establishing a feature point matching pair relationship; select the two images with the largest number of matches from the adjacent images. If the connection condition is met, connect the two images with the largest number of matches, that is, establish a feature point matching pair relationship between the two images, and mark the two images as connected. If the connection condition is not met, mark the fp taken out as a keep state; then, select the image with the largest number of matches from the unconnected adjacent images and the connected image. If the connection condition is met, connect and mark it as connected. If the connection condition is not met, mark the fp taken out as a keep state; repeat the above steps until all adjacent images are marked as connected; calculate the average landmark point length of the newly generated landmark point relationship graph; if the average landmark point length is greater than the landmark point length threshold, the fp taken out is set to a discarded state. Finally, for all images in the retained state, use them as key frames for 3D reconstruction input.

[0046] In summary, according to the method for generating 3D reconstruction data of an embodiment of the present invention, first, a panoramic video is obtained, and the panoramic video is extracted frame by frame to obtain a corresponding first image sequence; then, local feature points corresponding to each video frame in the first image sequence are extracted, and feature point matching pairs between adjacent images are calculated based on the local feature points; then, the first image sequence is filtered based on the feature point matching to obtain a second image sequence; then, the video frames in the second image sequence are loop-marked to obtain 3D reconstruction data. This effectively improves the efficiency of 3D reconstruction and at the same time, improves the success rate of 3D reconstruction.

[0047] In a second aspect, an embodiment of the present invention proposes a computer-readable storage medium on which a three-dimensional reconstruction data generation program is stored. When the three-dimensional reconstruction data generation program is executed by a processor, the three-dimensional reconstruction data generation method as described above is implemented.

[0048] In a third aspect, an embodiment of the present invention provides a three-dimensional reconstruction data generation system, such as Figure 4 As shown, the three-dimensional reconstruction data generation system includes: an acquisition module 10, an extraction module 20, a filtering module 30 and a marking module 40.

[0049] The acquisition module 10 is used to acquire the panoramic video and extract the panoramic video frame by frame to obtain the corresponding first image sequence; The extraction module 20 is used to extract local feature points corresponding to each video frame in the first image sequence, and calculate feature point matching pairs between adjacent images based on the local feature points; The filtering module 30 is used to filter the first image sequence according to feature point matching to obtain a second image sequence; The marking module 40 is used to perform loop-closure marking on the video frames in the second image sequence to obtain three-dimensional reconstruction data.

[0050] In addition, it should be noted that the above description of the three-dimensional reconstruction data generating method is also applicable to the three-dimensional reconstruction data generating system, and will not be elaborated here.

[0051] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0052] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0053] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0054] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0055] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0056] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0058] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for generating three-dimensional reconstruction data, characterized in that: The following steps are involved: Acquire a panoramic video, and extract the panoramic video frame by frame to obtain a corresponding first image sequence; Extracting local feature points corresponding to each video frame in the first image sequence, and calculating feature point matching pairs between adjacent images based on the local feature points; filtering the first image sequence according to the feature point matching to obtain a second image sequence; Performing loop closure marking on the video frames in the second image sequence to obtain three-dimensional reconstruction data; Wherein, redundant image filtering is performed on the first image sequence according to the following formula: ； in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a first preset threshold; If the filtering condition is met, then remove the frame; The occlusion or motion blurred image filtering is performed on the first image sequence according to the following formula: ； in, Indicates the first image in the sequence Frame and The number of feature point matching pairs between frames, represents a second preset threshold; If the filtering condition is met, then remove the Frame and Video frames between frames.

2. The method for generating three-dimensional reconstruction data according to claim 1, wherein: The second image sequence includes a common video frame and a reference image. After filtering the first image sequence according to the feature point matching to obtain the second image sequence, the method further includes: Calculating the rotation matrix and translation vector of all adjacent images in the second image sequence; For any common video frame, calculating a rotation matrix of the common video frame relative to the reference image according to the rotation matrix of the adjacent image; Converting the rotation matrix of the ordinary video frame relative to the reference image into Euler angles, and using the Euler angles as the yaw angles corresponding to the ordinary video frame; The ordinary video frame is rotated according to the yaw angle to align the camera perspective of the second image sequence.

3. The method for generating three-dimensional reconstruction data according to claim 1, wherein: Performing loop closure marking on the video frames in the second image sequence includes: Establishing a landmark point relationship graph according to the feature point matching relationship between the video frames in the second image sequence, so as to characterize the co-viewing relationship of the video frames in the second image sequence through the landmark point relationship graph; Performing clustering based on the landmark point relationship graph to obtain multiple image categories; For each video frame in the second image sequence, calculating the distance between the current video frame and each of the image categories, and taking the image category with the smallest distance as the target image category; Selecting a corresponding video frame from the target image category to perform feature point matching with the current video frame, so as to determine a loop frame corresponding to the current video frame according to the matching result; An adjacent relationship between the current video frame and the loop frame is established.

4. The method for generating three-dimensional reconstruction data according to claim 3, wherein: Establishing a landmark point relationship graph according to the feature point matching relationship between the video frames in the second image sequence includes: For an image pair formed by any two video frames in the second image sequence, initializing landmark points according to a pair of feature points of the image pair; For any feature point in a video frame, determine whether the feature point matches the feature points in multiple images; If yes, the feature points matching the feature point are merged into the same landmark point; A graph structure is constructed, in which nodes represent video frames or landmark points, thereby representing the connection relationship between video frames and landmark points.

5. The method for generating three-dimensional reconstruction data according to claim 3, wherein: Clustering is performed based on the landmark point relationship graph to obtain multiple image categories, including: For any video frame in the second image sequence, calculating a global feature corresponding to the current video frame; Acquire a similar video frame corresponding to the current video frame in the second image sequence according to the global feature; Clustering is performed based on the landmark point relationship graph and the similar video frames to obtain multiple image categories.

6. The method for generating three-dimensional reconstruction data according to claim 4, characterized in that: The method further comprises: S201, setting the video frames in the second image sequence to an initial state; S202, traversing the second image sequence to determine whether there is a video frame in the second image sequence that is in an initial state; if yes, executing step S203; if no, executing step S211; S203, taking the video frame with the largest landmark length among the video frames in the initial state as the current video frame; S204, acquiring an adjacent image corresponding to the current video frame according to the landmark point relationship graph; S205, determining whether the number of adjacent images is 1; if yes, executing step S206; if no, executing step S207; S206, modifying the state of the current video frame to a reserved state, and returning to step S202; S207, pre-deleting the current video frame, updating the landmark point relationship graph according to the pre-deletion result, and calculating the average landmark point length of the updated landmark point relationship graph; S208, determining whether the average landmark length is greater than or equal to a preset length threshold; if yes, executing step S209; if no, executing step S210; S209, deleting the current video frame and returning to step S202; S210, changing the state of the current video frame to a reserved state, and returning to step S202; S211, taking the video frame in the reserved state as a key frame.

7. A computer-readable storage medium, characterized in that: A three-dimensional reconstruction data generation program is stored thereon, and when the three-dimensional reconstruction data generation program is executed by a processor, the three-dimensional reconstruction data generation method according to any one of claims 1 to 6 is implemented.

8. A three-dimensional reconstruction data generation system, characterized in that: include: An acquisition module, the acquisition module is used to acquire a panoramic video and extract the panoramic video frame by frame to obtain a corresponding first image sequence; An extraction module, the extraction module is used to extract local feature points corresponding to each video frame in the first image sequence, and calculate feature point matching pairs between adjacent images based on the local feature points; A filtering module, the filtering module is used to filter the first image sequence according to the feature point matching to obtain a second image sequence; A marking module is used to perform loop marking on the video frames in the second image sequence to obtain three-dimensional reconstruction data.

Images