Three-dimensional reconstruction method and apparatus

Abstract

The application discloses a three-dimensional reconstruction method and device, and relates to the technical field of computer vision. The specific implementation includes: acquiring a video and IMU data captured by a 360-degree panoramic camera during movement; establishing a correlation between each frame of image in the video and IMU data to obtain correlation information; determining VO information and IMU pose estimation according to the correlation information; determining VIO information according to the VO information and IMU pose estimation; determining a key frame image in the correlation information; and performing three-dimensional reconstruction according to the key frame image and VIO information. The application uses a 360-degree panoramic camera to collect video and IMU data, and solves the problem of insufficient scene coverage. The VIO information is used to determine the physical scale information of the panoramic camera, thereby solving the scale recovery problem of the panoramic camera, and improving the robustness in the three-dimensional reconstruction process without adding scene constraints or laser ranging sensors.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of computer vision technology, and in particular to three-dimensional reconstruction methods and apparatus. Background Technology

[0002] In the construction of high-precision visual maps in fields such as MR / AR / VR and robotics, there are two approaches: one is to use multiple ordinary cameras to acquire images and then complete 3D reconstruction through motion recovery structures. However, firstly, the field of view of the camera is small and cannot cover the entire space; secondly, the short baseline between adjacent cameras greatly affects the accuracy of scale recovery, resulting in poor scale recovery accuracy and robustness of 3D reconstruction.

[0003] Another approach uses a panoramic camera. Since scale information cannot be recovered from a single monocular image, scene constraints or laser rangefinders need to be added, which greatly increases the cost of 3D reconstruction. Summary of the Invention

[0004] To address the issue of excessively high costs associated with using panoramic cameras for 3D reconstruction in existing technologies, a 3D reconstruction method, apparatus, electronic device, and storage medium are provided.

[0005] According to a first aspect, a three-dimensional reconstruction method is provided, comprising: acquiring video and IMU data captured by a 360° panoramic camera during movement; establishing the correlation between each frame of the video and the IMU data to obtain correlation information; determining VO information and IMU pose estimation based on the correlation information; determining VIO information based on the VO information and IMU pose estimation; determining keyframe images in the correlation information; and performing three-dimensional reconstruction based on the keyframe images and VIO information.

[0006] According to a second aspect, a three-dimensional reconstruction apparatus is provided, comprising: a first unit for acquiring video and IMU data captured by a 360° panoramic camera during movement; a second unit for establishing a correlation relationship between each frame of the video and the IMU data to obtain correlation information; a third unit for determining VO information and IMU pose estimation based on the correlation information; a fourth unit for determining VIO information based on the VO information and IMU pose estimation; a fifth unit for determining keyframe images in the correlation information; and a sixth unit for performing three-dimensional reconstruction based on the keyframe images and VIO information.

[0007] According to a third aspect, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method as described in any embodiment of the three-dimensional reconstruction method.

[0008] According to a fourth aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in any embodiment of the three-dimensional reconstruction method.

[0009] According to the scheme in this application, video and IMU data are acquired using a 360° panoramic camera to solve the problem of insufficient scene coverage. Based on the correlation information, VO information and IMU pose estimation are determined, and then VIO information is determined, thereby determining the physical scale information of the panoramic camera. This solves the scale recovery problem of the panoramic camera, improves the robustness of the 3D reconstruction process, and reduces the cost of 3D reconstruction without adding scene constraints or laser rangefinders. Attached Figure Description

[0010] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0011] Figure 1 This is an exemplary system architecture diagram in which some embodiments of this application can be applied;

[0012] Figure 2 This is a flowchart of an embodiment of the three-dimensional reconstruction method according to this application;

[0013] Figure 3 This is a schematic diagram of an application scenario of the three-dimensional reconstruction method according to this application;

[0014] Figure 4 This is a flowchart of yet another embodiment of the three-dimensional reconstruction method according to this application;

[0015] Figure 5 This is a schematic diagram of a structure of one embodiment of the three-dimensional reconstruction apparatus according to this application;

[0016] Figure 6 This is a block diagram of an electronic device used to implement the three-dimensional reconstruction method of the embodiments of this application. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. For ease of understanding, some of the technical terms involved in this application are briefly introduced below.

[0019] An IMU (Inertial Measurement Unit) is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

[0020] Visual Odometry (VO) is a visual odometry technique that estimates the motion (i.e., changes in position and attitude) of a moving camera by analyzing consecutive frames of camera images.

[0021] VIO (Visual-Inertial Odometry) is a technique that combines visual camera and inertial measurement unit (IMU) data to achieve position and orientation estimation.

[0022] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the three-dimensional reconstruction method or three-dimensional reconstruction apparatus of this application can be applied.

[0024] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0025] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as video applications, live streaming applications, instant messaging tools, email clients, social media platform software, etc.

[0026] The terminal devices 101, 102, and 103 here can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to wearable head-mounted devices, smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0027] Server 105 can be a server that provides various services, such as a backend server that supports terminal devices 101, 102, and 103. The backend server can analyze and process received data such as video, and feed back the processing results (such as keyframe images) to the terminal devices.

[0028] It should be noted that the three-dimensional reconstruction method provided in this application embodiment can be executed by server 105 or terminal devices 101, 102, 103. Correspondingly, the three-dimensional reconstruction device can be set in server 105 or terminal devices 101, 102, 103.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0030] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of the three-dimensional reconstruction method according to this application. The three-dimensional reconstruction method includes the following steps:

[0031] Step 201: Acquire video and IMU data captured by the 360° panoramic camera during movement.

[0032] In this embodiment, the 3D reconstruction method runs on an execution entity (e.g., Figure 1 The server or terminal device shown can actively acquire or passively receive video and IMU data captured by the panoramic camera. In one example, the panoramic camera consists of front and rear binocular or multi-camera systems, and the IMU data includes three-axis acceleration and three-axis gyroscope data.

[0033] Step 202: Establish the association between each frame of the video and the IMU data to obtain the association information.

[0034] In this embodiment, the execution entity extracts each frame of the video and its corresponding timestamp information, and synchronizes and aligns the IMU data and each frame of the video according to the timestamp information.

[0035] Step 203: Determine VO information and IMU attitude estimation based on the associated information.

[0036] In this embodiment, the aforementioned execution entity converts the panoramic image in the associated information into an isometric projection map. The isometric projection map describes the projection from a unit sphere to the image plane. Then, the extrinsic parameters of the panoramic camera and IMU are calibrated using calibration tools and a checkerboard pattern. The extrinsic parameters include rotation information and translation information.

[0037] The aforementioned execution entity extracts FAST features from the isometric projection map. It's understandable that during the isometric projection transformation, pixels along the u-axis are stretched. We model the observed features using additive Gaussian noise, as follows:

[0038]

[0039] Among them, z f These are the measured values ​​of the feature points. σ is the true value of the feature point, v is the vertical angle of each feature point on the equidistant image plane, and σ is the true value of the feature point. f It is the corresponding noise of pixel features.

[0040] Then, FAST features were tracked using Lucas-Kanade optical flow and triangulated to determine VO information.

[0041] Furthermore, the aforementioned execution entity determines the IMU pose estimation by pre-integrating the IMU data of adjacent frame images in the associated information.

[0042] Step 204: Determine VIO information based on the VO information and IMU attitude estimation.

[0043] In this embodiment, the aforementioned execution entity fuses VO information and IMU pose estimation through a graph optimization framework to determine VIO information.

[0044] Step 205: Determine the keyframe images in the associated information.

[0045] In this embodiment, the aforementioned execution entity determines the keyframe image through preset logic.

[0046] In some embodiments, the execution entity determines the key frame image by means of average disparity. Specifically, the execution entity first determines the average disparity of each frame image in the associated information, and then determines the target frame image as the key frame image when the average disparity of the current frame image (i.e. the target frame image) exceeds a preset threshold relative to the average disparity of the previous frame image.

[0047] In other embodiments, the aforementioned execution entity determines the keyframe image by feature quality, which is determined by the number of features. Specifically, the execution entity determines the number of features for each frame image in the associated information, and if the number of features for the current frame image (i.e., the target frame image) is lower than a preset threshold, the target frame image is determined as a keyframe image.

[0048] Therefore, the embodiments of this application improve the efficiency of 3D reconstruction through more reasonable and efficient keyframe selection.

[0049] Step 206: Perform 3D reconstruction based on the keyframe images and VIO information.

[0050] In this embodiment, the aforementioned execution entity processes keyframe images and VIO information through a monocular vision system to perform aerial triangulation and sparse point cloud generation for subsequent 3D scene reconstruction.

[0051] The method provided in the above embodiments of this application utilizes a 360° panoramic camera to acquire video and IMU data, thus solving the problem of insufficient scene coverage. Based on correlation information, VO information and IMU pose estimation are determined. VIO information is then determined through VO information and IMU pose estimation, thereby determining the physical scale information of the panoramic camera. This solves the scale recovery problem of the panoramic camera while providing more accurate scale constraints, significantly improving the robustness of the 3D reconstruction process without the need for scene constraints or laser rangefinders.

[0052] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the three-dimensional reconstruction method according to this embodiment. Figure 3 In the application scenario, the execution entity 301 correlates the video and IMU data acquired by the panoramic camera to obtain correlation information 302. Then, it processes the correlation information 302 to obtain VO information 303 and IMU pose estimation 304. Afterward, the execution entity calls a graph optimization framework to fuse the VO information 303 and IMU pose estimation 304 to obtain VIO information 305. At the same time, the execution entity also selects keyframe images according to a pre-set strategy. Finally, it performs 3D reconstruction 306 based on the keyframe images and VIO information 305.

[0053] Further reference Figure 4This illustrates a flow 400 of another embodiment of the 3D reconstruction method. This flow 400 applies to step 206 and includes the following steps:

[0054] Step 401: Optimize the VIO information based on the keyframe image.

[0055] In this embodiment, the 3D reconstruction method runs on an execution entity (e.g., Figure 1 When processing associated information, the server or terminal device shown first determines a sliding window area. It can be understood that the associated information is a video stream containing multiple frames of images, and the sliding window area contains multiple consecutive frames of images, including multiple keyframe images.

[0056] As the sliding window moves through the video stream, the aforementioned execution entity calculates the projection error of each landmark point within the sliding window region and determines the IMU pre-integration constraints for adjacent keyframe images. Finally, based on the projection error and IMU pre-integration constraints obtained above, the panoramic camera pose corresponding to each keyframe image is adjusted to optimize the VIO information.

[0057] In some detailed embodiments of this application, the aforementioned execution entity utilizes the Bundle Adjustment (BA) method to optimize the projection error. To save computational resources, at time i, the system state variables to be optimized within the sliding window are defined as follows:

[0058] S = [X] n ,X n+1 ,…,X n+N ,λ m ,λ m+1 ,…,λ m+M ]

[0059]

[0060] Where S represents the system state variable, n and m represent the IMU body state variables, N represents the number of keyframes in the sliding window, M represents the number of landmark points observed in all keyframes within the sliding window, and λ is the inverse depth of the feature points.

[0061] X i Includes the attitude of the IMU body in the inertial coordinate system at time i. Displacement speed And the estimation of the offset of acceleration and angular velocity in the IMU body coordinate system.

[0062] The optimized model for bundle adjustment is as follows:

[0063]

[0064] Where, r p J represents the residual of prior information. p The Jacobian matrix representing the prior residuals with respect to the state vector; r B Represents the measurement residuals of the IMU. This represents the pre-integration information between adjacent keyframes bk and bk+1. The covariance matrix representing the pre-integration; r C Representing the reprojection residual in visual perception. This represents the observed pixel coordinates of the l-th landmark in the j-th frame. Let be the standard covariance of the image.

[0065] All three residuals mentioned above are represented using Mahalanobis distance. The residual of prior information refers to the pose and feature point constraints removed from the sliding window. The IMU measurement residual is generated by the IMU between adjacent frames in the sliding window; the difference between the estimated and measured values ​​is the residual. The visual reprojection residual is the reprojection error of map points in the sliding window onto the isometric images of the included panoramic cameras.

[0066] A sequence of 360-degree panoramic images was established using the aforementioned keyframe images, and the pose constraint relationship between adjacent keyframes was recorded.

[0067] Step 402: Perform 3D reconstruction based on the keyframe image and the optimized VIO information.

[0068] In this embodiment, the execution entity first converts the keyframe image into a multi-view image, and then performs 3D reconstruction based on the multi-view image and the optimized VIO information.

[0069] Therefore, this application further improves the robustness of the 3D reconstruction process by continuously optimizing VIO information.

[0070] In some examples, the aforementioned execution entity obtains a multi-view image set of all keyframe images by performing multi-view projection correction on the 360° keyframe images.

[0071] Specifically, the aforementioned execution entity converts the original input 360° panoramic image into a panoramic sphere space. The execution entity then surrounds this panoramic sphere space with a cube space of the same side length as the panoramic sphere, mapping points on the panoramic sphere surface onto the corresponding cube surfaces, resulting in six multi-view images. Next, the execution entity rotates 45° in a preset direction, again mapping points on the panoramic sphere surface onto the corresponding cube surfaces, obtaining another six multi-view images. Finally, the execution entity selects a corresponding number of corresponding images in both the horizontal and vertical directions from these twelve multi-view images to form a multi-view image set.

[0072] In some preferred examples, the aforementioned execution entity selects no fewer than 6 multi-view images in the horizontal direction and no fewer than 2 multi-view images in the vertical direction to form a multi-view image set.

[0073] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a three-dimensional reconstruction device, which is similar to... Figure 2 Corresponding to the method embodiment shown, in addition to the features described below, the device embodiment may also include [features related to...]. Figure 2 The method embodiments shown have the same or corresponding features or effects. This device can be specifically applied to various electronic devices.

[0074] like Figure 5 As shown, the 3D reconstruction device 500 of this embodiment includes: a first unit 501, a second unit 502, a third unit 503, a fourth unit, a fifth unit, and a sixth unit. Specifically, the first unit 501 is used to acquire video and IMU data captured by a 360° panoramic camera during movement; the second unit 502 is used to establish the correlation between each frame of the video and the IMU data to obtain correlation information; the third unit 503 is used to determine VO information and IMU pose estimation based on the correlation information; the fourth unit 504 is used to determine VIO information based on the VO information and IMU pose estimation; the fifth unit 505 determines the keyframe images in the correlation information; and the sixth unit 506 performs 3D reconstruction based on the keyframe images and VIO information.

[0075] In this embodiment, the specific processing of the first unit 501, the second unit 502, the third unit 503, the fourth unit 504, the fifth unit 505, and the sixth unit 506 of the three-dimensional reconstruction device 500, and the resulting technical effects, can be found by referring to [the relevant documentation]. Figure 2 The relevant descriptions of steps 201, 202, 203, 204, 205, and 206 in the corresponding embodiments will not be repeated here.

[0076] In some optional implementations of this embodiment, the sixth unit 506 is further configured to optimize the VIO information based on the keyframe image; and to perform three-dimensional reconstruction based on the keyframe image and the optimized VIO information.

[0077] In some optional implementations of this embodiment, the sixth unit 506 is further configured to convert the keyframe image into a multi-view image; and to perform three-dimensional reconstruction based on the multi-view image and the optimized VIO information.

[0078] In some optional implementations of this embodiment, the sixth unit 506 is also used to convert the keyframe image to the panoramic sphere space; using a cube space to surround the panoramic sphere space, so as to map the landmark points on the panoramic sphere space to the surface of the cube space to obtain multiple multi-view images; wherein, the side length of the cube space is equal to the diameter of the panoramic sphere space.

[0079] In some optional implementations of this embodiment, the sixth unit 506 is further configured to determine a sliding window region, the sliding window region containing multiple key frame images; calculate the projection error of each landmark point within the sliding window region; determine the IMU pre-integration constraints of adjacent key frame images; and adjust the camera pose corresponding to each key frame image according to the projection error and the IMU pre-integration constraints to optimize the VIO information.

[0080] In some optional implementations of this embodiment, the fifth unit 505 is further configured to determine the average disparity of each frame image in the associated information; in response to the average disparity of the target frame image relative to the average disparity of the previous frame image exceeding a preset threshold, the target frame image is determined as a key frame image.

[0081] In some optional implementations of this embodiment, the fifth unit 505 is further configured to determine the number of features of each frame image in the associated information; in response to the number of features of the target frame image being lower than a preset threshold, the target frame image is determined as a key frame image.

[0082] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0083] like Figure 6 The diagram shown is a block diagram of an electronic device using a three-dimensional reconstruction method according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0084] like Figure 6As shown, the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take the 601 processor as an example.

[0085] The memory 602 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the three-dimensional reconstruction method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the three-dimensional reconstruction method provided in this application.

[0086] Memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the three-dimensional reconstruction method in the embodiments of this application (e.g., appendix). Figure 5 (The first unit 501, the second unit 502, the third unit 503, the fourth unit 504, the fifth unit, and the sixth unit are shown). The processor 601 executes various functional applications and data processing of the server by running non-transient software programs, instructions, and modules stored in the memory 602, thereby implementing the three-dimensional reconstruction method in the above method embodiments.

[0087] Memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the 3D reconstruction electronic device. Furthermore, memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories can be connected to the 3D reconstruction electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0088] The electronic device for the 3D reconstruction method may further include an input device 603 and an output device 604. The processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0089] Input device 603 can receive input digital or character information, and generate key signal inputs related to user settings and function control of the 3D reconstruction electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 604 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0090] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0094] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first unit, a second unit, and a third unit. The names of these units do not necessarily limit the specific unit; for example, the first unit may also be described as "a unit that acquires video and IMU data captured by a 360° panoramic camera during movement."

[0097] In another aspect, this application also provides a computer-readable medium, which may be included in the apparatus described in the above embodiments; or it may exist independently and not assembled into the apparatus. The computer-readable medium carries one or more programs that, when executed by the apparatus, cause the apparatus to: acquire video and IMU data captured by a 360° panoramic camera during movement; establish a correlation relationship between each frame of the video and the IMU data to obtain correlation information; determine VO information and IMU pose estimation based on the correlation information; determine VIO information based on the VO information and IMU pose estimation; determine keyframe images in the correlation information; and perform three-dimensional reconstruction based on the keyframe images and VIO information.

[0098] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A three-dimensional reconstruction method, the method comprising: Acquire video and IMU data captured by a 360° panoramic camera during movement; Extract each frame of the video and its corresponding timestamp information, and establish the association between each frame of the video and the IMU data based on the timestamp information to obtain the association information; The panoramic image in the associated information is converted into an isometric projection map. FAST features are extracted from the isometric projection map. During the conversion of the isometric projection map, additive Gaussian noise is used to model the observed features, wherein the additive Gaussian noise follows the following distribution: ； in, It is the corresponding noise of pixel features. It is the vertical angle of each feature point on the isometric projection map. It is variance; The FAST features are tracked and triangulated to determine the VO information; Pre-integrate the IMU data of adjacent frame images in the associated information to determine the IMU pose estimation; VIO information is determined based on the VO information and IMU attitude estimation; Identify the keyframe images in the associated information; Optimizing the VIO information based on the keyframe images includes: determining a sliding window region containing multiple keyframe images; calculating the reprojection error of each landmark point within the sliding window region on the isometric projection map of the included panoramic camera, and determining the IMU pre-integration constraints of adjacent keyframe images; and adjusting the panoramic camera pose corresponding to each keyframe image based on the reprojection error and the IMU pre-integration constraints to optimize the VIO information. The keyframe images are converted to a panoramic sphere space; The panoramic sphere space is enclosed by a cube space to map the landmark points on the panoramic sphere space onto the surface of the cube space, resulting in multiple multi-view images; wherein the side length of the cube space is equal to the diameter of the panoramic sphere space. Three-dimensional reconstruction is performed based on the multi-view images and optimized VIO information.

2. The method according to claim 1, wherein, Determining the keyframe images in the associated information includes: Determine the average disparity of each frame of the image in the associated information; When the average disparity of the target frame image exceeds a preset threshold relative to the average disparity of the previous frame image, the target frame image is determined as a keyframe image.

3. The method according to claim 1, wherein, Determining the keyframe images in the associated information includes: Determine the number of features for each frame of the image in the associated information; If the number of features in a target frame image is lower than a preset threshold, the target frame image is identified as a keyframe image.

4. A three-dimensional reconstruction device, comprising: The first unit is used to acquire video and IMU data captured by the 360° panoramic camera during movement; The second unit is used to establish the association between each frame of the video and the IMU data to obtain association information; The third unit is used to convert the panoramic image in the associated information into an isometric projection map, extract FAST features from the isometric projection map, and model the observed features using additive Gaussian noise during the conversion process, wherein the additive Gaussian noise follows the following distribution: ； in, It is the corresponding noise of pixel features. It is the vertical angle of each feature point on the isometric projection map. The variance is used to track and triangulate the FAST features to determine the VO information; the IMU data of adjacent frame images in the association information are pre-integrated to determine the IMU pose estimate. The fourth unit is used to determine VIO information based on the VO information and IMU attitude estimation; The fifth unit is used to determine the keyframe images in the associated information; The sixth unit is used to optimize the VIO information based on the keyframe images, including: determining a sliding window region containing multiple keyframe images; calculating the reprojection error of each landmark point within the sliding window region on the isometric projection map of the included panoramic camera, and determining the IMU pre-integration constraints of adjacent keyframe images; adjusting the panoramic camera pose corresponding to each keyframe image according to the reprojection error and the IMU pre-integration constraints to optimize the VIO information. The sixth unit is also used to convert the keyframe image to a panoramic sphere space; to surround the panoramic sphere space with a cube space to map the landmark points on the panoramic sphere space onto the surface of the cube space, thereby obtaining multiple multi-view images; wherein the side length of the cube space is equal to the diameter of the panoramic sphere space; and to perform three-dimensional reconstruction based on the multi-view images and the optimized VIO information.

5. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.

Images