Scene reconstruction method, electronic device and medium based on two-dimensional image

Abstract

Embodiments of the present invention provide a two-dimensional image-based scene reconstruction method, electronic device, and medium. These methods acquire multiple scene images containing visual tags, determine camera pose information associated with each scene image based on the imaging of the visual tags in each scene image, spatially sort the multiple scene images based on the camera pose information associated with the scene images, and construct a three-dimensional scene model based on the sorted scene images. This method can rapidly complete the spatial sorting of large-scale two-dimensional images, thereby enabling rapid and accurate reconstruction of a three-dimensional scene model.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image science, and in particular to a method, electronic device and storage medium for reconstructing a three-dimensional scene model using a two-dimensional image. Background Art

[0002] Computing 3D features of a scene from multiple 2D images and reconstructing a 3D scene model is an important research direction in computer vision and graphics. To reconstruct a 3D scene model from 2D images, one approach is to use a dedicated camera with precise position and pose (hereinafter collectively referred to as pose) to capture a large number of 2D images of the scene. The camera's pose information is recorded for each image. This pose information can then be used to spatially sort the images and reconstruct the 3D scene. While this approach offers high accuracy, it requires specialized personnel using dedicated cameras, which is costly. Another approach is to reconstruct a 3D scene from unordered 2D images. This approach does not require a dedicated camera to record the pose information for each image. Instead, feature matching is computed pairwise across all 2D images, allowing them to be spatially sorted for 3D reconstruction. However, computing feature matching pairwise across all 2D images is a very time-consuming task, with a time complexity of O(n^2); for example, for 1000 images, 1000*1000 / 2 = 1000 feature matching operations are required. Furthermore, this method of calculating feature matching is prone to errors and can lead to incorrect reconstruction results, which is particularly serious for images with repeated textures or similar scenes. Therefore, there is currently no better method for 3D scene reconstruction from large-scale, disorderly collected 2D images. Summary of the Invention

[0003] The solution of the present invention provides a scene reconstruction method based on two-dimensional images, electronic equipment and medium, so as to quickly and accurately reconstruct a three-dimensional scene model.

[0004] The above objectives are achieved through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present invention, a method for scene reconstruction based on two-dimensional images is provided, comprising: step a) acquiring, for a scene to be reconstructed, a plurality of scene images, each scene image containing at least one visual tag pre-arranged in the scene; step b) determining, based at least in part on the imaging of the visual tag in each scene image, camera pose information associated with the scene image; and step c) spatially sorting the plurality of scene images based on the camera pose information associated with the scene images and establishing a three-dimensional scene model based on the sorted scene images.

[0006] In some embodiments of the present invention, step c) may include selecting scene images associated with the same visual label from the acquired multiple scene images; spatially sorting the selected scene images; and performing local scene reconstruction based on the sorted scene images. Step c) may also include filtering out portions of the scene images from the selected scene images based on camera pose information associated with the scene images.

[0007] In some embodiments of the present invention, step c) may include: selecting scene images associated with the same camera position and / or the same camera posture from the multiple acquired scene images based on the camera pose information of the scene images; and spatially sorting the selected scene images and performing local scene reconstruction based on the sorted scene images.

[0008] In some embodiments of the present invention, the camera pose information associated with the scene image determined in step b) is at least one of the following: the pose information of the camera relative to the visual tags contained in the scene image when the scene image is captured; the pose information of the camera in the world coordinate system when the scene image is captured; or the pose information of the camera in the scene coordinate system when the scene image is captured.

[0009] In some embodiments of the present invention, step b) may include: determining the posture information of the camera relative to the visual tag when shooting the scene image based on the imaging of the visual tag in each scene image; identifying the identification information transmitted by the visual tag in each scene image; obtaining the posture information of the visual tag in the world coordinate system and / or the scene coordinate system based on the identification information; and determining the posture information of the camera in the world coordinate system and / or the scene coordinate system when shooting the scene image based on the posture information of the camera relative to the visual tag and the posture information of the visual tag in the world coordinate system and / or the scene coordinate system.

[0010] In some embodiments of the present invention, the method may further include: acquiring new scene images, wherein each new scene image contains at least one visual tag pre-arranged in the scene; determining camera pose information associated with each new scene image based at least in part on the imaging of the visual tag in the new scene image; based on the camera pose information associated with each new scene image, selecting other scene images or scene features associated with the camera pose information; and reconstructing a local scene model based on the new scene image and the selected other scene images or scene features.

[0011] According to a second aspect of an embodiment of the present invention, a scene reconstruction device based on a two-dimensional image is also provided, which includes a scene image acquisition module, a camera pose determination module and a scene reconstruction module. The scene image acquisition module is used to acquire a plurality of scene images for a scene that needs to be reconstructed, each scene image containing at least one visual tag pre-arranged in the scene. The camera pose determination module is used to determine the camera pose information associated with each scene image based at least in part on the imaging of the visual tag in the scene image. The scene reconstruction module is used to spatially sort the plurality of scene images based on the camera pose information associated with the scene images and establish a three-dimensional scene model based on the sorted scene images.

[0012] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising an image acquisition device and a controller. The image acquisition device is configured to acquire scene images of a scene to be reconstructed, each acquired scene image containing at least one visual tag pre-arranged in the scene. The controller is configured to determine camera pose information associated with each scene image based at least in part on the imaging of the visual tag in the scene image; and spatially sort the multiple scene images based on the camera pose information associated with the scene images and establish a three-dimensional scene model based on the sorted scene images.

[0013] According to a fourth aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it can be used to implement the method according to the first aspect of the embodiment of the present invention.

[0014] According to a fifth aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, it can be used to implement the method according to the first aspect of an embodiment of the present invention.

[0015] Compared with the existing technology, the solution of the embodiments of the present application can utilize visual tags such as optical communication devices and QR codes arranged in real scenes to determine the position and posture information of the camera when taking each two-dimensional image, and quickly complete the spatial sorting of large-scale two-dimensional images based on such posture information, so as to quickly and accurately reconstruct the three-dimensional scene model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0017] Figure 1 Schematic diagram of an exemplary imaging process of an image acquisition device;

[0018] Figure 2A schematic flow chart of a two-dimensional image-based scene reconstruction method according to an embodiment of the present invention is shown;

[0019] Figure 3 A schematic diagram of an example process of filtering a scene image based on pose information of the scene image;

[0020] Figure 4 A functional module diagram of an apparatus for scene reconstruction based on a two-dimensional image according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] For the convenience of description, first combine Figure 1 The technologies related to the present invention are briefly described to help understand the embodiments of the present invention, but it should be noted that these technical descriptions do not necessarily constitute prior art.

[0023] The imaging model of an imaging device such as a camera (also known as an image acquisition device) reflects the mapping relationship from a spatial entity object to its image. The spatial position of an object point P can usually be identified by its coordinates in a reference coordinate system. The reference coordinate system can be a physical world coordinate system (abbreviated as the object coordinate system), or a scene coordinate system (for example, a coordinate system established for a room, building, park, etc.), or a coordinate system established with a spatial reference object at a known position as the origin. The position F of the imaging device is used as the origin. c A coordinate system established with an origin is called an image coordinate system, such as a camera coordinate system. A two-dimensional coordinate system established within the imaging plane, with the upper left corner of the image captured by the imaging device as the origin, is called an image plane coordinate system, or image coordinate system. The origin of a camera coordinate system is the optical center of the camera, with its Z axis perpendicular to the image plane along the camera's optical axis. The intersection of the camera's optical axis (i.e., the Z axis) and the image plane is the principal point.

[0024] like Figure 1 As shown, assume that the coordinates of the object point P in the reference coordinate system are (X, Y, Z), the corresponding image point is q, its coordinates in the camera coordinate system are (x, y, z), its coordinates in the image coordinate system are (u, v), and the coordinates of the principal point in the image coordinate system are (c x , c y ). The relationship between the coordinates (X, Y, Z) of the object point P in the reference coordinate system and the coordinates (x, y, z) in the camera coordinate system can be expressed as:

[0025]

[0026] Define variables: x′ = x / z, y′ = y / z;

[0027] Then, the coordinates of the object point P in the image plane coordinate system are:

[0028] u=f x *x′+c x And v = f y *y′+c y (2);

[0029] Among them, f x and f y are the focal lengths of the imaging device in the x-axis and y-axis directions, c x , c y is the coordinate of the principal point in the image plane coordinate system, f x 、f y 、c x 、c y Both are internal parameters of the imaging device and can be measured in advance. The rotation matrix R and the displacement vector t represent the relative position and attitude relationship between the camera coordinate system and the reference coordinate system, respectively, and are called external parameters of the imaging device. In three-dimensional space, rotation can be decomposed into two-dimensional rotations around their respective coordinate axes. If the angle ψ is rotated around the x, y, and z axes in sequence, and θ, then the total rotation matrix R is three matrices R x (ψ), R z (θ), that is: in,

[0030]

[0031]

[0032]

[0033] For the sake of simplicity, and because it is well known in the art, the calculation is not expanded here, and the rotation matrix is ​​simply written as follows:

[0034]

[0035] The displacement vector t can be simply written as follows:

[0036]

[0037] So we get the following relationship:

[0038]

[0039] Among them, s is the object-image conversion factor, which is equal to the ratio of the size of the image plane to the resolution of the imaging device, and is also known. In fact, the external parameter rotation matrix R and displacement vector t of the imaging device can also be used to describe or determine the position and posture of the origin of the camera coordinate system of the imaging device relative to the reference coordinate system. The internal parameters of the imaging device can be measured in advance by the device itself, and the determination of the external parameters R and t of the imaging device can be completed through the calibration process of the imaging device. Generally, the external parameters of the imaging device can be determined by using the imaging device to capture images of calibration objects such as a plane checkerboard calibration plate or other markers whose positions in the reference coordinate system are known, thereby determining the transformation or mapping relationship between the camera coordinate system of the imaging device and the reference coordinate system.

[0040] That is to say, by capturing a two-dimensional image containing a calibration object or marker whose posture is known through an imaging device, the relative position and posture information (collectively referred to as posture information) of the imaging device relative to the calibration object or marker when the two-dimensional image is captured can be determined; and further, based on the posture information of the calibration object or marker in the world coordinate system or the scene coordinate system, the determined relative posture information of the imaging device can be converted into the posture information of the imaging device in the world coordinate system or the scene coordinate system.

[0041] For the convenience of description, the above-mentioned calibration objects or markers may be collectively referred to as "visual tags". In one embodiment of the present invention, one or more visual tags are arranged in a real scene for which a three-dimensional scene model is to be established. The imaging device may use a relative positioning method to determine the position and posture information of the device relative to the visual tag by performing image acquisition on a pre-set visual tag. The visual tag may take any form that facilitates a device having an image acquisition device to determine its posture information. For example, the visual tag may be a marker with pre-calibrated physical size and shape. When the device performs image acquisition on the visual marker, the position and posture information of the device relative to the visual marker may be obtained based on the physical size and shape of the visual tag, the imaging of the visual tag in the acquired image, the focal length parameters when the image is acquired, and the like. For another example, the visual tag may be a marker with a plurality of points with pre-calibrated positions or a marker with a plurality of points with known positions set thereon, such as four points A, B, C and D. The corresponding image points in the image acquired for these points are A', B', C' and D'. After the device acquires the images of these points, refer to the above in combination with Figure 1 The description of the position coordinate information of the four points (X A , Y A , Z A )、(X B , Y B , Z B )、(X C , Y C , Z C) and (X D , Y D , Z D ) and the coordinates of the four corresponding image points A', B', C' and D' on the imaging plane (u A’ , v A’ )、(u B’ , v B’ )、(u C’ , v C’ ) and (u D’ , v D’ ), substitute into the above relation (3), solve the rotation matrix R and displacement vector t, and then we can get the position and posture information of the device relative to the visual tag.

[0042] In some embodiments, the position information of the visual tag in the world coordinate system or the scene coordinate system may be known or pre-calibrated, and then it can be combined with the position information of the imaging device relative to the visual tag to determine the position information of the imaging device in the world coordinate system or the scene coordinate system. In other embodiments, it is not necessary to know the position information of the visual tag in the world coordinate system or the scene coordinate system, and it is only necessary to be able to determine the position information of the device relative to the visual tag by analyzing the imaging of the visual tag.

[0043] In the following, an optical communication device will be used as an example of a visual tag for illustration. The optical communication device (also referred to as an optical tag) can transmit different information by emitting different lights, such as the optical communication devices described in Chinese patent publications CN104168060A, CN105740936A, patent applications CN201711374915.9, CN201711374042.1, and CN201711375274.9. In one example, a device with an image acquisition device can be used to capture an image of the optical tag to obtain identification information transmitted by the optical tag, and based on the identification information, information related to the optical tag is obtained, such as the location information, model information, physical size information, physical shape information, posture or orientation information of the optical tag. The device determines the position information of the device relative to the optical tag based on the acquired information related to the optical tag and the captured optical tag image, and further obtains the position information of the device in the reference coordinate system based on the position information of the optical tag in the reference coordinate system.

[0044] Figure 2 A flow chart of a method for scene reconstruction based on a two-dimensional image according to an embodiment of the present invention is given. Figure 2As shown, in step S201, multiple scene images are acquired for a scene to be reconstructed, each scene image containing at least one optical communication device pre-placed in the scene. In one embodiment, one or more optical communication devices may be pre-placed in the scene to be reconstructed, with each optical communication device located at a fixed position in the scene. In some embodiments, the device used to capture scene images can be any device capable of image capture, such as a user-carryable or user-controllable device such as a mobile phone with a camera, a tablet computer, smart glasses, AR glasses, a smart helmet, a smart watch, or a car; or a machine capable of autonomous movement, such as a drone, a self-driving car, or a robot. In other embodiments, the device used to capture scene images can be any computing device capable of image capture and, through a built-in application, analyzing the image of an optical communication device in the image to identify the information conveyed by the optical communication device. In still other embodiments, the device used to capture scene images can be any computing device capable of image capture and determining the positional information of the device relative to the optical communication device based on the image of the optical communication device in the captured image. In embodiments of the present invention, there are no special requirements for the device performing image capture and its operator, as long as it can capture an image of a scene containing an optical communication device. Different operators may use different devices to capture different local scenes, or different operators or the same operator may use the same device to capture different local scenes. It should be noted that acquiring multiple scene images in step S201 may refer to obtaining scene images by photographing, for example, by directly photographing the scene using a device performing image capture to obtain scene images; or it may refer to obtaining scene images by receiving, for example, by a device performing scene reconstruction receiving relevant scene images from one or more devices performing image capture. It is understood that the device used for scene reconstruction and the device used for image capture may be the same device or different devices.

[0045] Then, in step S202, the camera pose information associated with the scene image is determined based on the imaging of the optical communication device in each scene image, that is, the pose information of the imaging device (hereinafter referred to as the camera) when shooting the scene image. In some embodiments, the determined camera pose information may be the pose information of the camera relative to the optical communication device in the scene image. The pose information of the device used to shoot the image relative to the optical communication device can be determined in a variety of ways as mentioned above, which will not be repeated here. In some embodiments, the determined camera pose information may be other pose information obtained by transforming the camera relative to the pose information of the optical communication device in the scene image, such as the pose information of the camera in the world coordinate system or the scene coordinate system. As mentioned above, the pose information of the camera in the world coordinate system or the scene coordinate system can be determined based on the pose information of the camera relative to the optical communication device in the scene image and the pose information of the optical communication device in the world coordinate system or the scene coordinate system. In one embodiment, the pose information of the optical communication device in the world coordinate system or the scene coordinate system can be obtained based on the identification information transmitted by the optical communication device. For example, a device for scene reconstruction can obtain information transmitted by the optical communication device (e.g., identification information of the optical communication device) by analyzing the imaging of the optical communication device in the scene image, and obtain the position information of the corresponding optical communication device in the world coordinate system or the scene coordinate system through the information. For another example, a camera or a device on which the camera is installed (e.g., a mobile phone) can identify the information transmitted by the current optical communication device (i.e., the optical communication device in the scene image) (e.g., identification information of the optical communication device) when shooting a scene image containing the optical communication device, and send the information together with the scene image to the device for scene reconstruction. Afterwards, the device for scene reconstruction can obtain the position information of the corresponding optical communication device in the world coordinate system or the scene coordinate system through the identification information. For another example, a camera or a device on which the camera is installed (such as a mobile phone) can identify the information transmitted by the current optical communication device (that is, the optical communication device in the scene image) (for example, the identification information of the optical communication device) when capturing a scene image containing an optical communication device, and obtain the corresponding optical communication device's posture information in the world coordinate system or the scene coordinate system through the identification information. Afterwards, the camera or the device on which the camera is installed can send the posture information together with the scene image to a device for scene reconstruction.

[0046] In one embodiment, the identification information of the visual tag in the scene image captured by the device can also be determined by other means. For example, a wireless positioning device (for example, a UWB positioning device, a WIFI positioning device) can be deployed near the visual tag, and the position of the device can be determined by the wireless positioning device, thereby determining the identification information of the visual tag near the device.

[0047] Continue to refer Figure 2 In step S203, the multiple scene images are spatially sorted based on the camera pose information associated with each scene image, and a three-dimensional scene model is established based on the sorted scene images. Three-dimensional scene reconstruction can be performed using algorithms such as Multi-View Stereo (MVS), Scale-Invariant Feature Transform (SIFT), Structure from Motion (SfM), Multi-view Stereo (MVS), and Poisson Surface Reconstruction (PSR), without limitation. Compared to the prior art, the scene reconstruction method according to an embodiment of the present invention utilizes visual labels placed in the real scene to determine the pose information of the camera when capturing each two-dimensional image. Based on this pose information, the method rapidly completes the spatial sorting of large-scale two-dimensional images, reducing the time cost and computing resource consumption for image feature matching. Furthermore, there are no special requirements for the equipment used to capture the scene images or the operators, which not only reduces the hardware and labor costs of scene reconstruction but also makes the three-dimensional scene reconstruction method more convenient and flexible.

[0048] In some embodiments, in step S203, one group of associated scene images may be selected from the multiple scene images acquired, and then the selected group of scene images may be spatially sorted and local scene reconstruction may be performed based on the sorted group of scene images, and then the above process may be repeated until a complete scene model is obtained or all scene images are processed. In one embodiment, the selected group of associated scene images may be scene images associated with the same optical communication device (i.e., scene images containing the same optical communication device), thereby basically ensuring that the selected group of scene images is for the same local scene. The scene images selected in this way may be conducive to completing the spatial sorting of scene images more quickly, and also facilitate the subsequent scene reconstruction algorithm to complete the local scene reconstruction more quickly and accurately. In one embodiment, before spatially sorting the group of scene images associated with the same optical communication device, further filtering may be performed. For example, images taken when the distance to the optical communication device is too far or too close may be filtered out based on the camera pose information associated with each image. This is because images that are too far or too close are not on the same scale as most images, and there will be fewer matching feature points in the subsequent scene reconstruction, which has little effect on the accuracy of the reconstructed scene, but will result in unnecessary consumption of computing resources. In one embodiment, this may be achieved by Figure 3 The selected set of scene images is further filtered in the manner shown. Figure 3As shown, for multiple scene images associated with the same optical communication device, they can be sorted by their corresponding camera azimuth angles, and the epipolar angles in the epipolar constraints between each scene image are calculated. Then, for these epipolar angles, an epipolar angle is selected at intervals of a certain threshold (e.g., 4-7 degrees) and the scene image associated with the corresponding camera position is determined. This set of scene images, selected using the epipolar angles of two cameras, is more conducive to the stable and accurate calculation of the depth of image feature points during subsequent scene reconstruction.

[0049] In another embodiment, the selected set of associated scene images may refer to a set of scene images whose camera pose information is associated, such as a set of scene images with the same camera position but different poses, or a set of scene images with the same camera pose but different positions, or a set of scene images with camera positions and poses within a certain range, or the correlation of camera pose information may be defined based on actual needs. It should be understood that the term "identical" here does not specifically mean completely identical, but can be identical or similar, and a certain flexibility can be given based on actual needs.

[0050] In the above embodiment, the spatial sorting of the selected scene images can be, for example, spatial sorting based on the camera position and / or posture of each scene image. For example, scene images that are at the same distance from the optical communication device but shot at different angles relative to the optical communication device can be sorted from left to right, or from right to left, or scene images that are shot at the same angle but at different distances from the optical communication device can be sorted from far to near, or from near to far. The aforementioned sorting method can be adjusted according to actual needs and is not limited to this.

[0051] In the above embodiment, a group of associated scene images is selected to quickly complete the spatial sorting of some two-dimensional images related to a specific local scene, thereby enabling faster and more accurate reconstruction of a three-dimensional scene model.

[0052] In one embodiment, in addition to obtaining multiple scene images containing visual labels, some images that do not include visual labels can also be obtained. These images that do not include visual labels can be applied to three-dimensional scene reconstruction using traditional methods.

[0053] In actual situations, the scene model is usually not reconstructed in one go, and it is often necessary to continuously supplement and extend the reconstruction of certain local scenes. This process can be called incremental reconstruction. In some embodiments of the present invention, in order to better perform subsequent incremental reconstruction, during the scene reconstruction process, for example, the edge areas of the scene model or local scenes with poor reconstruction effects, the scene image or relevant features of the scene image related to the local scene (for example, the camera pose information associated with the scene image, the depth of the calculated scene image feature points, etc.) will be retained. For example, the relevant image information of the local scenes involved in these camera poses can be retained by retaining the scene images associated with one or more specified camera poses. In this way, after acquiring the newly captured scene images, these retained related scene images and their features can be combined for incremental reconstruction. For example, after capturing a local scene that needs to be supplemented or extended, newly captured scene images related to the local scene that needs to be reconstructed are acquired; after determining the camera pose information associated with each newly acquired scene image as described above, the scene image associated with the camera pose information associated with the newly acquired scene image can be selected from the previously retained scene images; and then, incremental reconstruction of the local scene can be performed based on the newly acquired scene image and the selected scene image. In one example, the incremental reconstruction process may include the following steps:

[0054] Step S1: Acquire multiple new scene images including optical communication devices. This step is similar to step S201 above and will not be described in detail here.

[0055] Step S2, determining the camera pose information associated with each new scene image based on the imaging of the optical communication device in the acquired new scene image. This step is similar to step S202 above.

[0056] Step S3: Based on the camera pose information associated with the new scene images, scene images or scene features associated with the newly determined camera pose information are selected from previously retained scene images and / or scene features related to the local scene to be reconstructed. For example, based on the camera pose information associated with the newly acquired image, scene images or scene features with the same or similar camera poses can be selected from the scene images or scene features retained in the previous scene reconstruction process.

[0057] Step S4: incrementally reconstruct the 3D scene model of the local scene to be updated based on the new scene image and the selected scene image or feature. The incremental reconstruction method is the same as the previous scene reconstruction method and will not be repeated here.

[0058] Figure 4The following is a block diagram of the functional modules of an apparatus 400 for scene reconstruction based on two-dimensional images according to one embodiment of the present invention. Although the block diagram depicts components in a functionally separate manner, such depiction is for illustrative purposes only. The components shown in the figure can be arbitrarily combined or divided into independent software, firmware, and / or hardware components. Moreover, regardless of how such components are combined or divided, they can be executed on the same computing device or distributed across multiple computing devices, where the multiple computing devices can be connected by one or more networks.

[0059] like Figure 4 As shown, the device 400 includes a scene image acquisition module 401, a camera pose determination module 402, and a scene reconstruction module 403. The scene image acquisition module 401, as described above in conjunction with step S201, acquires multiple scene images containing optical communication devices for the scene to be reconstructed. The camera pose determination module 402, as described above in conjunction with step 202, determines the pose information of the camera when capturing the scene image by analyzing the imaging of the optical communication device in each scene image. The scene reconstruction module 403, as described above in conjunction with step S203, spatially sorts the multiple scene images based on the determined camera pose information associated with each scene image and establishes a three-dimensional scene model based on the sorted scene images.

[0060] In one embodiment of the present invention, an electronic device for scene reconstruction based on a two-dimensional image is further provided, comprising an image acquisition device and a controller, wherein the controller is configured to execute the method described above in conjunction with steps 201, 202, and 203. In some embodiments, an image display medium (e.g., a display screen) may also be installed on the device. The device may be implemented in a variety of hardware forms, such as mobile phones, tablet computers, smart glasses, AR glasses, VR glasses, smart helmets, smart watches, cars, etc. that are easy for users to carry or control, but it is understood that the device may also be a machine that can move autonomously, such as a drone, an unmanned vehicle, a robot, etc.

[0061] In one embodiment of the present invention, the present invention may be implemented in the form of a computer program. The computer program may be stored in various computer-readable storage media (e.g., a hard disk, an optical disk, a flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

[0062] In another embodiment of the present invention, the present invention may be implemented in the form of an electronic device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of the present invention can be implemented.

[0063] References herein to "various embodiments," "some embodiments," "one embodiment," or "an embodiment" refer to a particular feature, structure, or property described in connection with the embodiment as being included in at least one embodiment. Thus, the appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment" throughout this document do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in connection with one embodiment may be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable. Expressions such as "according to A," "based on A," "through A," or "using A" as appearing herein are intended to be non-exclusive, that is, "according to A" may include "according only to A" or "according to A and B," unless otherwise stated or the context clearly indicates that the meaning is "according only to A." In this application, for the sake of clarity, some exemplary operating steps are described in a certain order. However, those skilled in the art will understand that each of these operating steps is not essential, and some of them can be omitted or replaced by other steps. These operating steps do not have to be performed sequentially in the manner shown. On the contrary, some of these operating steps can be performed in a different order or in parallel according to actual needs, as long as the new execution method is not illogical or inoperable.

[0064] Having thus described several aspects of at least one embodiment of the present invention, it will be appreciated that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended to be within the spirit and scope of the present invention. While the present invention has been described with reference to certain embodiments, the present invention is not limited to the embodiments described herein and encompasses various changes and variations that may be made without departing from the scope of the present invention.

Claims

1. A scene reconstruction method based on a two-dimensional image, comprising: Step a) for a scene to be reconstructed, one or more visual tags are arranged in the scene, and the scene is directly photographed using any device capable of image acquisition to obtain multiple scene images, each scene image containing at least one visual tag pre-arranged in the scene; Step b) determining camera pose information associated with each scene image based at least in part on the imaging of the visual tag in the scene image; Step c) spatially sorting the plurality of scene images based on camera pose information associated with the scene images and establishing a three-dimensional scene model based on the sorted scene images; The step c) includes selecting scene images associated with the same visual label from the multiple scene images obtained, filtering out scene images that are more than a certain distance away from the visual label from the selected scene images based on the camera pose information associated with the scene images, spatially sorting the filtered scene images and reconstructing the local scene based on the sorted scene images, and then repeating the above process until a complete scene model is obtained or all scene images are processed.

2. The method according to claim 1, wherein the camera pose information associated with the scene image determined in step b) is at least one of the following: The pose information of the camera relative to the visual tags contained in the scene image when the scene image is captured; The camera's position in the world coordinate system when capturing the scene image; or The camera's position and posture information in the scene coordinate system when capturing the scene image.

3. The method according to claim 1, wherein said step b) comprises: Determine, based on the imaging of the visual tag in each scene image, the pose information of the camera relative to the visual tag when the scene image is captured; Identify the identity information conveyed by the visual labels in each scene image; Acquire the position information of the visual tag in the world coordinate system and / or the scene coordinate system based on the identification information; The pose information of the camera in the world coordinate system and / or the scene coordinate system when capturing the scene image is determined based on the pose information of the camera relative to the visual tag and the pose information of the visual tag in the world coordinate system and / or the scene coordinate system.

4. The method according to claim 1, further comprising: Acquire new scene images, wherein each new scene image includes at least one visual label pre-arranged in the scene; determining camera pose information associated with each new scene image based at least in part on the imaging of the visual tag in the new scene image; Based on the camera pose information associated with each new scene image, selecting other scene images or scene features associated with the camera pose information; Based on the new scene image and other selected scene images or scene features, a local scene model is reconstructed.

5. A scene reconstruction device based on a two-dimensional image, comprising: A scene image acquisition module is configured to directly capture a scene to be reconstructed using any device capable of image acquisition to acquire a plurality of scene images, wherein one or more visual tags are pre-arranged in the scene, and each acquired scene image contains at least one visual tag pre-arranged in the scene; a camera pose determination module is configured to determine camera pose information associated with each scene image based at least in part on the imaging of the visual tag in the scene image; a scene reconstruction module, configured to spatially sort a plurality of scene images based on camera pose information associated with the scene images and to establish a three-dimensional scene model based on the sorted scene images; The scene reconstruction module is configured to: select scene images associated with the same visual label from the multiple scene images obtained, filter out scene images that are more than a certain distance away from the visual label from the selected scene images based on the camera pose information associated with the scene images, spatially sort the filtered scene images and perform local scene reconstruction based on the sorted scene images, and then repeat the above process until a complete scene model is obtained or all scene images are processed.

6. An electronic device comprising an image acquisition device and a controller, wherein the image acquisition device is configured to capture an image of a scene to be reconstructed by directly photographing the scene, wherein one or more visual tags are pre-arranged in the scene, and each scene image captured by the image acquisition device contains at least one visual tag pre-arranged in the scene, and the controller is configured to: determining camera pose information associated with each scene image based at least in part on the imaging of the visual tag in the scene image; and Based on the camera pose information associated with the scene images, multiple scene images are spatially sorted and a three-dimensional scene model is established based on the sorted scene images. The method includes selecting scene images associated with the same visual label from the multiple acquired scene images, filtering out scene images that are more than a certain distance from the visual label from the selected scene images according to the camera pose information associated with the scene images, spatially sorting the filtered scene images and reconstructing the local scene based on the sorted scene images, and then repeating the above process until a complete scene model is obtained or all scene images are processed.

7. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the computer program can be used to implement the method according to any one of claims 1 to 4.

8. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the computer program can be used to implement the method according to any one of claims 1 to 4.

Images