Three-dimensional reconstruction method and three-dimensional reconstruction system

Abstract

The invention discloses a three-dimensional reconstruction method and a three-dimensional reconstruction system, and the method comprises the steps: carrying out the point feature extraction of each first image of a to-be-reconstructed scene, and obtaining a first point feature corresponding to each first image; determining a first line feature corresponding to each first image based on the first point feature corresponding to each first image; generating three-dimensional point cloud information corresponding to the scene to be reconstructed based on the first point feature and the first line feature of each first image, and each first image and the corresponding depth image; the three-dimensional point cloud information comprises first point cloud information and second point cloud information, the first point cloud information corresponds to a first point feature of each first image, and the second point cloud information corresponds to a first line feature of each first image; and based on the first point cloud information and the second point cloud information, performing three-dimensional scene reconstruction on the to-be-reconstructed scene to obtain an initial three-dimensional model. According to the embodiment of the invention, a more real virtual scene can be reconstructed, and the authenticity of the virtual scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and particularly relates to a three-dimensional reconstruction method and a three-dimensional reconstruction system. Background Art

[0002] With the development of computer technology, virtual reality technology and augmented reality technology have been applied in more and more scenarios. By constructing virtual scenes for different objects, an immersive roaming experience can be brought to users. Currently, although the virtual scenes constructed by existing scene construction methods can meet the browsing needs of users, they often have a large difference from the actual scenes, resulting in low authenticity and inability to effectively interact with users. Summary of the Invention

[0003] In view of the above problems of the prior art, the present invention discloses a three-dimensional reconstruction method and a three-dimensional reconstruction system, which can reconstruct a more realistic virtual scene and improve the authenticity of the virtual scene. The technical solutions disclosed by the present invention are as follows:

[0004] According to one aspect of the disclosed embodiments of the present invention, a three-dimensional reconstruction method is provided, including:

[0005] Obtaining at least two first images of the scene to be reconstructed and a depth image corresponding to each first image;

[0006] Performing point feature extraction on each of the first images to obtain a first point feature corresponding to each of the first images;

[0007] Based on the first point feature corresponding to each of the first images, determining a first line feature corresponding to each of the first images;

[0008] Generating three-dimensional point cloud information corresponding to the scene to be reconstructed based on the first point feature and the first line feature of each of the first images, and the each first image and the corresponding depth image; the three-dimensional point cloud information includes first point cloud information and second point cloud information, the first point cloud information corresponds to the first point feature of each of the first images, and the second point cloud information corresponds to the first line feature of each of the first images;

[0009] Performing three-dimensional scene reconstruction on the scene to be reconstructed based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model.

[0010] Optionally, the performing point feature extraction on each of the first images to obtain a first point feature corresponding to each of the first images includes:

[0011] Performing scaling processing on each of the first images to obtain a plurality of second images corresponding to each of the first images; the scale information of any two second images is different;

[0012] Extract point features from each second image to obtain second point features corresponding to each second image; the second point features are used to represent second feature points in the corresponding each second image.

[0013] Based on the second point features corresponding to the multiple second images corresponding to each first image, determine the first point features of each first image.

[0014] Optionally, the determining the first line features corresponding to each first image based on the first point features corresponding to each first image includes:

[0015] Based on the second point features corresponding to each second image and the third point features corresponding to the second point features, determine the second line features corresponding to each second image; the third point features are used to represent third feature points in the corresponding each second image, and the third feature points are located within a preset range of the second feature points.

[0016] Perform a fusion process on the second line features corresponding to the multiple second images to determine the first line features of each first image.

[0017] Optionally, the multiple second images include a first sub-image and a second sub-image, and the scale information corresponding to the second sub-image is smaller than the scale information corresponding to the first sub-image. The performing a fusion process on the second line features corresponding to the multiple second images to determine the first line features of each first image includes:

[0018] Perform a magnification process on the second sub-image and the second line features corresponding to the second sub-image to obtain a third sub-image and third line features corresponding to the third sub-image, and the scale information of the third sub-image is equal to the scale information corresponding to the first sub-image.

[0019] Perform a matching process on the third line features corresponding to the third sub-image and the second line features corresponding to the first sub-image to obtain line feature pairs.

[0020] Based on the angle between the feature line segments corresponding to the third line features and the feature line segments corresponding to the second line features in the line feature pairs, perform a fusion process on the second line features and the third line features in the line feature pairs to obtain the first line features of each first image.

[0021] Optionally, the determining the second line features corresponding to each second image based on the second point features corresponding to each second image and the third point features corresponding to the second point features includes:

[0022] Calculate gradient information based on the second point features corresponding to each second image and the third point features.

[0023] Based on the gradient information, perform line feature extraction on each of the second images to obtain the second line features.

[0024] Optionally, the generating the three-dimensional point cloud information corresponding to the to-be-reconstructed scene based on the first point features and the first line features of each of the first images, and each of the first images and the corresponding depth image includes:

[0025] Determine a first reprojection error corresponding to the first point feature and a second reprojection error corresponding to the first line feature;

[0026] Perform weighted summation processing on the first reprojection error and the second reprojection error to obtain a target reprojection error;

[0027] Based on the target reprojection error, determine the pose transformation information between two adjacent first images;

[0028] Based on the pose transformation information, perform stitching processing on each of the first images and the corresponding depth image to obtain the three-dimensional point cloud information.

[0029] Optionally, the performing three-dimensional scene reconstruction on the to-be-reconstructed scene based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model includes:

[0030] Generate triangular mesh cells based on the first point cloud information and the second point cloud information;

[0031] Based on the triangular mesh cells, perform three-dimensional scene reconstruction on the to-be-reconstructed scene to obtain the initial three-dimensional model.

[0032] Optionally, after the performing three-dimensional scene reconstruction on the to-be-reconstructed scene based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model, the method further includes:

[0033] Obtain a sound signal corresponding to the to-be-reconstructed scene;

[0034] Add the sound signal to the initial three-dimensional model to obtain a target three-dimensional model.

[0035] According to another aspect of the disclosed embodiments of the present invention, there is provided a three-dimensional reconstruction system, including a scene construction module, where the scene construction module is configured to perform three-dimensional reconstruction on a to-be-reconstructed scene based on any one of the above three-dimensional reconstruction methods.

[0036] Optionally, the three-dimensional reconstruction system further includes an object component module, the object component module is connected to the scene construction module, and the object component module is configured to provide a three-dimensional model of object components in the to-be-reconstructed scene.

[0037] According to another aspect of the disclosed embodiments of the present invention, there is provided an electronic device for three-dimensional reconstruction, including a processor and a memory. At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the three-dimensional reconstruction method described in any one of the above.

[0038] According to another aspect of the disclosed embodiments of the present invention, there is provided a computer-readable storage medium. At least one instruction is stored in the computer storage medium, and the at least one instruction is loaded and executed by a processor to implement the three-dimensional reconstruction method described in any one of the above.

[0039] According to another aspect of the disclosed embodiments of the present invention, there is provided a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the three-dimensional reconstruction method described in any one of the above disclosed embodiments of the present invention.

[0040] The three-dimensional reconstruction method provided by the present invention has the following technical effects:

[0041] The present invention obtains at least two first images of a scene to be reconstructed and depth images corresponding to each first image, extracts point features from each first image to obtain first point features corresponding to each first image; based on the first point features corresponding to each first image, determines first line features corresponding to each first image; based on the first point features and first line features of each first image, as well as each first image and its corresponding depth image, generates three-dimensional point cloud information corresponding to the scene to be reconstructed; the three-dimensional point cloud information includes first point cloud information and second point cloud information, the first point cloud information corresponds to the first point features of each first image, and the second point cloud information corresponds to the first line features of each first image; based on the first point cloud information and the second point cloud information, performs three-dimensional scene reconstruction on the scene to be reconstructed to obtain an initial three-dimensional model, so that a more realistic virtual scene can be reconstructed, improving the authenticity of the virtual scene and enabling more realistic and effective interaction between the user and the virtual scene.

[0042] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1Schematic diagram of an application environment of a 3D reconstruction method shown according to an exemplary embodiment;

[0045] Figure 2 Flowchart of a 3D reconstruction method shown according to an exemplary embodiment;

[0046] Figure 3 Flowchart of determining the first line features of each first image shown according to an exemplary embodiment;

[0047] Figure 4 Flowchart of generating 3D point cloud information corresponding to a scene to be reconstructed shown according to an exemplary embodiment;

[0048] Figure 5 Schematic diagram of a 3D reconstruction system shown according to an exemplary embodiment;

[0049] Figure 6 Block diagram of a terminal electronic device for 3D reconstruction shown according to an exemplary embodiment;

[0050] Figure 7 Block diagram of a server electronic device for 3D reconstruction shown according to an exemplary embodiment. Detailed implementation manners

[0051] To enable those of ordinary skill in the art to better understand the technical solutions disclosed in the present invention, the technical solutions in the embodiments disclosed in the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention disclosed herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0053] Please refer to Figure 1 , Figure 1FIG. 0 is a schematic diagram of an application environment of a 3D reconstruction method shown according to an exemplary embodiment. The application environment may at least include a server 100 and a terminal 200.

[0054] In an alternative embodiment, the server 100 may be used to provide back-end services for the terminal 200. The server 100 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0055] In an alternative embodiment, the terminal 200 may be used to provide services such as 3D reconstruction for users. Specifically, the terminal 200 may include, but is not limited to, types of electronic devices such as smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, vehicle-mounted terminals, smart TVs, etc.; it may also be software running on the above-mentioned electronic devices, such as applications, applets, etc. The operating systems running on the electronic devices in the embodiments of the present application may include, but are not limited to, Android systems, IOS systems, Linux, Windows, etc.

[0056] In addition, it should be noted that Figure 1 The shown is only an application environment of a 3D reconstruction method, and the embodiments of the present specification are not limited thereto.

[0057] In the embodiments of the present specification, the above-mentioned server 100 and terminal 200 may be directly or indirectly connected by wired or wireless communication methods, and the present application does not limit this.

[0058] The following introduces a 3D reconstruction method of the present application. Figure 2 FIG. 19 is a flowchart of a 3D reconstruction method shown according to an exemplary embodiment. The present specification provides the method operation steps as described in the embodiment or flowchart, but based on routine or non-creative labor, there may be more or fewer operation steps. The order of steps listed in the embodiment is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual system or server product executes, it may be executed in the order of the method shown in the embodiment or the drawings or executed in parallel (for example, in an environment of parallel processors or multi-threaded processing). Specifically, as Figure 2 shown, the above method may include:

[0059] S201: Obtain at least two first images of the scene to be reconstructed and the depth image corresponding to each first image.

[0060] In a specific embodiment, the scene to be reconstructed can be any scene to be three-dimensionally reconstructed, such as a kitchen scene. The first image can be a two-dimensional planar image (such as an RGB image) and can be obtained by camera shooting. The depth image can be an image containing depth information (such as distance information).

[0061] S203: Extract point features from each first image to obtain first point features corresponding to each first image.

[0062] In a specific embodiment, each first image can be subjected to grid processing, and then point features are extracted from each grid corresponding to the first image to obtain point features corresponding to each grid. The set of point features corresponding to each grid in the first image is determined as the first point features corresponding to the first image. The first point features can be used to characterize the first feature points corresponding to the first image.

[0063] Optionally, the above-mentioned extraction of point features from each first image to obtain first point features corresponding to each first image may include:

[0064] Perform scaling processing on each first image to obtain a plurality of second images corresponding to each first image;

[0065] Extract point features from each second image to obtain second point features corresponding to each second image;

[0066] Determine the first point features of each first image based on the second point features corresponding to the plurality of second images corresponding to each first image.

[0067] In a specific embodiment, the scale information of any two second images is different. The second point features can be used to characterize the second feature points in each corresponding second image. The plurality of second images can include a first sub-image and a second sub-image. The scale information corresponding to the second sub-image can be smaller than the scale information corresponding to the first sub-image. The plurality of second images arranged according to the scale information can be regarded as an image pyramid, and each second image can correspond to a layer in the image pyramid.

[0068] In a specific embodiment, each second image can be subjected to grid processing, and then point features are extracted from each grid corresponding to the second image to obtain point features corresponding to each grid. The set of point features corresponding to each grid in the second image is determined as the second point features corresponding to the second image.

[0069] In practical applications, for second images with different scale information, the overall number of feature points can be set accordingly. When extracting point features for each grid, the number of feature points to be extracted can be determined according to the richness of the image texture corresponding to the grid. The richer the image texture corresponding to each grid, the more feature points can be extracted for the grid. Additionally, in the case where the number of feature points to be extracted is lower than a preset number threshold and no feature points of the grid are extracted, the feature point extraction for the grid is no longer performed, and the number of feature points corresponding to the grid can be allocated to the grid area with rich texture information to ensure uniform and reasonable feature extraction. The above preset number threshold can be set according to actual application requirements, for example, set to 5.

[0070] S205: Determine the first line feature corresponding to each first image based on the first point feature corresponding to each first image.

[0071] In an optional embodiment, the determining the first line feature corresponding to each first image based on the first point feature corresponding to each first image may include:

[0072] Determine the second line feature corresponding to each second image based on the second point feature corresponding to each second image and the third point feature corresponding to the second point feature;

[0073] Perform a fusion process on the second line features corresponding to multiple second images to determine the first line feature corresponding to each first image.

[0074] In a specific embodiment, the third point feature can be used to represent the third feature points in each corresponding second image, and the third feature points can be located within a preset range of the second feature points. The preset range and the number of third feature points can be set according to actual application requirements. For example, the preset range can be a circular area range centered on the second feature point, and the number of third feature points can be set to 12.

[0075] In a specific embodiment, in each second image, the connection line between the second feature point and the third feature point can be determined as the feature line segment corresponding to the second line feature. During the fusion process of the second line features corresponding to multiple second images, the second line features corresponding to adjacent two layers in the image pyramid layer can be fused to obtain the first line feature corresponding to the corresponding first image. Correspondingly, during the fusion of the second line features, the line features are fused based on the two end feature points corresponding to the second line feature, so that the first point feature of each first image can be obtained.

[0076] In an optional embodiment, the determining the second line feature corresponding to each second image based on the second point feature corresponding to each second image and the third point feature corresponding to the second point feature may include:

[0077] Calculate gradient information based on the second point features and the third point features corresponding to each second image;

[0078] Extract line features from each second image based on the gradient information to obtain second line features.

[0079] In practical applications, calculate the gradient value according to the connection line between the second feature point and its corresponding third feature point, extract line features, and construct multi-scale line features.

[0080] In an optional embodiment, as Figure 3 shown, the above-mentioned fusion processing of the second line features corresponding to multiple second images to determine the first line feature of each first image may include:

[0081] S301: Enlarge the second sub-image and the second line feature corresponding to the second sub-image to obtain a third sub-image and a third line feature corresponding to the third sub-image.

[0082] Specifically, the scale information of the third sub-image may be equal to the scale information corresponding to the first sub-image. In the image pyramid, for two adjacent second sub-images, the second sub-image with smaller scale information can be enlarged to have the same scale information as the second sub-image with larger scale information, and at the same time, the second line feature therein is enlarged accordingly to obtain the third sub-image and the third line feature corresponding to the third sub-image.

[0083] S303: Match the third line feature corresponding to the third sub-image with the second line feature corresponding to the first sub-image to obtain a line feature pair.

[0084] In a specific embodiment, the third line feature and the second line feature can be matched according to the angle between the third line feature and the second line feature to obtain a line feature pair.

[0085] S305: Based on the angle between the feature line segment corresponding to the third line feature and the feature line segment corresponding to the second line feature in the line feature pair, perform fusion processing on the second line feature and the third line feature in the line feature pair to obtain the first line feature of each first image.

[0086] In a specific embodiment, when the angles between the two feature line segments are the same, the longest line segment among the line segments connecting the four endpoints of the two feature line segments is determined as the fused feature line segment. When the angle between the two feature line segments is greater than the first preset angle threshold and less than the second preset angle threshold, the longest line segment among the line segments connecting the four endpoints of the two feature line segments is determined as the fused feature line segment, and the intermediate angle of this angle is determined as the orientation of the fused feature line segment. When the angle between the two feature line segments is greater than the second preset angle threshold, no fusion is performed, and the two feature line segments are respectively determined as the corresponding first-line features. Specifically, the first preset angle threshold and the second preset angle threshold can be set according to actual application requirements.

[0087] In practical applications, the feature line segment features of low resolution are gradually fused upward. After the feature line segments of low resolution are enlarged to the size of the upper layer, they are compared with the feature line segments of high resolution. When the angles between the two feature line segments are the same, the two feature line segments are fused into one line segment, and the length is the maximum value of the lengths of the multiple line segments formed by the four endpoints of the two feature line segments; when the angle between the two feature line segments is greater than the first preset angle threshold and less than the second preset angle threshold, the two feature line segments are fused into one line segment, and the length is the maximum value of the lengths of the multiple line segments formed by the four endpoints of the two feature line segments, and the angle of the fused feature line segment is adjusted to the average value of the angles between the two feature line segments; when the angle between the two feature line segments is greater than the second preset angle threshold, the two line segments are retained; finally, the first-line features of the corresponding first image are output.

[0088] S207: Generate the three-dimensional point cloud information corresponding to the scene to be reconstructed based on the first point features and the first-line features of each first image, as well as each first image and the corresponding depth image.

[0089] In a specific embodiment, the three-dimensional point cloud information may include first point cloud information and second point cloud information. The first point cloud information may correspond to the first point features of each first image, and the second point cloud information may correspond to the first-line features of each first image.

[0090] In an alternative embodiment, as Figure 4 shown, the generation of the three-dimensional point cloud information corresponding to the scene to be reconstructed based on the first point features and the first-line features of each first image, as well as each first image and the corresponding depth image may include:

[0091] S401: Determine the first reprojection error corresponding to the first point feature and the second reprojection error corresponding to the first-line feature.

[0092] Specifically, denote the feature points corresponding to the first point feature as {x1,...xi}, the first reprojection error corresponding to the first feature point can be determined by the following formula:

[0093]

[0094] where, e p represents the first reprojection error, x i represents the feature point corresponding to the first feature point, and P i represents the point after projection by the pose P.

[0095] Specifically, the second reprojection error corresponding to the first line feature can be the sum of the reprojection errors of the two endpoints of the feature line segment corresponding to the first line feature:

[0096]

[0097] where, e l represents the second reprojection error, x j and x k respectively represent the two endpoints of the feature line segment, and P i and P k respectively represent the points corresponding to the two endpoints after projection by the pose P.

[0098] S403: Perform weighted summation processing on the first reprojection error and the second reprojection error to obtain the target reprojection error.

[0099] In a specific embodiment, the target reprojection error can be determined according to the following formula:

[0100] e = λ1·e p + λ2·e l

[0101] where, e represents the target reprojection error, and λ1 and λ2 respectively represent the p and e l corresponding weight information.

[0102] Specifically, λ1 and λ2 can be determined by the following formula:

[0103]

[0104] where i, j, and k are respectively the number of feature points corresponding to the first feature point and the number of the two endpoints of the first line feature.

[0105] S405: Determine the pose transformation information between two adjacent first images based on the target reprojection error.

[0106] Specifically, the pose parameters are continuously adjusted through an iterative optimization algorithm to minimize the reprojection error and solve for the optimal pose. The geometric transformation relationship between images, i.e., the pose transformation information, is estimated through feature matching.

[0107] S407: Based on the pose transformation information, each first image and its corresponding depth image are stitched to obtain three-dimensional point cloud information.

[0108] Specifically, the estimated pose matrix (i.e., the pose transformation information) is used to match and geometrically transform (such as perspective transformation, affine transformation) each first image and its corresponding depth image to align them in spatial position and then fuse them to obtain a fused image including three-dimensional point cloud information.

[0109] S209: Based on the first point cloud information and the second point cloud information, three-dimensional scene reconstruction is performed on the scene to be reconstructed to obtain an initial three-dimensional model.

[0110] In practical applications, before performing three-dimensional scene reconstruction, noise points and outliers in the obtained point cloud can be screened and removed first, for example, by using a statistical filter for screening and removal.

[0111] Optionally, the above-mentioned three-dimensional scene reconstruction of the scene to be reconstructed based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model may include:

[0112] Generating triangular mesh cells based on the first point cloud information and the second point cloud information;

[0113] Based on the triangular mesh cells, three-dimensional scene reconstruction is performed on the scene to be reconstructed to obtain an initial three-dimensional model.

[0114] In a specific embodiment, the first point cloud information and the second point cloud information can be used as growth points respectively to generate triangular mesh cells, and the vertices of the triangular mesh cells can be used as growth points again to continue generating triangular mesh cells. The grid reconstruction of the three-dimensional point cloud map is continuously repeated to obtain the initial three-dimensional model. Furthermore, the color information of the vertices of the triangular mesh cells is averaged to obtain the color information of the triangular mesh cells, and a three-dimensional model with color information is obtained.

[0115] In an optional embodiment, after performing three-dimensional scene reconstruction on the scene to be reconstructed based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model, the above method may further include:

[0116] Obtaining the sound signal corresponding to the scene to be reconstructed;

[0117] Adding the sound signal to the initial three-dimensional model to obtain a target three-dimensional model.

[0118] Specifically, the sound signal may include the background noise signal of the scene and the working sound signal of the electrical appliance, etc. In practical applications, the collected sound signal is attached to the initial three-dimensional model map to obtain the corresponding target three-dimensional model map with audio data.

[0119] As can be seen from the technical solutions provided in the embodiments of this specification above, in this specification, at least two first images of the scene to be reconstructed and the depth image corresponding to each first image are obtained, point features are extracted from each first image to obtain the first point features corresponding to each first image; based on the first point features corresponding to each first image, the first line features corresponding to each first image are determined; based on the first point features and first line features of each first image, as well as each first image and the corresponding depth image, three-dimensional point cloud information corresponding to the scene to be reconstructed is generated; the three-dimensional point cloud information includes first point cloud information and second point cloud information, the first point cloud information corresponds to the first point features of each first image, and the second point cloud information corresponds to the first line features of each first image; based on the first point cloud information and the second point cloud information, three-dimensional scene reconstruction is performed on the scene to be reconstructed to obtain an initial three-dimensional model, so that a more realistic virtual scene can be reconstructed, the authenticity of the virtual scene is improved, and the user can have a more realistic and effective interaction with the virtual scene.

[0120] An embodiment of the present invention also provides a three-dimensional reconstruction system, as Figure 5 shown. The system may include a scene construction module, and the scene construction module is used to perform three-dimensional reconstruction on the scene to be reconstructed based on the above three-dimensional reconstruction method.

[0121] Optionally, the three-dimensional reconstruction system may further include an object component module. The object component module is connected to the scene construction module, and the object component module is used to provide three-dimensional models of object components in the scene to be reconstructed.

[0122] Specifically, the object components may include kitchen appliances and kitchen utensils, etc. The three-dimensional models of the object components may correspond to the actual characteristics of the object components, and specifically may include dimensions and appearances, etc.

[0123] Optionally, the three-dimensional reconstruction system may further include an audio acquisition module and a scene acquisition module. Specifically, the audio acquisition module may be used to acquire the sounds such as the opening, closing, and working of kitchen appliances and the background noise in the kitchen environment, and the scene acquisition module may be used to acquire the scene data of the kitchen environment, specifically including kitchen scene images and corresponding depth images.

[0124] In practical applications, the selected kitchen appliance model is imported and placed in the 3D model map in the virtual reality device. Users can change the color information of the 3D model map according to their preferences and can also determine whether the size of the kitchen appliance meets the installation requirements of the kitchen, so as to select a kitchen appliance device that meets the installation requirements. The characteristics of the kitchen appliance can also be presented in the virtual reality device. For example, the changes in the appearance and sound during the opening or closing process of the kitchen appliance. When the kitchen appliance is not actually installed in the kitchen, users can more realistically and immersively experience the appearance style and operating state of the kitchen appliance, enhancing effective interaction.

[0125] Optionally, the 3D reconstruction system may have a scene update function and an object update function to update the scene and the object respectively.

[0126] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0127] Figure 6 is a block diagram of a terminal electronic device for 3D reconstruction shown according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as Figure 6 shown. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a 3D reconstruction method is implemented. The display screen of the electronic device may be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, a touchpad, or a mouse, etc.

[0128] Figure 7 is a block diagram of a server electronic device for 3D reconstruction shown according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as Figure 7As shown. The electronic device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a three-dimensional reconstruction method.

[0129] Those skilled in the art can understand that Figure 6 or Figure 7 the structure shown in is only a block diagram of some structures related to the disclosed solution of the present invention, and does not constitute a limitation on the electronic device to which the disclosed solution of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In an exemplary embodiment, there is also provided an electronic device for three-dimensional reconstruction, including a processor and a memory. At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the three-dimensional reconstruction method in the disclosed embodiment of the present invention.

[0131] In an exemplary embodiment, there is also provided a computer-readable storage medium. At least one instruction is stored in the computer storage medium, and the at least one instruction is loaded and executed by a processor to implement the three-dimensional reconstruction method in the disclosed embodiment of the present invention.

[0132] In an exemplary embodiment, there is also provided a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the three-dimensional reconstruction method in the disclosed embodiment of the present invention.

[0133] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate

[0134] SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0135] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the invention. The present invention is intended to cover any variations, uses, or adaptations of the invention disclosed, which follow the general principles of the invention disclosed and include known common knowledge or conventional technical means in the technical field not disclosed by the present invention. The specification and examples are only regarded as exemplary, and the true scope and spirit of the invention disclosed are pointed out by the following claims.

[0136] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Obtaining at least two first images of the scene to be reconstructed and depth images corresponding to each first image; Performing point feature extraction on each of the first images to obtain first point features corresponding to each of the first images; the first point features are used to characterize first feature points in the corresponding first images Based on the first point features corresponding to each of the first images, determining first line features corresponding to each of the first images; Based on the first point features and first line features of each of the first images, as well as each of the first images and the corresponding depth images, generating three-dimensional point cloud information corresponding to the scene to be reconstructed; the three-dimensional point cloud information includes first point cloud information and second point cloud information, the first point cloud information corresponds to the first point features of each of the first images, and the second point cloud information corresponds to the first line features of each of the first images; Based on the first point cloud information and the second point cloud information, performing three-dimensional scene reconstruction on the scene to be reconstructed to obtain an initial three-dimensional model.

2. The method according to claim 1, wherein The performing point feature extraction on each of the first images to obtain first point features corresponding to each of the first images includes: Performing scaling processing on each of the first images to obtain a plurality of second images corresponding to each of the first images; the scale information of any two second images is different; Performing point feature extraction on each of the second images to obtain second point features corresponding to each of the second images; the second point features are used to characterize second feature points in the corresponding second images; Based on the second point features corresponding to the plurality of second images corresponding to each of the first images, determining the first point features of each of the first images.

3. The method according to claim 2, characterized in that, The based on the first point features corresponding to each of the first images, determining first line features corresponding to each of the first images includes: Based on the second point features corresponding to each of the second images and third point features corresponding to the second point features, determining second line features corresponding to each of the second images; the third point features are used to characterize third feature points in the corresponding second images, and the third feature points are within a preset range of the second feature points; Performing fusion processing on the second line features corresponding to the plurality of second images to determine the first line features of each of the first images.

4. The method according to claim 3, wherein The plurality of second images include a first sub-image and a second sub-image, and the scale information of the second sub-image is smaller than the scale information of the first sub-image. The performing fusion processing on the second line features corresponding to the plurality of second images to determine the first line features of each of the first images includes: Performing magnification processing on the second sub-image and the second line features corresponding to the second sub-image to obtain a third sub-image and third line features corresponding to the third sub-image, and the scale information of the third sub-image is equal to the scale information of the first sub-image; Performing matching processing on the third line features corresponding to the third sub-image and the second line features corresponding to the first sub-image to obtain line feature pairs; Based on the angle between the feature line segments corresponding to the third line feature and the second line feature in the line feature pair, perform a fusion process on the second line feature and the third line feature in the line feature pair to obtain the first line feature of each first image.

5. The method according to claim 3, wherein The determination of the second line feature corresponding to each second image based on the second point feature corresponding to each second image and the third point feature corresponding to the second point feature includes: Calculate gradient information based on the second point feature corresponding to each second image and the third point feature; Based on the gradient information, perform line feature extraction on each second image to obtain the second line feature.

6. The method according to claim 1, wherein The generation of the three-dimensional point cloud information corresponding to the to-be-reconstructed scene based on the first point feature and the first line feature of each first image, and each first image and the corresponding depth image includes: Determine the first reprojection error corresponding to the first point feature and the second reprojection error corresponding to the first line feature; Perform a weighted summation process on the first reprojection error and the second reprojection error to obtain a target reprojection error; Based on the target reprojection error, determine the pose transformation information between two adjacent first images; Based on the pose transformation information, perform a stitching process on each first image and the corresponding depth image to obtain the three-dimensional point cloud information.

7. The method according to claim 1, characterized in that, The three-dimensional scene reconstruction of the to-be-reconstructed scene based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model includes: Generate triangular mesh cells based on the first point cloud information and the second point cloud information; Based on the triangular mesh cells, perform three-dimensional scene reconstruction on the to-be-reconstructed scene to obtain the initial three-dimensional model.

8. The method according to claim 1, characterized in that, After the three-dimensional scene reconstruction of the to-be-reconstructed scene based on the first point cloud information and the second point cloud information to obtain an initial three-dimensional model, the method further includes: Obtain the sound signal corresponding to the to-be-reconstructed scene; Add the sound signal to the initial three-dimensional model to obtain a target three-dimensional model.

9. A three-dimensional reconstruction system, characterized in that, The three-dimensional reconstruction system includes a scene construction module, and the scene construction module is used to perform three-dimensional reconstruction on the to-be-reconstructed scene based on the three-dimensional reconstruction method according to any one of claims 1 to 8.

10. The three-dimensional reconstruction system according to claim 9, wherein The three-dimensional reconstruction system further includes an object component module, the object component module is connected to the scene construction module, and the object component module is used to provide a three-dimensional model of the object components in the to-be-reconstructed scene.

Images