Indoor 3D reconstruction and whole-home design method and system based on big data

Abstract

The present application discloses a method and system for indoor three-dimensional reconstruction and whole-home design based on big data, including: obtaining a whole-house video to be reconstructed, and calculating the video keyframes, camera internal parameters and camera pose, and then constructing a three-dimensional sparse point cloud; encoding the three-dimensional sparse point cloud to obtain high-dimensional features; using a trained target natural language model to decode the high-dimensional features to obtain the house structure information and house layout information; determining the furniture bounding box based on the house structure information, and extracting the first multi-view image of the furniture bounding box; matching products in a vector database based on the first multi-view image to obtain target furniture products; performing three-dimensional reconstruction based on the house structure information and house layout information, and configuring the target furniture products based on the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results. The present application can quickly and accurately generate a whole-house three-dimensional model, improve design efficiency, and can be widely used in the field of computer technology.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for indoor three-dimensional reconstruction and whole-home design based on big data. Background Art

[0002] In the process of whole house renovation or home improvement, how to efficiently and conveniently rebuild the 3D model of the customer's home has always been the focus of business attention. In the digital age, designers often use field-specific computer-aided design software to reconstruct the 3D model of the user's home step by step after on-site surveying.

[0003] Indoor 3D reconstruction generally requires obtaining a 3D point cloud of the scene. One way to obtain point clouds is to use special instruments, such as binocular cameras, laser radars, etc., to scan the spatial environment and synthesize point clouds. The generated point clouds are highly accurate and have few gaps, but the equipment is expensive and not easy to promote. Another way is to extract key frames from videos, calculate camera poses, and reconstruct sparse point clouds (SFM). Although the point cloud data obtained by this method is of poor quality, it is low-cost and easy to promote. However, low-quality point clouds obtained by video cannot generally be used directly. It is necessary to identify relevant scene semantic information point by point through dense reconstruction or pointnet and other models, and then obtain the spatial information of the scene and furniture through a series of fusion, segmentation, correction and other methods. The relevant calculations are quite time-consuming and consume huge computing power. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and system for indoor three-dimensional reconstruction and whole-home design based on big data, aiming to achieve fast and accurate three-dimensional modeling and whole-home design of indoor videos taken by users.

[0005] To achieve the above objectives, one aspect of an embodiment of the present application proposes a method for indoor 3D reconstruction and whole-home design based on big data, the method comprising:

[0006] Obtaining a whole-house video to be reconstructed, and calculating video key frames, camera intrinsic parameters, and camera pose according to the whole-house video to be reconstructed;

[0007] Constructing a three-dimensional sparse point cloud according to the video key frame, the camera intrinsic parameter and the camera pose;

[0008] Encoding the three-dimensional sparse point cloud to obtain high-dimensional features;

[0009] Using a trained target natural language model to decode the high-dimensional features to obtain apartment structure information and apartment layout information;

[0010] Determine a furniture bounding box according to the apartment structure information, and extract a first multi-view image of the furniture bounding box;

[0011] According to the first multi-view images, product matching is performed in a vector database to obtain a target furniture product;

[0012] Three-dimensional reconstruction is performed according to the apartment structure information and the apartment layout information, and the target furniture product is configured according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results.

[0013] In some embodiments, in the step of obtaining the whole-house video to be reconstructed and calculating the video key frame, camera intrinsic parameter and camera pose according to the whole-house video, calculating the video key frame, camera intrinsic parameter and camera pose according to the whole-house video includes:

[0014] Based on the optical flow calculation method, a video key frame is calculated according to the whole house video to be reconstructed;

[0015] The camera internal parameters and camera pose are calculated based on the camera's shooting information and the whole-house video to be reconstructed.

[0016] In some embodiments, encoding the three-dimensional sparse point cloud to obtain high-dimensional features includes:

[0017] Sampling the three-dimensional sparse point cloud according to the point cloud density to obtain first point cloud data;

[0018] A convolution calculation is performed on the position information of the first point cloud data to obtain a high-dimensional feature.

[0019] In some embodiments, in the step of using a trained natural language model to decode the high-dimensional features to obtain structural information and layout information, the step of training the natural language model includes:

[0020] Construct a training dataset of the relationship between point clouds and labels;

[0021] Build an initial natural language model;

[0022] The training data set is used to perform comparative learning training on the initial natural language model to obtain a target natural language model.

[0023] In some embodiments, the step of constructing a training data set for the relationship between point clouds and labels includes:

[0024] Read the location and orientation information of walls, doors, windows, and furniture products from the data files of historical apartment plans to generate apartment data information with consistent scale;

[0025] Generate an activity path according to the apartment data information;

[0026] Constructing a shooting trajectory according to the activity path and shooting a whole-house video for model training;

[0027] Generating text structure layout information of the whole house video;

[0028] The whole-house video and the text structure layout information are integrated into a training data set.

[0029] In some embodiments, in the step of performing product matching in a vector database according to the first multi-view image to obtain a target furniture product, the step of constructing the vector database includes:

[0030] Obtaining a second multi-angle picture of the existing furniture product, and using a target convolutional neural network to perform calculation and category encoding processing on each of the second multi-angle pictures to obtain encoding information;

[0031] The second multi-angle image, the coding information and the product information of the existing furniture product are integrated and saved in a vector database.

[0032] In some embodiments, performing product matching in a vector database according to the first multi-view image to obtain a target furniture product includes:

[0033] Using a target convolutional neural network to encode each of the first multi-view images to obtain a first category code;

[0034] According to the category of the furniture product, determine the target category code that has the closest Euclidean distance to the first category code from the vector database, and obtain the target product information corresponding to the first multi-view image;

[0035] The product corresponding to the target product information that is matched the most times is used as the target furniture product.

[0036] To achieve the above purpose, another aspect of the embodiment of the present application proposes an indoor 3D reconstruction and whole-home design system based on big data, the system comprising:

[0037] The first module is used to obtain the whole-house video to be reconstructed, and calculate the video key frame, camera internal parameters and camera pose according to the whole-house video to be reconstructed;

[0038] The second module is used to construct a three-dimensional sparse point cloud according to the video key frame, the camera internal parameters and the camera pose;

[0039] The third module is used to encode the three-dimensional sparse point cloud to obtain high-dimensional features;

[0040] The fourth module is used to decode the high-dimensional features using the trained target natural language model to obtain apartment structure information and apartment layout information;

[0041] A fifth module is used to determine a furniture bounding box according to the apartment structure information, and extract a first multi-view image of the furniture bounding box;

[0042] A sixth module is used to perform product matching in a vector database according to the first multi-view image to obtain a target furniture product;

[0043] The seventh module is used to perform three-dimensional reconstruction according to the apartment structure information and the apartment layout information, and configure the target furniture product according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0045] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0046] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for indoor three-dimensional reconstruction and whole-home design based on big data. The solution generates spatial information of scenes and furniture by directly decoding the high-dimensional structure of sparse point clouds, and then searches for similar products in the image area corresponding to the product for placement and configuration, which is conducive to quickly and accurately generating a three-dimensional model of the whole house and improving design efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0048] Figure 1 It is a flow chart of a method for indoor three-dimensional reconstruction and whole-home design based on big data provided in an embodiment of the present application;

[0049] Figure 2 is a flow chart for obtaining a training data set provided in an embodiment of the present application;

[0050] Figure 3 is a flow chart of the picture adaptation product provided in the embodiment of the present application;

[0051] Figure 4 This is an example diagram of the generated trajectory perspective offset provided in an embodiment of the present application;

[0052] Figure 5This is an example diagram of text information based on apartment structure provided in an embodiment of the present application;

[0053] Figure 6 is a model structure diagram of a target natural language processing model provided in an embodiment of the present application;

[0054] Figure 7 It is a structural schematic diagram of an indoor three-dimensional reconstruction and whole-home design system based on big data provided in an embodiment of the present application;

[0055] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0057] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] When doing whole-house renovation / home improvement, how to efficiently and conveniently reconstruct the 3D model of the customer's home has always been the focus of business attention. In the digital age, designers often use field-specific computer-aided design software to reconstruct the 3D model of the user's home step by step after on-site surveying, presenting realistic design results to users. Using 3D reconstruction technology, data is collected by automated equipment to automatically reconstruct the 3D model, reducing the modeling workload of designers and improving design efficiency.

[0063] Indoor 3D reconstruction generally requires obtaining a 3D point cloud of the scene. One way to obtain point clouds is to use special instruments, such as binocular cameras, lidar, etc., to scan the spatial environment and synthesize point clouds. The generated point clouds are highly accurate and have few gaps, but the equipment is expensive and difficult to promote. Another way is to extract key frames from videos, calculate camera poses, and reconstruct sparse point clouds (SFM). Although the point cloud data obtained by this method is of poor quality, it is low-cost and easy to promote. Low-quality point clouds obtained from videos generally cannot be used directly. It is necessary to identify relevant scene semantic information point by point through dense reconstruction / pointnet and other models, and then obtain the spatial information of the scene and furniture through a series of fusion, segmentation, correction and other methods. The relevant calculations are quite time-consuming and consume huge computing power, making them difficult to popularize.

[0064] In order to quickly perform 3D modeling and whole-home design on the indoor videos shot by users, the embodiment of the present application proposes to directly use the text decoder to generate the spatial information of the scene and furniture for the high-dimensional encoding of the sparse point cloud, and then search for similar products through the image area corresponding to the product for placement, and quickly generate a 3D model of the whole house to improve design efficiency. Among them, the present application also proposes a training data set preparation scheme for generating a large amount of video / point cloud and scene structure text based on indoor 3D design software and historical design scheme data. The model trained based on the training data set is conducive to the rapid generation of the whole house scene structure for the sparse point cloud generated by SFM, and then more efficiently matches similar products to reconstruct the whole house 3D scene.

[0065] The indoor three-dimensional reconstruction and whole-home design method based on big data provided in the embodiment of the present application relates to the field of computer technology. The indoor three-dimensional reconstruction and whole-home design method based on big data provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements indoor three-dimensional reconstruction and whole-home design methods based on big data, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0068] Figure 1is an optional flow chart of the indoor three-dimensional reconstruction and whole-home design method based on big data provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S700.

[0069] Step S100, obtaining a whole-house video to be reconstructed, and calculating video key frames, camera internal parameters and camera pose according to the whole-house video to be reconstructed.

[0070] Step S200, constructing a three-dimensional sparse point cloud according to the video key frames, the camera intrinsic parameters and the camera pose.

[0071] Step S300, encoding the three-dimensional sparse point cloud to obtain high-dimensional features.

[0072] Step S400: Use the trained target natural language model to decode the high-dimensional features to obtain apartment structure information and apartment layout information.

[0073] Step S500: determining a furniture bounding box according to the apartment structure information, and extracting a first multi-view image of the furniture bounding box.

[0074] Step S600: performing product matching in a vector database based on the first multi-view image to obtain a target furniture product.

[0075] Step S700, performing three-dimensional reconstruction according to the apartment structure information and the apartment layout information, and configuring the target furniture product according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results.

[0076] Steps S100 to S700 shown in the embodiment of the present application directly decode the high-dimensional structure of the sparse point cloud to generate spatial information of the scene and furniture, and then search for similar products in the image area corresponding to the product for placement, which is conducive to quickly and accurately generating a three-dimensional model of the entire house and improving design efficiency.

[0077] In some embodiments, step S100 may include but is not limited to the following steps S110 to S120:

[0078] Step S110, based on the optical flow calculation method, calculate and obtain the video key frame according to the whole house video to be reconstructed;

[0079] Step S120, calculating the camera intrinsic parameters and camera pose according to the camera shooting information and the whole-house video to be reconstructed.

[0080] In some embodiments, step S300 may include but is not limited to the following steps S310 to S320:

[0081] Step S310, sampling the three-dimensional sparse point cloud according to the point cloud density to obtain first point cloud data;

[0082] Step S320: performing convolution calculation on the position information of the first point cloud data to obtain high-dimensional features.

[0083] In some embodiments, the target natural language model of step S400 may be obtained by following steps 1 to 3:

[0084] Step 1: construct a training dataset of the relationship between point cloud and labels;

[0085] Step 2: construct an initial natural language model;

[0086] Step three, using the training data set to perform comparative learning training on the initial natural language model to obtain a target natural language model.

[0087] In some embodiments, step S410 may include but is not limited to the following steps S411 to S415:

[0088] Step S411, reading the location and direction information of walls, doors, windows, and furniture products from the data file of the historical apartment plan, and generating apartment plan data information with the same scale;

[0089] Step S412, generating an activity path according to the apartment type data information;

[0090] Step S413, constructing a shooting trajectory according to the activity path and shooting a whole-house video for model training;

[0091] Step S414, generating text structure layout information of the whole house video;

[0092] Step S415: Integrate the whole-house video and the text structure layout information into a training data set.

[0093] In some embodiments, step S600 may include but is not limited to the following steps S610 to S630:

[0094] Step S610, using a target convolutional neural network to encode each of the first multi-view images to obtain a first category code;

[0095] Step S620, according to the category of the furniture product, determining the target category code with the closest Euclidean distance to the first category code from the vector database, and obtaining the target product information corresponding to the first multi-view image;

[0096] Step S630: The product corresponding to the target product information that is matched the most times is used as the target furniture product.

[0097] In some embodiments, the step of constructing the vector database in step S600 may include the following steps a-b:

[0098] a. Obtain a second multi-angle picture of an existing furniture product, and use a target convolutional neural network to calculate and classify each of the second multi-angle pictures to obtain coding information;

[0099] b. Integrate the second multi-angle image, the coding information and the product information of the existing furniture product and save them into a vector database.

[0100] The following is a detailed description and explanation of the solution of the embodiment of the present invention in combination with a specific scenario application example of three-dimensional reconstruction and whole-home design:

[0101] In an embodiment of the present application, a method for indoor three-dimensional reconstruction and whole-home design based on big data is provided. The method can be applied to perform indoor three-dimensional reconstruction based on a whole-house video to be reconstructed provided by a user.

[0102] In general, first of all, users can upload indoor videos shot by mobile devices (i.e., whole-house videos to be reconstructed), calculate video key frames based on optical flow operations, and infer camera intrinsic parameters and camera poses based on the camera exif (Exchangeable Image File) information combined with the video. Based on this information, the SFM algorithm is used to obtain the sparse point cloud of the scene. Among them, the SFM algorithm is a method for reconstructing a three-dimensional scene based on a sequence of images taken from different perspectives. It infers the three-dimensional structure of the scene and the movement of the camera by analyzing images captured by the camera at different perspectives, thereby creating a three-dimensional model, measuring the camera trajectory, and estimating depth information.

[0103] Afterwards, the point cloud is input into Figure 2 The neural network is trained with a sequence data set generated by the steps shown. This network is encoded using a general point cloud encoder such as pointnet, but unlike the general network that outputs information about each point in the point cloud, the decoding of the network in the embodiment of the present application is directly handled by the decoder of the natural language network, which directly outputs the apartment structure information and furniture layout information in one step, simplifying a series of dense reconstruction, point cloud screening, reorganization and reasoning work in the middle. After quickly correcting the biased results through collision detection, the bounding box of the furniture in the relevant key frame is found by coordinate conversion based on the layout information of the furniture, and image information of the furniture at different angles is obtained. By Figure 3 The image matching process matches the most similar products in the database and directly uses them as the design that is most consistent with the original layout.

[0104] Finally, the apartment structure information is reconstructed in three dimensions, and the corresponding furniture product models are dragged and dropped to the designated locations to complete the interior three-dimensional reconstruction and the entire home design work.

[0105] In some embodiments, the step of obtaining a point cloud-label dataset used by a pre-trained convolutional neural network specifically includes:

[0106] (1) Extract and normalize historical design plans.

[0107] From the data file that saves the historical plan, read the information such as the position and direction of walls, doors, windows, and furniture products, set the origin of the coordinate axis at the lower left corner of the global bounding box, and perform 0-1 normalization calculation on the global coordinate point in the direction of the coordinate axis, so as to generate apartment data information with consistent scale.

[0108] (2) Generate activity paths based on the house structure calculation.

[0109] The doors, windows and walls of the apartment are projected onto a two-dimensional plane, the door part is set to empty, the indoor space is converted into a two-dimensional pixel block, and then the mid-skeleton information of the pixel block is extracted through the OpenCV operator.

[0110] Since the generated skeleton has many unnecessary branches, it cannot be directly used as the activity space for people in the house. It is necessary to iteratively optimize the extracted skeleton points one by one, and try to find the value closest to the optimization target by moving each pixel in four directions: up, down, left, and right.

[0111] Based on this, the relevant build targets are as follows:

[0112] (1) Points whose distance is less than a certain threshold need to be merged into one point;

[0113] (2) Determine the iterative movement direction of the pixel points based on the minimum optimization loss value.

[0114] The loss function is thus configured as:

[0115]

[0116] In the formula, e is the line segment of the skeleton, contour is the contour boundary of the wall and furniture, and the minimum distance ensures that the line segment of the skeleton is as close to the middle of the contour as possible, that is, the center line of the pedestrian walkway. The latter is the dot product of the previous line segment and the next line segment of the skeleton. When they are perpendicular, the dot product is the smallest, so the point moves in the vertical direction.

[0117] (3) Construct the shooting trajectory and render the shooting video.

[0118] In order to simulate the real photo taking situation, in addition to simulating the behavior trajectory of people indoors, it is also necessary to simulate the random shaking of the camera. Here, the behavior trajectory is set to a horizontal plane at a certain height, and a point is randomly sampled at a certain distance nearby, such as Figure 4As shown, the B-spline algorithm is then used to calculate the continuous spatial trajectory to form the camera's operating perspective.

[0119] (4) Generate text structure layout information data.

[0120] For example, the text information based on the apartment structure can be designed as follows:

[0121] The wall is described by the coordinates of the endpoints of the wall. For each wall, when there is a door or window on the wall, the horizontal distance from the midpoint of the door / window to one end of the wall, the height of the bottom of the window, the width of the door and window, and the height of the door and window are described below the wall.

[0122] Each piece of furniture is described by the upper left corner point, lower right corner point, furniture category and furniture orientation (expressed as a rotation angle on the horizontal plane) of the furniture's three-dimensional bounding box.

[0123] The text information based on the house type structure can be referred to as an example Figure 5 , where Wall1 and Wall2 represent walls, and the following x and y represent the coordinates of the wall endpoints. The -window / door below represents the doors and windows on this wall; l represents the horizontal distance from the midpoint of the door / window to the point (x1, y2), z represents the height of the bottom of the window, w is the width of the door and window, and h is the height of the door and window. After describing all the walls, the following Object is the information of the furniture, x, y, z represent the upper left corner and lower right corner of the furniture's three-dimensional bounding box, c is the category of the furniture, and d is the orientation of the furniture, expressed as a rotation angle on the horizontal plane.

[0124] In some embodiments, performing video processing and point cloud encoding on the whole-house video to be reconstructed obtained from the user or the training data set video generated above may specifically include:

[0125] (1) Extract video key frames.

[0126] After obtaining the video shot by the user / the dataset video produced, key frames are selected from the video to construct the point cloud to reduce redundant calculations and speed up the calculation. The core is to evaluate the magnitude of scene changes in the video based on the optical flow vector. When the change is large enough, it is intercepted as a key frame. By calling the sparse optical flow estimation algorithm in the OpenCV algorithm library, the threshold is set to filter out the key frames.

[0127] (2) Use SFM to calculate sparse point cloud.

[0128] Use the key point extraction algorithm in computer graphics (such as OCR operator) to extract the feature points in the key frame, and use the FAST algorithm for matching. Obtain the focal length and aspect ratio of the camera according to the camera's exif file, use the RANSAC algorithm to estimate the ability of parameters from a set of data containing outliers, and take multi-point linear estimation of the camera's overall intrinsic parameter matrix and the extrinsic parameter matrix (i.e., posture) corresponding to each picture. Put all the feature points into the same world coordinate system according to the solved relative coordinates to complete the construction of the sparse point cloud.

[0129] (3) For the constructed sparse point cloud.

[0130] The encoder uses the PointNet network to encode the point cloud. The encoding method of this network is to first sample the point cloud according to the point cloud density, and then perform convolution calculation on the position information of the point cloud, so that each point integrates the information of its nearby points and converts it into a high-dimensional feature representation.

[0131] (4) Decoding of apartment structure information and collision optimization.

[0132] The decoder uses the transformer architecture. Transformer is a deep learning model based on the self-attention mechanism. Here, the high-dimensional feature representation of the point cloud is used as input. During big data training, the corresponding text expressions under different point cloud features are learned, so that the house structure information encoding is output word by word during reasoning. Figure 6 The algorithm model structure encodes the point coordinate information in the point cloud to generate the code of each point, and then generates the code of the entire point cloud through the pooling layer. The code is input into the large language model LLM, and the house structure text information is produced according to the above, from the initial character <sos>Start by letting LLM learn to output the next character one by one until the end character is output <eos>Through a large amount of point cloud-unit type text data training, the LLM model learns to predict the corresponding unit type text based on the point cloud encoding.

[0133] The three-dimensional bounding box corresponding to the obtained furniture information is subjected to collision detection, and the position is adjusted in four directions, front, back, left, and right, to optimize the output result.

[0134] In some embodiments, the steps of matching similar furniture products may specifically include:

[0135] (1) Product image information extraction.

[0136] According to the coordinates of the 3D bounding box in the apartment structure information, the corresponding 2D bounding box in the video keyframe is calculated by the calculated internal and external parameter matrices, so as to extract multi-view images of the product. Multi-view images can be obtained by 3D rendering the model and taking screenshots in the front view, side view, and isometric view directions.

[0137] (2) Product coding model training and coding library construction.

[0138] Get multi-angle pictures of existing products, and use convolutional neural networks (resnet50, etc.) to calculate the code for each picture. Further train the convolutional neural network with the training idea of ​​contrastive learning, extract the codes two by two, and the loss is small when they belong to the same product. The loss is large when they belong to different products, so that the code distribution of each product from different angles is close. And based on the trained model, each product picture and related product information are saved together with the code in the corresponding vector database.

[0139] The loss formula for contrastive learning is as follows.

[0140]

[0141] In the formula, q is the encoding of a picture, k + is the code of other pictures of the product at different angles corresponding to this picture, k i represents the encoding of all randomly selected images, τ represents the weight factor, and is the hyperparameter to adjust the loss variation. Thus, through training, the model can encode images of different angles of the product similarly.

[0142] In some embodiments, the step of matching the furniture product to the bounding box (model matching) specifically includes:

[0143] (1) Take out multiple extracted product images and encode them using the above convolutional neural network. According to the product category, find the category code set from the vector database, calculate the code with the closest Euclidean distance, and index the corresponding product information.

[0144] (2) The product that matches the most images corresponding to the bounding box is selected as the product corresponding to the bounding box.

[0145] (3) Generate a house model based on the house information in the furniture design software, and then scale the product model according to the bounding box to place it at the corresponding position of the house model in the corresponding orientation. The orientation information is given in the text information d of the house structure above, completing the interior three-dimensional reconstruction and the whole home design.

[0146] The embodiments of the present application have at least the following beneficial effects:

[0147] By directly using the text decoder to generate the spatial information of the scene and furniture through the high-dimensional encoding of the sparse point cloud, and then searching for similar products in the image area corresponding to the product for placement, the three-dimensional model of the whole house can be generated quickly and accurately, improving the efficiency of three-dimensional reconstruction and whole-home design.

[0148] See also Figure 7 The embodiment of the present application also provides a system 100 for indoor 3D reconstruction and whole-home design based on big data, which can implement the above-mentioned method for indoor 3D reconstruction and whole-home design based on big data. The system includes:

[0149] The first module 101 is used to obtain a whole-house video to be reconstructed, and calculate video key frames, camera internal parameters and camera pose according to the whole-house video to be reconstructed;

[0150] The second module 102 is used to construct a three-dimensional sparse point cloud according to the video key frame, the camera intrinsic parameter and the camera pose;

[0151] The third module 103 is used to encode the three-dimensional sparse point cloud to obtain high-dimensional features;

[0152] The fourth module 104 is used to decode the high-dimensional features using the trained target natural language model to obtain apartment structure information and apartment layout information;

[0153] The fifth module 105 is used to determine the furniture bounding box according to the apartment structure information, and extract the first multi-view image of the furniture bounding box;

[0154] The sixth module 106 is used to perform product matching in a vector database according to the first multi-view image to obtain a target furniture product;

[0155] The seventh module 107 is used to perform three-dimensional reconstruction according to the apartment structure information and the apartment layout information, and configure the target furniture product according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results.

[0156] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0157] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned indoor three-dimensional reconstruction and whole-home design method based on big data when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0160] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0161] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes the indoor three-dimensional reconstruction and whole-home design method based on big data in the embodiment of this application;

[0162] Input / output interface 203, used to implement information input and output;

[0163] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0164] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0165] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0166] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned indoor three-dimensional reconstruction and whole-home design method based on big data.

[0167] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0170] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0171] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0172] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0179] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.< / eos> < / sos>

Claims

1. A method for indoor 3D reconstruction and whole-home design based on big data, characterized in that: The following steps are involved: Obtaining a whole-house video to be reconstructed, and calculating video key frames, camera intrinsic parameters, and camera pose according to the whole-house video to be reconstructed; Constructing a three-dimensional sparse point cloud according to the video key frame, the camera intrinsic parameter and the camera pose; Encoding the three-dimensional sparse point cloud to obtain high-dimensional features; The high-dimensional features are decoded using a trained target natural language model to obtain apartment structure information and apartment layout information of a text structure; Determine a furniture bounding box according to the apartment structure information, and extract a first multi-view image of the furniture bounding box; According to the first multi-view images, product matching is performed in a vector database to obtain a target furniture product; Performing three-dimensional reconstruction according to the apartment structure information and the apartment layout information, and configuring the target furniture product according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results; Among them, the target natural language model integrates the deep learning model of the transformer architecture and the large language model LLM, the high-dimensional features are input into the deep learning model of the transformer architecture, the housing structure information encoding is obtained by inference through the trained deep learning model of the transformer architecture, and then the housing structure information encoding is input into the trained large language model LLM, and the text of the housing structure information and the text of the housing layout information are predicted by the large language model LLM.

2. The method according to claim 1, characterized in that In the step of obtaining the whole-house video to be reconstructed, and calculating the video key frame, camera internal parameter and camera pose according to the whole-house video, calculating the video key frame, camera internal parameter and camera pose according to the whole-house video includes: Based on the optical flow calculation method, a video key frame is calculated according to the whole house video to be reconstructed; The camera internal parameters and camera pose are calculated based on the camera's shooting information and the whole-house video to be reconstructed.

3. The method according to claim 1, characterized in that The encoding of the three-dimensional sparse point cloud to obtain high-dimensional features includes: Sampling the three-dimensional sparse point cloud according to the point cloud density to obtain first point cloud data; A convolution calculation is performed on the position information of the first point cloud data to obtain a high-dimensional feature.

4. The method according to claim 1, characterized in that: In the step of using a trained natural language model to decode the high-dimensional features to obtain the apartment structure information and apartment layout information of a text structure, the natural language model training step includes: Construct a training dataset of the relationship between point clouds and labels; Build an initial natural language model; The training data set is used to perform comparative learning training on the initial natural language model to obtain a target natural language model.

5. The method according to claim 4, characterized in that The training data set for constructing the relationship between point cloud and label includes: Read the location and orientation information of walls, doors, windows, and furniture products from the data files of historical apartment plans to generate apartment data information with consistent scale; Generate an activity path according to the apartment data information; Constructing a shooting trajectory according to the activity path and shooting a whole-house video for model training; Generating text structure layout information of the whole house video; The whole-house video and the text structure layout information are integrated into a training data set.

6. The method according to claim 1, characterized in that In the step of performing product matching in a vector database according to the first multi-view image to obtain a target furniture product, the step of constructing the vector database includes: Obtaining a second multi-angle picture of the existing furniture product, and using a target convolutional neural network to perform calculation and category encoding processing on each of the second multi-angle pictures to obtain encoding information; The second multi-angle image, the coding information and the product information of the existing furniture product are integrated and saved in a vector database.

7. The method according to claim 1, characterized in that The step of performing product matching in a vector database according to the first multi-view image to obtain a target furniture product includes: Using a target convolutional neural network to encode each of the first multi-view images to obtain a first category code; According to the category of the furniture product, determine the target category code that has the closest Euclidean distance to the first category code from the vector database, and obtain the target product information corresponding to the first multi-view image; The product corresponding to the target product information that is matched the most times is used as the target furniture product.

8. The indoor 3D reconstruction and whole-home design system based on big data is characterized by: include: The first module is used to obtain the whole-house video to be reconstructed, and calculate the video key frame, camera internal parameters and camera pose according to the whole-house video to be reconstructed; The second module is used to construct a three-dimensional sparse point cloud according to the video key frame, the camera internal parameters and the camera pose; The third module is used to encode the three-dimensional sparse point cloud to obtain high-dimensional features; The fourth module is used to decode the high-dimensional features using the trained target natural language model to obtain the apartment structure information and apartment layout information of the text structure; A fifth module is used to determine a furniture bounding box according to the apartment structure information, and extract a first multi-view image of the furniture bounding box; A sixth module is used to perform product matching in a vector database according to the first multi-view image to obtain a target furniture product; The seventh module is used to perform three-dimensional reconstruction according to the apartment structure information and the apartment layout information, and configure the target furniture product according to the furniture bounding box to obtain three-dimensional reconstruction and whole-home design results; Among them, the target natural language model integrates the deep learning model of the transformer architecture and the large language model LLM, the high-dimensional features are input into the deep learning model of the transformer architecture, the housing structure information encoding is obtained by inference through the trained deep learning model of the transformer architecture, and then the housing structure information encoding is input into the trained large language model LLM, and the text of the housing structure information and the text of the housing layout information are predicted by the large language model LLM.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images