A method and device for three-dimensional modeling of indoor scenes

Abstract

The present invention discloses a method and device for three-dimensional modeling of indoor scenes, and relates to the field of computer technology. A specific implementation of the method includes: synchronously collecting single-line laser point cloud data, depth map and color image data; determining the initial pose data of the point cloud according to the single-line laser point cloud data; determining the precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data; using the precise pose data of the point cloud, mapping the depth map and color image data to a unified coordinate system to obtain a three-dimensional panoramic model. This implementation overcomes the problem of most indoor scene information being omitted, and at the same time has relatively loose requirements on sampling scenes, can flexibly deploy sampling equipment, and greatly reduces the amount of calculation during the modeling process. While ensuring the stability of modeling, it also improves the accuracy of modeling.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for three-dimensional modeling of indoor scenes. Background Art

[0002] The 3D modeling technology of indoor scenes is widely used in 3D virtual scene roaming, interior decoration auxiliary design, 3D game development, digital city, smart home, indoor mapping and other aspects due to its intuitive, accurate, convenient and rich information content. It has great practical value. Due to the particularity of the indoor environment, the GPS signal is blocked by buildings and cannot be used, and the inertial navigation system and wheel encoder have cumulative errors. In the field of indoor scene modeling, the increasingly mature computer vision, data fusion, visual navigation and 3D modeling technologies provide theoretical basis and technical support for indoor scene 3D modeling. At present, the more successful application is the use of laser SLAM (simultaneous localization and mapping) real-time positioning and map construction technology (or positioning mapping technology).

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] 1. The SLAM modeling method based on single-line lidar can only obtain a planar model of the indoor scene, and most of the indoor scene information is omitted.

[0005] 2. The visual SLAM method based on camera images is greatly affected by ambient light and environmental structure texture, and there are problems with modeling stability and accuracy.

[0006] 3. It has strict requirements for closed-loop scenarios and has high operational restrictions when collecting data, which makes it impossible to deploy it in practice and requires a large amount of computation.

[0007] 4. SLAM modeling method based on multi-line laser radar, multi-line laser sensors are expensive. Summary of the invention

[0008] In view of this, an embodiment of the present invention provides a method and device for three-dimensional modeling of indoor scenes, which can solve the problems of most indoor scene information being omitted and the stability and accuracy of modeling. At the same time, the requirements for sampling scenes are relatively loose, sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced.

[0009] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for three-dimensional modeling of indoor scenes is provided, comprising: synchronously acquiring single-line laser point cloud data, depth map and color image data; determining initial pose data of the point cloud based on the single-line laser point cloud data; determining precise pose data of the point cloud based on the initial pose data of the point cloud and the depth map and color image data; and mapping the depth map and color image data into a unified coordinate system using the precise pose data of the point cloud to obtain a three-dimensional panoramic model.

[0010] Optionally, determining the initial pose data of the point cloud based on the single-line laser point cloud data includes: the single-line laser point cloud data includes coordinate data of the single-line laser point cloud; utilizing single-line laser positioning and mapping technology to determine the rotational displacement and translational displacement of the single-line laser point cloud based on the coordinate data of the single-line laser point cloud; and using the rotational displacement and translational displacement as the initial pose data of the point cloud.

[0011] Optionally, the precise pose data of the point cloud is determined based on the initial pose data of the point cloud and the depth map and color image data, including: the depth map and color image data include coordinate data of the point cloud on the depth color image; and the precise pose data of the point cloud is determined in the depth color image based on the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image using an iterative closest point algorithm.

[0012] Optionally, using an iterative closest point algorithm, according to the initial pose data of the point cloud and the coordinate data of the point cloud on the deep color image, determining the precise pose data of the point cloud in the deep color image, including: using an iterative closest point algorithm, according to the formula

[0013]

[0014] The rotation displacement Rd that will minimize the error E(i,i+1) i and translation displacement Td i As the accurate pose data of the i-th frame point cloud; in the depth color image, is the coordinate data of the point cloud closest to the k-th point cloud in the i+1-th frame; the coordinate data Dpc of the k-th point cloud in the i+1-th frame (i+1)k The coordinate data Dpc of the k-th point cloud in the i-th frame ik , the rotation displacement R of the point cloud after the i-th frame i , translation displacement T i Obtained after transformation; in the iteration, the rotation displacement R0 in the initial pose data of the point cloud i is the rotation displacement R i The initial value is the translation displacement T0 in the initial pose data of the point cloud.i is the translation displacement T i The initial value of ; n is the number of point clouds in the i-th frame.

[0015] Optionally, the depth map and color image data are mapped into a unified coordinate system using the precise pose data of the point cloud to obtain a three-dimensional panoramic model, including: the depth color image data also includes color data; according to the following transformation formula:

[0016] DpC ik =Rd i -1 *Dpc (i+1)k -Td i ,i=m-1,m-2,…,1;

[0017] The coordinate data of the point cloud in the depth color image is mapped step by step to the coordinate system where the first frame of the depth color image is located, and the coordinate data of the point cloud in the depth color image in the coordinate system is obtained, where m is the total number of frames of the acquired depth color image; a three-dimensional panoramic model is constructed based on the coordinate data and color data.

[0018] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a device for three-dimensional modeling of indoor scenes is provided, including: an acquisition module, used to: synchronously acquire single-line laser point cloud data, depth map and color image data; a first processing module, used to: determine the initial pose data of the point cloud according to the single-line laser point cloud data; a second processing module, used to: determine the precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data; a modeling module, used to: use the precise pose data of the point cloud to map the depth map and color image data into a unified coordinate system to obtain a three-dimensional panoramic model.

[0019] Optionally, the first processing module is also used for: the single-line laser point cloud data includes coordinate data of the single-line laser point cloud; using the single-line laser positioning mapping technology, according to the coordinate data of the single-line laser point cloud, determining the rotational displacement and translational displacement of the single-line laser point cloud; and using the rotational displacement and translational displacement as the initial posture data of the point cloud.

[0020] Optionally, the second processing module is also used for: the depth map and color image data include coordinate data of the point cloud on the depth color image; using an iterative closest point algorithm, based on the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image, determining the precise pose data of the point cloud in the depth color image.

[0021] Optionally, the second processing module is further used to: use an iterative closest point algorithm according to the formula The rotation displacement Rd that will minimize the error E(i,i+1) i and translation displacement Td i As the accurate pose data of the i-th frame point cloud; in the depth color image, is the coordinate data of the point cloud closest to the k-th point cloud in the i+1-th frame; the coordinate data Dpc of the k-th point cloud in the i+1-th frame (i+1)k The coordinate data Dpc of the k-th point cloud in the i-th frame ik , the rotation displacement R of the point cloud after the i-th frame i , translation displacement T i Obtained after transformation; in the iteration, the rotation displacement R0 in the initial pose data of the point cloud i is the rotation displacement R i The initial value is the translation displacement T0 in the initial pose data of the point cloud. i is the translation displacement T i The initial value of ; n is the number of point clouds in the i-th frame.

[0022] Optionally, the modeling module is further configured to: the depth color image data further includes color data; according to the following transformation formula:

[0023] DpC ik =Rd i -1 *Dpc (i+1)k -Td i ,i=m-1,m-2,…,1;

[0024] The coordinate data of the point cloud in the depth color image is mapped step by step to the coordinate system where the first frame of the depth color image is located, and the coordinate data of the point cloud in the depth color image in the coordinate system is obtained, where m is the total number of frames of the acquired depth color image; a three-dimensional panoramic model is constructed based on the coordinate data and color data.

[0025] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for three-dimensional modeling of indoor scenes as provided in an embodiment of the present invention.

[0026] To achieve the above objective, according to one aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for three-dimensional modeling of indoor scenes provided in an embodiment of the present invention is implemented.

[0027] One embodiment of the above invention has the following advantages or beneficial effects: according to the precise posture obtained by synchronously collecting single-line laser point cloud data, depth map and color image data, the depth color image RGBD data is transformed into a unified coordinate system, and the technical means of establishing a three-dimensional model overcomes the problem that most indoor scene information is omitted. At the same time, the requirements for sampling scenes are relatively loose, and sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of modeling, it also improves the accuracy of modeling.

[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0030] Figure 1 is a schematic diagram of the basic process of the method for three-dimensional modeling of indoor scenes according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of basic modules of an apparatus for three-dimensional modeling of indoor scenes according to an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of a preferred process of a method for three-dimensional modeling of an indoor scene according to an embodiment of the present invention;

[0033] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0034] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0036] In the existing indoor modeling technology, multi-line laser sensors are expensive, so single-line laser radar is more commonly used. Although the SLAM modeling method based on single-line laser radar can quickly and stably model most indoor scenes, it can only obtain the plane model of the indoor scene, and most of the indoor scene information is omitted. The visual SLAM method based on camera images is greatly affected by ambient light and environmental structure texture. The depth camera can capture the dense point cloud information and color information of the environment at the same time, but due to its small point cloud perception range (generally within 8 meters), the SLAM method based on the depth camera also has problems with the stability and accuracy of the map, and has strict requirements for closed-loop scenes (that is, the acquisition line must be closed, and the acquisition start point must coincide with the acquisition end point). The high operation restrictions during data collection make it impossible to deploy in practice, and the amount of calculation is large. The high operation restrictions during data collection are mainly caused by the small perception range of the depth camera. Due to the small perception range, the features of the perception range must be obvious enough to solve the pose. Therefore, scenes with consistent features such as corridors should be avoided during collection. The acquisition line should be designed as a closed loop. At the same time, a single closed loop line cannot be too long. If it is too long, the error drift will be large, the modeling will not be successful, and re-acquisition is required. Poor stability also means that the modeling cannot guarantee success in one go.

[0037] Figure 1 FIG. 2 is a schematic diagram of the basic process of the method for three-dimensional modeling of indoor scenes according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a method for three-dimensional modeling of an indoor scene, comprising:

[0038] Step S101. synchronously collect single-line laser point cloud data, depth map and color image data;

[0039] Step S102. Determine the initial pose data of the point cloud according to the single-line laser point cloud data;

[0040] Step S103. Determine the precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data;

[0041] Step S104: Using the precise position and posture data of the point cloud, the depth map and the color image data are mapped into a unified coordinate system to obtain a three-dimensional panoramic model.

[0042] According to the precise pose obtained by collecting single-line laser point cloud data, depth map and color image data at the same frequency, the depth color image RGBD data is transformed into a unified coordinate system to establish a three-dimensional model. This technical means overcomes the problem of missing most indoor scene information. At the same time, the requirements for sampling scenes are relatively loose, sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of modeling, it also improves the accuracy of modeling.

[0043] In an embodiment of the present invention, the initial pose data of the point cloud is determined according to the single-line laser point cloud data, including: the single-line laser point cloud data includes the coordinate data of the single-line laser point cloud; using the single-line laser positioning mapping technology, according to the coordinate data of the single-line laser point cloud, the rotational displacement and the translational displacement of the single-line laser point cloud are determined; and the rotational displacement and the translational displacement are used as the initial pose data of the point cloud.

[0044] The initial pose data is obtained according to the SLAM algorithm based on the single-line laser radar, which greatly reduces the amount of calculation in the modeling process. While ensuring the stability of the modeling, it also improves the accuracy and efficiency of the modeling.

[0045] In an embodiment of the present invention, the precise pose data of the point cloud is determined based on the initial pose data of the point cloud and the depth map and color image data, including: the depth map and color image data include coordinate data of the point cloud on the depth color image; using an iterative closest point algorithm, based on the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image, the precise pose data of the point cloud is determined in the depth color image.

[0046] By utilizing the density of the point cloud in the depth color image, the ICP (Iterative Closest Points) algorithm is used to accurately solve the initial pose to obtain the precise pose, which overcomes the problem of most indoor scene information being missed. At the same time, the requirements for the sampling scene are relatively loose, the sampling equipment can be flexibly deployed, and the accuracy of modeling is also improved.

[0047] In an embodiment of the present invention, an iterative closest point algorithm is used to determine the precise pose data of the point cloud in the deep color image according to the initial pose data of the point cloud and the coordinate data of the point cloud on the deep color image, including: using the iterative closest point algorithm, according to the formula

[0048]

[0049] The rotation displacement Rd that will minimize the error E(i,i+1) i and translation displacement Td i As the accurate pose data of the i-th frame point cloud; in the depth color image, is the coordinate data of the point cloud closest to the k-th point cloud in the i+1-th frame; the coordinate data Dpc of the k-th point cloud in the i+1-th frame (i+1)k The coordinate data Dpc of the k-th point cloud in the i-th frame ik , the rotation displacement R of the point cloud after the i-th frame i , translation displacement T iObtained after transformation; in the iteration, the rotation displacement R0 in the initial pose data of the point cloud i is the rotation displacement R i The initial value is the translation displacement T0 in the initial pose data of the point cloud. i is the translation displacement T i The initial value of ; n is the number of point clouds in the i-th frame. This overcomes the problem of missing most indoor scene information. At the same time, the sampling scene requirements are relatively loose, the sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of the modeling, it also improves the accuracy of the modeling.

[0050] In an embodiment of the present invention, the depth map and color image data are mapped to a unified coordinate system using the precise pose data of the point cloud to obtain a three-dimensional panoramic model, including: the depth color image data also includes color data; according to the following transformation formula:

[0051] DpC ik =Rd i -1 *Dpc (i+1)k -Td i ,i=m-1,m-2,…,1;

[0052] The coordinate data of the point cloud in the depth color image is mapped step by step to the coordinate system where the first frame of the depth color image is located, and the coordinate data of the point cloud in the depth color image in the coordinate system is obtained, where m is the total number of frames of the acquired depth color image; a three-dimensional panoramic model is constructed based on the coordinate data and color data. According to the precise posture obtained by acquiring single-line laser point cloud data, depth map and color image data at the same frequency, the depth map and color image data (i.e., RGBD data) are gradually transformed into a unified coordinate system to establish a technical means of three-dimensional modeling, which overcomes the problem of most indoor scene information being omitted. At the same time, the requirements for sampling scenes are relatively loose, and sampling equipment can be flexibly deployed, which greatly reduces the amount of calculation in the modeling process. While ensuring the stability of modeling, it also improves the accuracy of modeling.

[0053] Figure 2 Schematic diagram of basic modules of the device for three-dimensional modeling of indoor scenes according to an embodiment of the present invention. Figure 2As shown, an embodiment of the present invention provides a device 200 for three-dimensional modeling of indoor scenes, including: an acquisition module 201, used to: synchronously acquire single-line laser point cloud data and depth map and color image data; a first processing module 202, used to: determine the initial pose data of the point cloud according to the single-line laser point cloud data; a second processing module 203, used to: determine the precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data; a modeling module 204, used to: use the precise pose data of the point cloud to map the depth map and color image data into a unified coordinate system to obtain a three-dimensional panoramic model.

[0054] According to the precise pose obtained by collecting single-line laser point cloud data, depth map and color image data at the same frequency, the depth color image RGBD data is transformed into a unified coordinate system to establish a three-dimensional model. This technical means overcomes the problem of missing most indoor scene information. At the same time, the requirements for sampling scenes are relatively loose, sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of modeling, it also improves the accuracy of modeling.

[0055] In an embodiment of the present invention, the first processing module 202 is also used for: the single-line laser point cloud data includes coordinate data of the single-line laser point cloud; using the single-line laser positioning mapping technology, according to the coordinate data of the single-line laser point cloud, determining the rotational displacement and translational displacement of the single-line laser point cloud; and using the rotational displacement and translational displacement as the initial posture data of the point cloud.

[0056] The initial pose data is obtained according to the SLAM algorithm based on the single-line laser radar, which greatly reduces the amount of calculation in the modeling process. While ensuring the stability of the modeling, it also improves the accuracy and efficiency of the modeling.

[0057] In an embodiment of the present invention, the second processing module 203 is also used for: the depth map and color image data include coordinate data of the point cloud on the depth color image; using an iterative closest point algorithm, according to the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image, determining the precise pose data of the point cloud in the depth color image.

[0058] By utilizing the density of the point cloud in the depth color image and using ICP to accurately solve the initial pose, the precise pose is obtained, which overcomes the problem of most indoor scene information being missed. At the same time, the requirements for the sampling scene are relatively loose, the sampling equipment can be flexibly deployed, and the accuracy of modeling is improved.

[0059] In the embodiment of the present invention, the second processing module 203 is further used to: use an iterative closest point algorithm according to the formula The rotation displacement Rd that will minimize the error E(i,i+1)i and translation displacement Td i As the accurate pose data of the i-th frame point cloud; in the depth color image, is the coordinate data of the point cloud closest to the k-th point cloud in the i+1-th frame; the coordinate data Dpc of the k-th point cloud in the i+1-th frame (i+1)k The coordinate data Dpc of the k-th point cloud in the i-th frame ik , the rotation displacement R of the point cloud after the i-th frame i , translation displacement T i Obtained after transformation; in the iteration, the rotation displacement R0 in the initial pose data of the point cloud i is the rotation displacement R i The initial value is the translation displacement T0 in the initial pose data of the point cloud. i is the translation displacement T i The initial value of ; n is the number of point clouds in the i-th frame. This overcomes the problem of missing most indoor scene information. At the same time, the sampling scene requirements are relatively loose, the sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of the modeling, it also improves the accuracy of the modeling.

[0060] In the embodiment of the present invention, the modeling module 204 is further used for: the depth color image data also includes color data; according to the following transformation formula:

[0061] DpC ik =Rd i -1 *Dpc (i+1)k -Td i ,i=m-1,m-2,…,1;

[0062] The coordinate data of the point cloud in the depth color image is mapped step by step to the coordinate system where the first frame of the depth color image is located, and the coordinate data of the point cloud in the depth color image in the coordinate system is obtained, where m is the total number of frames of the collected depth color image; a three-dimensional panoramic model is constructed based on the coordinate data and color data. According to the precise posture obtained by collecting single-line laser point cloud data, depth map and color image data at the same frequency, the RGBD data of the depth color image is gradually transformed into a unified coordinate system to establish a technical means of three-dimensional modeling, which overcomes the problem of most indoor scene information being omitted. At the same time, the requirements for sampling scenes are relatively loose, and sampling equipment can be flexibly deployed. The amount of calculation in the modeling process is greatly reduced, which ensures the stability of modeling while also improving the accuracy of modeling.

[0063] Figure 3 Schematic diagram of the preferred process of the method for indoor scene 3D modeling according to an embodiment of the present invention. Figure 3The three-dimensional modeling method of the embodiment of the present invention is specifically described. First, a single-line laser radar and a depth camera are installed on the collection vehicle, and the single-line laser point cloud PC is collected at the same frequency under ROS (Robot System). i The single-line laser radar scanner has a rotating device on it. The point cloud obtained by scanning a circle is called a point cloud frame. Using the SLAM method of the single-line laser radar, the position and posture of the point cloud at each moment are obtained. For example, the i-th frame of the single-line laser point cloud Pc i To point cloud PC i+1 The pose transformation is recorded as R0 i and T0 i , R0 i Indicates the rotation displacement of the point cloud between the i-th frame and the i+1-th frame, T0 i It represents the translation displacement of the point cloud between the i-th frame and the i+1-th frame. Since the acquisition vehicle is a rigid body, this displacement is also the displacement of the point cloud in the depth color image.

[0064] R i and T i As the initial pose of the point cloud in the depth color image, in order to utilize the density of the depth point cloud, the ICP (Iterative Closest Points) algorithm based on the initial pose is used to further refine the pose. The specific equation is as follows:

[0065] Solve this formula for R i and T i In order to minimize the value of E(i,i+1); this solution process is iterative, and the initial value is crucial, otherwise it is easy to fall into the local optimal solution; so combined with the initial pose obtained by laser SLAM; iterative solution is performed with the goal of minimizing E(i,i+1), or when the change in the error value E(i,i+1) is less than the set threshold, the iteration is terminated to obtain the precise pose. Use this precise pose to map the depth point cloud images at different times to a unified coordinate system, and combine the RGB color information of the depth map sensor to obtain the final three-dimensional panoramic model. Specifically: use the rotation and translation between each frame to transform it to the coordinate system of the starting position. For example, the point cloud corresponding to the i+1th frame can be transformed into the coordinate system of the initial acquisition moment (or the first frame) through the following continuous transformation: Dpc ik =Rd i -1 *Dpc (i+1)k -Td i,i=m-1,m-2,…,1. In this way, all point clouds are transformed into a unified coordinate system, and a three-dimensional point cloud model is established. At the same time, since there is a one-to-one correspondence between the depth point cloud and the depth image in the depth sensor device, the point cloud also has color information.

[0066] Figure 4 An exemplary system architecture 400 is shown to which the method for 3D modeling of an indoor scene or the apparatus for 3D modeling of an indoor scene according to an embodiment of the present invention may be applied.

[0067] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0070] Server 405 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 401, 402, and 403. The backend management server may analyze and process received data such as product information query requests, and feed back the processing results to the terminal device.

[0071] It should be noted that the method for three-dimensional modeling of indoor scenes provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the device for three-dimensional modeling of indoor scenes is generally arranged in the server 405 .

[0072] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0073] According to an embodiment of the present invention, the embodiment of the present invention further provides an electronic device and a computer-readable medium.

[0074] The electronic device of the embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of three-dimensional modeling of indoor scenes proposed in the embodiment of the present invention.

[0075] The computer-readable medium of the embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method for three-dimensional modeling of indoor scenes proposed in the embodiment of the present invention is implemented.

[0076] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0077] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0078] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0079] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0080] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor comprising: an acquisition module, a first processing module, a second processing module and a modeling module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the acquisition module may also be described as a "module for collecting data".

[0083] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: synchronously collecting single-line laser point cloud data and depth map and color image data; determining the initial pose data of the point cloud based on the single-line laser point cloud data; determining the precise pose data of the point cloud based on the initial pose data of the point cloud and the depth map and color image data; using the precise pose data of the point cloud, mapping the depth map and color image data to a unified coordinate system to obtain a three-dimensional panoramic model.

[0084] It can be seen from the method of the embodiment of the present invention that the technical means of building a three-dimensional model by transforming the depth color image RGBD data into a unified coordinate system based on the precise posture obtained by collecting single-line laser point cloud data, depth map and color image data at the same frequency overcomes the problem of most indoor scene information being omitted. At the same time, the requirements for sampling scenes are relatively loose, sampling equipment can be flexibly deployed, and the amount of calculation in the modeling process is greatly reduced. While ensuring the stability of modeling, it also improves the accuracy of modeling.

[0085] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for three-dimensional modeling of indoor scenes, characterized in that: include: Synchronously collect single-line laser point cloud data, depth map and color image data; Determining initial position and posture data of the point cloud according to the single-line laser point cloud data; Determining precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data; The depth map and color image data are mapped into a unified coordinate system using the precise position and posture data of the point cloud to obtain a three-dimensional panoramic model.

2. The method according to claim 1, characterized in that: Determining initial position data of the point cloud according to the single-line laser point cloud data includes: The single-line laser point cloud data includes coordinate data of the single-line laser point cloud; Using single-line laser positioning mapping technology, according to the coordinate data of the single-line laser point cloud, the rotation displacement and translation displacement of the single-line laser point cloud are determined; The rotational displacement and the translational displacement are used as initial pose data of the point cloud.

3. The method according to claim 2, characterized in that Determining accurate pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data, including: The depth map and color image data include coordinate data of a point cloud on a depth color image; The iterative closest point algorithm is used to determine the precise pose data of the point cloud in the depth color image according to the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image.

4. The method according to claim 3, characterized in that: Determining accurate pose data of the point cloud in the deep color image using an iterative closest point algorithm based on the initial pose data of the point cloud and the coordinate data of the point cloud on the deep color image includes: Using the iterative closest point algorithm, according to the formula ， Will make the error Minimum rotation displacement and translation displacement As the Accurate pose data of frame point cloud; Among them, in the deep color image, It is Frame and The coordinate data of the point cloud closest to the first point cloud; The first Coordinate data of point clouds By The first Coordinate data of point clouds , after the The rotation displacement of the frame point cloud , translation displacement Obtained after transformation; the rotation displacement in the initial pose data of the point cloud in the iteration is the rotation displacement The initial value is the translation displacement in the initial pose data of the point cloud is the translation displacement The initial value of For the The number of point clouds in the frame.

5. The method according to claim 4, characterized in that Using the accurate pose data of the point cloud, the depth map and the color image data are mapped into a unified coordinate system to obtain a three-dimensional panoramic model, including: The deep color image data also includes color data; According to the following transformation formula: ； Mapping the coordinate data of the point cloud in the depth color image step by step to the coordinate system where the first frame of the depth color image is located, to obtain the coordinate data of the point cloud in the depth color image in the coordinate system, is the total number of frames of the acquired depth color image; A three-dimensional panoramic model is constructed based on the coordinate data and color data.

6. A device for three-dimensional modeling of indoor scenes, characterized in that: include: Acquisition module, used to: synchronously acquire single-line laser point cloud data, depth map and color image data; A first processing module is used to: determine initial position and posture data of the point cloud according to the single-line laser point cloud data; A second processing module is used to determine the precise pose data of the point cloud according to the initial pose data of the point cloud and the depth map and color image data; The modeling module is used to: utilize the accurate position and posture data of the point cloud to map the depth map and color image data into a unified coordinate system to obtain a three-dimensional panoramic model.

7. The device according to claim 6, characterized in that The first processing module is also used for: The single-line laser point cloud data includes coordinate data of the single-line laser point cloud; Using single-line laser positioning mapping technology, according to the coordinate data of the single-line laser point cloud, the rotation displacement and translation displacement of the single-line laser point cloud are determined; The rotational displacement and the translational displacement are used as initial pose data of the point cloud.

8. The device according to claim 7, characterized in that The second processing module is also used for: The depth map and color image data include coordinate data of a point cloud on a depth color image; The iterative closest point algorithm is used to determine the precise pose data of the point cloud in the depth color image according to the initial pose data of the point cloud and the coordinate data of the point cloud on the depth color image.

9. The device according to claim 8, characterized in that The second processing module is also used for: Using the iterative closest point algorithm, according to the formula ， Will make the error Minimum rotation displacement and translation displacement As the Accurate pose data of frame point cloud; Among them, in the deep color image, It is Frame and The coordinate data of the point cloud closest to the first point cloud; The first Coordinate data of point clouds By The first Coordinate data of point clouds , after the The rotation displacement of the frame point cloud , translation displacement Obtained after transformation; the rotation displacement in the initial pose data of the point cloud in the iteration is the rotation displacement The initial value is the translation displacement in the initial pose data of the point cloud is the translation displacement The initial value of For the The number of point clouds in the frame.

10. The device according to claim 9, characterized in that The modeling module is also used for: The deep color image data also includes color data; According to the following transformation formula: ； Mapping the coordinate data of the point cloud in the depth color image step by step to the coordinate system where the first frame of the depth color image is located, to obtain the coordinate data of the point cloud in the depth color image in the coordinate system, is the total number of frames of the acquired depth color image; A three-dimensional panoramic model is constructed based on the coordinate data and color data.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images