Three-dimensional model reconstruction method and related device

Abstract

The invention discloses a three-dimensional model reconstruction method and a related device, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an original picture set corresponding to a to-be-executed reconstruction task, carrying out the initialization processing of the original picture set, obtaining camera pose parameters corresponding to each original picture in the original picture set, and obtaining an initial three-dimensional model, and according to the original picture set and the camera pose parameters corresponding to the original pictures in the original picture set, reconstructing the initial three-dimensional model by using a plurality of processes to obtain a reconstructed three-dimensional model. According to the method, the initial three-dimensional model is reconstructed based on a plurality of processes, the number of steps of iterative optimization can be reduced, and the reconstruction speed is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional model reconstruction method and related devices. Background Art

[0002] 3D model reconstruction technology is one of the key technologies in the field of computer vision. It aims to restore the structure of a 3D scene from a 2D image. This technology has a wide range of applications in game development, film production, virtual reality, augmented reality, cultural relics protection, medical imaging and many other fields.

[0003] Although 3D model reconstruction technology has made certain progress, traditional 3D model reconstruction technology still has the problems of large computational complexity and high requirements on computing resources, resulting in slow reconstruction speed.

[0004] Therefore, how to provide a 3D model reconstruction technology to reduce the computational complexity of 3D model reconstruction, lower the requirements for computing resources, thereby increasing the reconstruction speed and enabling efficient 3D model reconstruction on ordinary computing devices has become an urgent problem to be solved by technicians in this field. Summary of the invention

[0005] In view of the above problems, the present application provides a three-dimensional model reconstruction method and related devices to achieve the purpose of improving the reconstruction speed. The specific scheme is as follows: The first aspect of the present application provides a three-dimensional model reconstruction method, comprising: Obtain the original image set corresponding to the reconstruction task to be performed; Initializing the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model; According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, the initial three-dimensional model is reconstructed using multiple processes to obtain a reconstructed three-dimensional model.

[0006] In a possible implementation, reconstructing the initial three-dimensional model using multiple processes according to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain the reconstructed three-dimensional model includes: According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple rounds of iterative optimization are performed on the parameters in the initial three-dimensional model using multiple processes until the multiple rounds of iterative optimization are completed to obtain a reconstructed three-dimensional model; wherein each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes multi-step iterative optimization.

[0007] In a possible implementation, each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes: Determine the training data corresponding to this round of iterative optimization; Determine a training data subset corresponding to the current round of iterative optimization of each of the processes, wherein the training data subset includes multiple groups of training data, and each group of training data includes an original image and a camera pose parameter corresponding to the original image; Each of the processes uses the corresponding training data subset to perform multi-step iterative optimization on the parameters in the initial three-dimensional model, wherein in each iterative optimization step, each process processes a set of training data in its own training data subset, the processing of each of the processes is executed in parallel, and in the iterative optimization steps with a preset number of steps interval, the parameters of the three-dimensional models on each of the processes are synchronized and density controlled.

[0008] In a possible implementation, determining the training data corresponding to the current round of iterative optimization includes: Determine the random seed corresponding to this round of iterative optimization; Using the random seed to shuffle the order of the original pictures in the original picture set, to obtain a shuffled original picture set; The shuffled original picture set and the camera pose parameters corresponding to each original picture are used as the training data corresponding to this round of iterative optimization.

[0009] In a possible implementation, determining the training data subset corresponding to the current round of iterative optimization of each process includes: Dividing the shuffled original picture set into a plurality of original picture subsets, each original picture subset contains a plurality of original pictures, each original picture subset contains the same number of original pictures, and two adjacent original picture subsets contain some identical original pictures; For each of the original picture subsets, the original picture subset and the camera pose parameters corresponding to each original picture in the original picture subset are provided to the process corresponding to the original picture subset as the training data subset corresponding to the process.

[0010] In a possible implementation, each iterative optimization step is implemented as follows: Each of the processes uses the three-dimensional model on the process to render the camera pose parameters in the training data to obtain a rendered image, calculates a loss based on the difference between the rendered image and the original image in the training data, and adjusts the gradient based on the loss calculation parameter; Synchronizing the parameter adjustment gradients corresponding to the processes to obtain synchronized gradients; Each of the processes uses the synchronized gradients to adjust parameters of the three-dimensional model in the process.

[0011] A second aspect of the present application provides a three-dimensional model reconstruction device, comprising: An acquisition unit, used to acquire a set of original images corresponding to a reconstruction task to be performed; An initialization processing unit, used to perform initialization processing on the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model; The reconstruction unit is used to reconstruct the initial three-dimensional model using multiple processes according to the original picture set and the camera posture parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model.

[0012] In a possible implementation, the reconstruction unit is specifically configured to: According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple rounds of iterative optimization are performed on the parameters in the initial three-dimensional model using multiple processes until the multiple rounds of iterative optimization are completed to obtain a reconstructed three-dimensional model; wherein each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes multi-step iterative optimization.

[0013] In a possible implementation, when the reconstruction unit is used to perform each round of iterative optimization on the parameters in the initial three-dimensional model using multiple processes, it is specifically used to: Determine the training data corresponding to this round of iterative optimization; Determine a training data subset corresponding to the current round of iterative optimization of each of the processes, wherein the training data subset includes multiple groups of training data, and each group of training data includes an original image and a camera pose parameter corresponding to the original image; Each of the processes uses the corresponding training data subset to perform multi-step iterative optimization on the parameters in the initial three-dimensional model, wherein in each iterative optimization step, each process processes a set of training data in its own training data subset, the processing of each of the processes is executed in parallel, and in the iterative optimization steps with a preset number of steps interval, the parameters of the three-dimensional models on each of the processes are synchronized and density controlled.

[0014] In a possible implementation, when the reconstruction unit is used to determine the training data corresponding to the current round of iterative optimization, it is specifically used to: Determine the random seed corresponding to this round of iterative optimization; Using the random seed to shuffle the order of the original pictures in the original picture set, to obtain a shuffled original picture set; The shuffled original picture set and the camera pose parameters corresponding to each original picture are used as the training data corresponding to this round of iterative optimization.

[0015] In a possible implementation, when the reconstruction unit is used to determine the training data subset corresponding to the current round of iterative optimization of each process, it is specifically used to: Dividing the shuffled original picture set into a plurality of original picture subsets, each original picture subset contains a plurality of original pictures, each original picture subset contains the same number of original pictures, and two adjacent original picture subsets contain some identical original pictures; For each of the original picture subsets, the original picture subset and the camera pose parameters corresponding to each original picture in the original picture subset are provided to the process corresponding to the original picture subset as the training data subset corresponding to the process.

[0016] In a possible implementation, when the reconstruction unit is used to implement each iterative optimization step, it is specifically used to: Each of the processes uses the three-dimensional model on the process to render the camera pose parameters in the training data to obtain a rendered image, calculates a loss based on the difference between the rendered image and the original image in the training data, and adjusts the gradient based on the loss calculation parameter; Synchronizing the parameter adjustment gradients corresponding to the processes to obtain synchronized gradients; Each of the processes uses the synchronized gradients to adjust parameters of the three-dimensional model in the process.

[0017] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the three-dimensional model reconstruction method of the first aspect or any implementation of the first aspect.

[0018] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the three-dimensional model reconstruction method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0019] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the three-dimensional model reconstruction method of the above-mentioned first aspect or any implementation method of the first aspect.

[0020] By means of the above technical scheme, the three-dimensional model reconstruction method and related devices provided by the present application, after obtaining the original picture set corresponding to the reconstruction task to be executed, first initialize the original picture set to obtain the camera pose parameters corresponding to each original picture in the original picture set, and the initial three-dimensional model, and then reconstruct the initial three-dimensional model using multiple processes based on the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model. Since the initial three-dimensional model is reconstructed using multiple processes, the number of iterative optimization steps can be reduced, thereby improving the reconstruction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A schematic diagram of the architecture of a three-dimensional model reconstruction system provided in this application; Figure 2 A schematic diagram of the hardware structure of a terminal provided in this application; Figure 3 A schematic diagram of the hardware structure of a server provided for this application; Figure 4 A schematic diagram of a process of a three-dimensional model reconstruction method provided in this application; Figure 5 A schematic diagram of the structure of a three-dimensional model reconstruction device provided in this application; Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0024] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0025] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0026] Considering that the current 3D model reconstruction system relies on professional hardware equipment, such as laser scanners, structured light projectors, etc., these devices are not only expensive to purchase but also complex to maintain, which limits the popularization and application of 3D model reconstruction technology.

[0027] In view of the above problems, a three-dimensional model reconstruction system is provided in this application. Figure 1 , Figure 1 A schematic diagram of the architecture of a three-dimensional model reconstruction system provided in this application.

[0028] like Figure 1 As shown, the system may include a device for carrying a target object (i.e. Figure 1 The invention comprises a turntable 11 of the object shown in the figure; a plurality of light sources 12 arranged around the turntable; a plurality of brackets 13 arranged around the turntable for carrying cameras 14, the brackets being movable and height-adjustable; a plurality of cameras mounted on the brackets, the cameras being able to move around the turntable as the brackets move, and being able to be adjusted above or below the turntable as the height of the brackets is adjusted; a picture acquisition control device 15, the picture acquisition control device being connected to the plurality of cameras and being used to control picture acquisition of the plurality of cameras; a storage device 16, the storage device being connected to the picture acquisition control device and being used to store pictures acquired by the plurality of cameras. A three-dimensional model reconstruction device 17, the three-dimensional model reconstruction device being connected to the picture acquisition control device and the storage device, being used to receive a reconstruction task including acquisition task information sent by the picture acquisition control device, and being used to obtain pictures corresponding to the reconstruction task from the storage device.

[0029] It should be noted that in this application, the camera can be a common SLR camera on the market; the number of cameras can be determined based on the scene requirements of the 3D model reconstruction, and this application does not impose any restrictions. For example, the number of cameras can be greater than or equal to 3. It should be noted that multiple cameras can be evenly distributed from top to bottom, so that when the turntable rotates, multiple cameras can clearly capture the target object carried on the turntable from all angles except the bottom (for example, the top, all sides, etc.).

[0030] The three-dimensional model reconstruction system provided in this application utilizes a common SLR camera on the market as an acquisition device, avoiding the need for expensive customized equipment and significantly reducing system construction and operating costs.

[0031] The picture acquisition control device can be a terminal. The terminal in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not impose any restrictions on this.

[0032] A picture acquisition control application can be installed on the picture acquisition control device. The above-mentioned picture acquisition control application can provide a picture acquisition control interface. A "start acquisition" control can be set on the picture acquisition control interface. After the user clicks the "start acquisition" control, the picture acquisition control interface can display a collection task information input sub-interface. After the user enters the collection task information (such as collection task name, collection task identifier, etc.) in the collection task information input sub-interface, the picture acquisition control interface can display a first information prompt pop-up window. The first information prompt pop-up window can display such as "Please place the object on the turntable" to prompt the user to place the target object on the turntable.

[0033] After the user places the target object on the turntable, the picture acquisition control interface can display a second information prompt pop-up window, and the second information prompt pop-up window can display a message such as "Please turn on the turntable". Considering that in order to ensure the quality of the collected pictures, the turntable rotates one circle and the number of pictures taken by each camera should be within a certain number (such as 80 to 120 pictures). Therefore, the turntable speed needs to be controlled within a preset speed range. After the user turns on the turntable, the picture acquisition control application can automatically identify the speed of the turntable. If the speed of the turntable is too fast or too slow relative to the preset speed range, the picture acquisition control interface can display a third information prompt pop-up window, and the third information prompt pop-up window can display a message such as "The turntable speed is too fast / too slow, please slow down / speed up". The user can manually adjust the speed of the turntable to within the preset speed range.

[0034] As another possible implementation method, the image acquisition control device can also be connected to a turntable, and a turntable control program can be added to the image acquisition control application as a plug-in. After the image acquisition control application identifies that the speed of the turntable is too fast or too slow relative to a preset speed range, the turntable control program can automatically adjust the speed of the turntable to within the preset speed range.

[0035] After the speed of the turntable is adjusted to within the preset speed range, the picture acquisition control application can control multiple cameras to complete autofocus. After multiple cameras complete autofocus, they control multiple cameras to start taking pictures. During the process of multiple cameras taking pictures, the picture acquisition control interface can display a fourth information prompt pop-up window. The fourth information prompt pop-up window can display information such as "Acquisition in progress, xx pictures have been collected, and acquisition is expected to be completed in xx time". The pictures taken by multiple cameras will also be transmitted to the picture acquisition control application. The picture acquisition control application can upload the pictures taken by multiple cameras to a storage device.

[0036] In the present application, the storage device may be a server, specifically, a cloud storage server, and the cloud storage server may be a public cloud storage server or a self-built cloud storage server.

[0037] After the acquisition is completed, the image acquisition control application can send the reconstruction task containing the acquisition task information to the 3D model reconstruction device, and then display the "Start Acquisition" control to prompt the user that the acquisition task has been completed and a new acquisition task can be executed. The execution method of the new acquisition task can refer to the above description and will not be repeated here.

[0038] In a possible implementation, in this application, the picture acquisition control application can provide functions such as viewing historical acquisition tasks and re-acquisition, and this application does not impose any limitations on this.

[0039] In the present application, the three-dimensional model reconstruction device can be a server. Considering that the calculation of the reconstruction task requires a lot of resources, in order to ensure the reconstruction efficiency, in the present application, the three-dimensional model reconstruction device can be a multi-GPU (graphics processing unit) server. The multi-GPU server includes multiple GPU servers, each GPU server has multiple GPU graphics cards, and each GPU graphics card corresponds to a process.

[0040] In the present application, on the one hand, each GPU server can execute a reconstruction task, and the three-dimensional model reconstruction device can execute multiple reconstruction tasks in parallel, thereby improving the execution efficiency of the reconstruction task. On the other hand, each GPU server has multiple processes. When each GPU server executes a reconstruction task, the reconstruction task can be processed in parallel by multiple processes in the GPU server to further improve the execution efficiency of the reconstruction task.

[0041] In the present application, a message queue can be set in the three-dimensional model reconstruction device. After the three-dimensional model reconstruction device receives a reconstruction task, if multiple GPU servers are executing the reconstruction task, the reconstruction task can be placed in the message queue. When there is an idle GPU server, the reconstruction task can be assigned to the GPU server, and the GPU server will execute the reconstruction task.

[0042] In this application, multiple processes in the GPU server can use 3DGS (3D Gaussian Splatting) to perform reconstruction tasks in parallel. Considering that 3DGS is difficult to implement multi-batch reconstruction, in this application, the parallel execution of reconstruction tasks adopts a data parallel strategy.

[0043] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0044] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a first memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a first processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0045] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the first processor 170, and can receive and execute the command sent by the first processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0046] Among them, the input device 132 can receive input data and the like.

[0047] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display an interface for image acquisition control, processing results, etc.

[0048] The first memory 120 can be used to store instructions and data. The first memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the first processor 170 with hardware, software and data resources including management computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0049] The first processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the first memory 120 and calling data stored in the first memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the first processor 170 may include one or more processing units; preferably, the first processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the first processor 170. In some embodiments, the processor, the memory, and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The first processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the first memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0050] Among them, the first memory 120 can be used to store software codes related to image acquisition control, the first processor 170 can execute the steps of image acquisition control, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0051] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the first processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0052] In this embodiment of the present application, the radio frequency unit 110 can send the collected pictures to a storage device, send the reconstruction task containing the collection task information to a three-dimensional model reconstruction device, etc.

[0053] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0054] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the first processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0055] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0056] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0057] Figure 3 An optional hardware structure diagram of the server 200 is shown.

[0058] like Figure 3 As shown, the server 200 includes a first bus 201, a second processor 202, a communication interface 203, and a second memory 204. The second processor 202, the second memory 204, and the communication interface 203 communicate with each other via the first bus 201.

[0059] The first bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0060] The second processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0061] The second memory 204 may include a volatile memory, such as a random access memory (RAM). The second memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD) or a solid state drive (SSD).

[0062] The second memory 204 may be used to store software codes related to three-dimensional model reconstruction, and the second processor 202 may execute the steps of three-dimensional model reconstruction of the chip, and may also schedule other units to implement corresponding functions.

[0063] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as the first processor 170 and the second processor 202) can be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0064] The following is a detailed introduction to the three-dimensional model reconstruction method of an embodiment of the present application in conjunction with the accompanying drawings. The three-dimensional model reconstruction method can be executed by any GPU server in the above-mentioned three-dimensional model reconstruction device.

[0065] Reference Figure 4 , Figure 4 A schematic diagram of a process of a three-dimensional model reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, a three-dimensional model reconstruction method provided in an embodiment of the present application may include the following steps: S101: Obtaining a set of original images corresponding to a reconstruction task to be performed; In the present application, after obtaining the reconstruction task to be executed, the 3D model reconstruction device needs to first obtain the corresponding original image set (i.e., the original images collected by multiple cameras) from the storage device based on the acquisition task information corresponding to the reconstruction task.

[0066] S102: Initializing the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model.

[0067] In the present application, SFM (Structure from Motion) can be used to estimate the camera pose parameters corresponding to each original image in the original image set, as well as the initial point cloud corresponding to the original image.

[0068] In the present application, each point in the initial point cloud can be converted into a 3D Gaussian distribution, which mainly includes parameters such as center position, opacity, covariance matrix, color, etc. The combination of each 3D Gaussian distribution is the initial three-dimensional model.

[0069] S103: Reconstructing the initial three-dimensional model using multiple processes according to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model.

[0070] In the present application, based on the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple processes can be used to perform multiple rounds of iterative optimization on the parameters in the initial three-dimensional model until multiple rounds of iterative optimization are completed to obtain a reconstructed three-dimensional model.

[0071] Specifically, according to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple processes can be used to perform multiple rounds of iterative optimization on the parameters in the initial three-dimensional model to obtain a reconstructed three-dimensional model; for the first round of iterative optimization, the three-dimensional model to be optimized is the initial three-dimensional model, and for non-first round of iterative optimization, the three-dimensional model to be optimized is the three-dimensional model obtained after the previous round of iterative optimization.

[0072] It should be noted that each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes multiple iterations. The end of multiple rounds of iterative optimization means that the loss of each process drops to a threshold, or the multiple rounds of iterative optimization are completed.

[0073] The three-dimensional model reconstruction method provided in the present embodiment, after obtaining the original picture set corresponding to the reconstruction task to be performed, first initializes the original picture set to obtain the camera pose parameters corresponding to each original picture in the original picture set, and the initial three-dimensional model, and then reconstructs the initial three-dimensional model using multiple processes based on the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model. Since the initial three-dimensional model is reconstructed using multiple processes, the number of iterative optimization steps can be reduced, thereby improving the reconstruction speed.

[0074] Next, the specific implementation method of each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes is introduced as follows: S201: Determine the training data corresponding to this round of iterative optimization; In a possible implementation, the training data used in each round of iterative optimization may be the original picture set and the camera pose parameters corresponding to each original picture in the original picture set.

[0075] In order to improve the optimization effect, in the present application, before each round of iterative optimization, a random seed can be used to shuffle the order of the original pictures in the original picture set to obtain a shuffled original picture set, and the shuffled original picture set and the camera pose parameters corresponding to each original picture are used as training data for this round of iterative optimization.

[0076] The random seed is determined as follows: all processes generate a random number, and the random number of all processes is summed to obtain the random seed. The random seed is broadcasted, and each process can obtain the same random seed.

[0077] It should be noted that the random seed can be determined in the above manner for each round of iterative optimization.

[0078] S202: Determine a training data subset corresponding to the current round of iterative optimization of each process, wherein the training data subset includes multiple groups of training data, and each group of training data includes an original image and a camera pose parameter corresponding to the original image; In a possible implementation, a specific implementation method of determining the training data subset corresponding to the current round of iterative optimization of each process may include the following steps: S2021: Divide the shuffled original picture set into multiple original picture subsets, each original picture subset contains multiple original pictures, each original picture subset contains the same number of original pictures, and two adjacent original picture subsets have some identical original pictures; As an implementable method, in the present application, the total amount of original pictures in the shuffled original picture set, the sorting information of each original picture in the shuffled original picture set, the total number of processes, and the sorting information of each process can be first obtained; then based on the total amount of original pictures, the sorting information of each original picture, the total number of processes, and the sorting information of each process, the shuffled original picture set is divided into multiple original picture subsets. For ease of understanding, assuming that the total amount of original pictures (i.e., the number of original pictures in the shuffled original picture set) is N, the sorting information of each original picture is 1 to N, the total number of processes (i.e., the number of processes) is COUNT, the sorting information of each process is RANK, and the value of RANK starts from 0, then the sequence number Start of the starting original picture and the sequence number End of the ending original picture in the original picture subset corresponding to each process are respectively: Start = RANK × (N / / COUNT) End = (RANK+1) ×(N / / COUNT) + N %COUNT Among them, N / / COUNT means the total number of original images divided by the total number of processes, and N % COUNT means the remainder of the total number of original images over the total number of processes. For example, if the total number of original images is 17 and the total number of processes is 3, then N / / COUNT = 5, N % COUNT = 2. Assuming that the sorting information of the three processes is 0, 1, and 2, the sequence numbers of the starting original images and the ending original images in the original image subsets corresponding to the three processes are [0, 7], [5, 12], and [10, 17], respectively. It can be seen that the original image subsets corresponding to the three processes contain the same number of original images, and there are some identical original images between two adjacent original image subsets.

[0079] S2022: For each of the original picture subsets, provide the original picture subset and the camera pose parameters corresponding to each original picture in the original picture subset to the process corresponding to the original picture subset as the training data subset corresponding to the process.

[0080] When determining each original picture subset, it is determined based on the sorting information of each process. Each original picture subset corresponds to a process. In the present application, the original picture subset can be assigned to the corresponding process based on the correspondence between the sorting information of each process and the original picture subset.

[0081] After the original image subsets are allocated in this step, the original images required for each process in this round of iterative optimization can be determined. After the original images required for each process in this round of iterative optimization are determined, the camera pose parameters corresponding to each original image are provided to each process, and each process can obtain the corresponding training data subset.

[0082] S203: Each of the processes uses the corresponding training data subset to perform multi-step iterative optimization on the parameters in the initial three-dimensional model, wherein in each iterative optimization step, each process processes a set of training data in its own training data subset, and the processing of each process is executed in parallel, and in the iterative optimization step with a preset number of steps, the parameters of the three-dimensional models on each process are synchronized and density controlled.

[0083] After determining the corresponding training data subset for each process, each process can load the initial three-dimensional model and perform multi-step iterative optimization on the parameters of the initial three-dimensional model using its corresponding training data subset.

[0084] Taking into account that the reconstruction algorithm adopted by each process is difficult to achieve multi-batch reconstruction, therefore, in the present application, the number of iterative optimization steps set in each round of iterative optimization is the same as the number of groups of training data contained in the training data subset corresponding to each process. In each step of iterative optimization, each process can only process one group of training data, and each process can execute the processing of the corresponding training data in parallel.

[0085] Specifically, in each iterative optimization step and each process, a group of training data is taken out from the obtained training data subset as the training data corresponding to the iterative optimization step, and the group of training data includes an original image and the camera pose parameters corresponding to the original image.

[0086] Among them, the implementation method of each iterative optimization step can be as follows: each process uses the three-dimensional model on the process to render the camera pose parameters in the training data to obtain a rendered image, and calculates the loss based on the difference between the rendered image and the original image in the training data, and adjusts the gradient of the parameters based on the loss calculation; synchronizes the parameter adjustment gradients corresponding to each of the processes to obtain synchronized gradients; each process uses the synchronized gradients to adjust the parameters of the three-dimensional model on the process.

[0087] After each process has processed its own set of training data, the parameter adjustment gradients corresponding to each process are obtained, and the parameter adjustment gradients corresponding to each process are averaged to obtain an average gradient, which is used as the synchronized gradient. The average gradient is broadcasted, so that each process can use the average gradient to adjust the parameters of the three-dimensional model on the process.

[0088] In the present application, in the iterative optimization step with a preset number of steps, the parameters of the three-dimensional models on each process are synchronized and the density is controlled. When synchronizing the parameters, the evaluation scores of the models in different processes are compared, so as to broadcast the model parameters with the highest score to all processes.

[0089] Specifically, it is necessary to calculate the image quality evaluation index score of the model in each process, such as the PSNR (Peaksignal-to-noise ratio) score. The larger the score, the better the quality. The process ID with the highest score is obtained. The parameters of the model in the process with the highest score are broadcast to other processes so that other processes update the parameters of the model to the parameters of the model in the process with the highest score. Then, each process can use the parameters of the model in the process with the highest score to control the density of the 3D model in its own process.

[0090] Density control of 3D models includes point densification and point pruning. For example, if the transparency of a point is lower than a threshold, pruning will be triggered. If the position gradient of a point is too large, the point will be densified (copy the point or add a new point).

[0091] It should be noted that after multiple rounds of iterative optimization, each process has a reconstructed 3D model. In this application, the evaluation scores of the 3D models reconstructed in different processes can be compared, and the one with the highest score can be taken as the final reconstructed 3D model.

[0092] A three-dimensional model reconstruction method provided in an embodiment of the present application is introduced above, and a device for executing the above-mentioned three-dimensional model reconstruction method will be introduced below.

[0093] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a three-dimensional model reconstruction device provided in an embodiment of the present application. Figure 5 As shown, the three-dimensional model reconstruction device comprises: An acquisition unit 51 is used to acquire a set of original images corresponding to a reconstruction task to be performed; An initialization processing unit 52 is used to perform initialization processing on the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model; The reconstruction unit 53 is used to reconstruct the initial three-dimensional model using multiple processes according to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model.

[0094] In a possible implementation, the reconstruction unit is specifically configured to: According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple rounds of iterative optimization are performed on the parameters in the initial three-dimensional model using multiple processes until the multiple rounds of iterative optimization are completed to obtain a reconstructed three-dimensional model; wherein each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes multi-step iterative optimization.

[0095] In a possible implementation, the reconstruction unit is used to use multiple processes to perform each round of iterative optimization on the parameters in the initial three-dimensional model, specifically to: Determine the training data corresponding to this round of iterative optimization; Determine a training data subset corresponding to the current round of iterative optimization of each of the processes, wherein the training data subset includes multiple groups of training data, and each group of training data includes an original image and a camera pose parameter corresponding to the original image; Each of the processes uses the corresponding training data subset to perform multi-step iterative optimization on the parameters in the initial three-dimensional model, wherein in each iterative optimization step, each process processes a set of training data in its own training data subset, the processing of each of the processes is executed in parallel, and in the iterative optimization steps with a preset number of steps interval, the parameters of the three-dimensional models on each of the processes are synchronized and density controlled.

[0096] In a possible implementation, when the reconstruction unit is used to determine the training data corresponding to the current round of iterative optimization, it is specifically used to: Determine the random seed corresponding to this round of iterative optimization; Using the random seed to shuffle the order of the original pictures in the original picture set, to obtain a shuffled original picture set; The shuffled original picture set and the camera pose parameters corresponding to each original picture are used as the training data corresponding to this round of iterative optimization.

[0097] In a possible implementation, when the reconstruction unit is used to determine the training data subset corresponding to the current round of iterative optimization of each process, it is specifically used to: Dividing the shuffled original picture set into a plurality of original picture subsets, each original picture subset contains a plurality of original pictures, each original picture subset contains the same number of original pictures, and two adjacent original picture subsets contain some identical original pictures; For each of the original picture subsets, the original picture subset and the camera pose parameters corresponding to each original picture in the original picture subset are provided to the process corresponding to the original picture subset as the training data subset corresponding to the process.

[0098] In a possible implementation, when the reconstruction unit is used to implement each iterative optimization step, it is specifically used to: Each of the processes uses the three-dimensional model on the process to render the camera pose parameters in the training data to obtain a rendered image, calculates a loss based on the difference between the rendered image and the original image in the training data, and adjusts the gradient based on the loss calculation parameter; Synchronizing the parameter adjustment gradients corresponding to the processes to obtain synchronized gradients; Each of the processes uses the synchronized gradients to adjust parameters of the three-dimensional model in the process.

[0099] The present application also provides an electronic device in an embodiment. Figure 6As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0100] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a second bus 604. An input / output (I / O) interface 605 is also connected to the second bus 604.

[0101] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0102] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the three-dimensional model reconstruction methods provided in the embodiments of the present application.

[0103] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any three-dimensional model reconstruction method provided in the embodiment of the present application.

[0104] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0105] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0106] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0107] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A three-dimensional model reconstruction method, characterized in that: include: Obtain the original image set corresponding to the reconstruction task to be performed; Initializing the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model; According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, the initial three-dimensional model is reconstructed using multiple processes to obtain a reconstructed three-dimensional model.

2. The method according to claim 1, characterized in that The reconstructing the initial three-dimensional model by using multiple processes according to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set to obtain the reconstructed three-dimensional model includes: According to the original picture set and the camera pose parameters corresponding to each original picture in the original picture set, multiple rounds of iterative optimization are performed on the parameters in the initial three-dimensional model using multiple processes until the multiple rounds of iterative optimization are completed to obtain a reconstructed three-dimensional model; wherein each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes multi-step iterative optimization.

3. The method according to claim 2, characterized in that Each round of iterative optimization of the parameters in the initial three-dimensional model using multiple processes includes: Determine the training data corresponding to this round of iterative optimization; Determine a training data subset corresponding to the current round of iterative optimization of each of the processes, wherein the training data subset includes multiple groups of training data, and each group of training data includes an original image and a camera pose parameter corresponding to the original image; Each of the processes uses the corresponding training data subset to perform multi-step iterative optimization on the parameters in the initial three-dimensional model, wherein in each iterative optimization step, each process processes a set of training data in its own training data subset, the processing of each of the processes is executed in parallel, and in the iterative optimization steps with a preset number of steps interval, the parameters of the three-dimensional models on each of the processes are synchronized and density controlled.

4. The method according to claim 3, characterized in that Determining the training data corresponding to the current round of iterative optimization includes: Determine the random seed corresponding to this round of iterative optimization; Using the random seed to shuffle the order of the original pictures in the original picture set, to obtain a shuffled original picture set; The shuffled original picture set and the camera pose parameters corresponding to each original picture are used as the training data corresponding to this round of iterative optimization.

5. The method according to claim 4, characterized in that Determining the training data subset corresponding to the current round of iterative optimization of each process includes: Dividing the shuffled original picture set into a plurality of original picture subsets, each original picture subset contains a plurality of original pictures, each original picture subset contains the same number of original pictures, and two adjacent original picture subsets contain some identical original pictures; For each of the original picture subsets, the original picture subset and the camera pose parameters corresponding to each original picture in the original picture subset are provided to the process corresponding to the original picture subset as the training data subset corresponding to the process.

6. The method according to claim 3, characterized in that The implementation of each iterative optimization step includes: Each of the processes uses the three-dimensional model on the process to render the camera pose parameters in the training data to obtain a rendered image, calculates a loss based on the difference between the rendered image and the original image in the training data, and adjusts the gradient based on the loss calculation parameter; Synchronizing the parameter adjustment gradients corresponding to the processes to obtain synchronized gradients; Each of the processes uses the synchronized gradients to adjust parameters of the three-dimensional model in the process.

7. A three-dimensional model reconstruction device, characterized in that: include: An acquisition unit, used to acquire a set of original images corresponding to a reconstruction task to be performed; An initialization processing unit, used to perform initialization processing on the original picture set to obtain camera pose parameters corresponding to each original picture in the original picture set, as well as an initial three-dimensional model; The reconstruction unit is used to reconstruct the initial three-dimensional model using multiple processes according to the original picture set and the camera posture parameters corresponding to each original picture in the original picture set to obtain a reconstructed three-dimensional model.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the three-dimensional model reconstruction method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the three-dimensional model reconstruction method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the three-dimensional model reconstruction method as described in any one of claims 1 to 6.

Images