Texture optimization method and device for three-dimensional reconstruction

Abstract

The application discloses a texture optimization method and device for three-dimensional reconstruction. A specific embodiment of the method comprises: acquiring a three-dimensional model and at least one frame of key frame image corresponding to the three-dimensional model; projecting the three-dimensional model to obtain a projection image corresponding to each frame of key frame image; performing image alignment on each frame of key frame image and the corresponding projection image to obtain at least one frame of aligned key frame image; and determining texture information of the three-dimensional model according to the at least one frame of aligned key frame image. The application provides a texture optimization method for three-dimensional reconstruction, improves the definition of the texture information of the three-dimensional model, and improves the quality of the three-dimensional model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a texture optimization method and device for three-dimensional reconstruction. Background Art

[0002] 3D reconstruction technology is a key technique in computer graphics and computer vision. With the continuous development of e-commerce, the demand for 3D display of objects is also increasing, and texture mapping plays a key role in this display. In recent years, with the emergence of various consumer-grade depth cameras, 3D reconstruction technology based on depth images has developed rapidly. However, in many cases, the texture information of the 3D models obtained by 3D reconstruction technology is relatively fuzzy. Summary of the Invention

[0003] The embodiments of the present application provide a texture optimization method and apparatus for three-dimensional reconstruction.

[0004] In a first aspect, an embodiment of the present application provides a texture optimization method for three-dimensional reconstruction, comprising: acquiring a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; projecting the three-dimensional model to obtain a projection image that corresponds one-to-one to the at least one key frame image; aligning each key frame image in the at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image; and determining the texture information of the three-dimensional model based on the at least one aligned key frame image.

[0005] In some embodiments, the above-mentioned image alignment of each key frame image in at least one frame of key frame images with the corresponding projection image to obtain at least one aligned key frame image includes: performing the following operations for each key frame image in at least one frame of key frame images: determining the offset information between the key frame image and the projection image corresponding to the key frame image based on the optical flow method; and adjusting the key frame image according to the offset information to obtain the aligned key frame image corresponding to the key frame image.

[0006] In some embodiments, the determining of the offset information between the key frame image and the projected image corresponding to the key frame image based on the optical flow method includes: determining a region of interest for the key frame image and the projected image corresponding to the key frame image; cropping the key frame image and the projected image corresponding to the key frame image according to the region of interest to obtain a cropped key frame image and a cropped projected image, respectively; and determining the offset information between the cropped key frame image and the cropped projected image based on the optical flow method; and

[0007] The above-mentioned adjusting the key frame image according to the offset information to obtain the aligned key frame image corresponding to the key frame image includes: adjusting the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain the aligned cropped image; using the aligned cropped image as the region of interest of the aligned key frame image to obtain the aligned key frame image corresponding to the key frame image.

[0008] In some embodiments, the above-mentioned projected three-dimensional model obtains a projection image corresponding one-to-one to at least one frame of key frame images, including: for each key frame image in the at least one frame of key frame images, performing the following operations: determining the camera posture information and camera intrinsic parameter information corresponding to the key frame image; determining the projection information based on the camera intrinsic parameter information; and obtaining the projection image of the three-dimensional model corresponding to the key frame image based on the camera posture information and the projection information.

[0009] In some embodiments, determining the texture information of the three-dimensional model based on at least one aligned key frame image includes: mapping at least one aligned key frame image to the three-dimensional model based on a back-projection relationship to determine the texture information of the three-dimensional model.

[0010] In the second aspect, an embodiment of the present application provides a texture optimization device for three-dimensional reconstruction, including: an acquisition unit, configured to acquire a three-dimensional model and at least one frame of key frame images corresponding to the three-dimensional model; a projection unit, configured to project the three-dimensional model to obtain a projection image that corresponds one-to-one to the at least one frame of key frame images; an alignment unit, configured to align each key frame image in the at least one frame of key frame images with the corresponding projection image to obtain at least one frame of aligned key frame images; and a determination unit, configured to determine the texture information of the three-dimensional model based on the at least one frame of aligned key frame images.

[0011] In some embodiments, the alignment unit is further configured to: perform the following operations for each key frame image in at least one key frame image: determine the offset information between the key frame image and the projected image corresponding to the key frame image based on the optical flow method; adjust the key frame image according to the offset information to obtain the aligned key frame image corresponding to the key frame image.

[0012] In some embodiments, the alignment unit is further configured to: determine the respective regions of interest of the key frame image and the projection image corresponding to the key frame image; crop the key frame image and the projection image corresponding to the key frame image according to the regions of interest to obtain a cropped key frame image and a cropped projection image, respectively; determine the offset information between the cropped key frame image and the cropped projection image based on the optical flow method; adjust the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain an aligned cropped image; use the aligned cropped image as the region of interest of the aligned key frame image to obtain the aligned key frame image corresponding to the key frame image.

[0013] In some embodiments, the projection unit is configured to: perform the following operations for each key frame image in at least one key frame image: determine the camera posture information and camera intrinsic parameter information corresponding to the key frame image; determine the projection information based on the camera intrinsic parameter information; and obtain a projection image of the three-dimensional model corresponding to the key frame image based on the camera posture information and the projection information.

[0014] In some embodiments, the determining unit is configured to: map at least one aligned key frame image to the three-dimensional model according to the back-projection relationship, and determine the texture information of the three-dimensional model.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0017] The texture optimization method and device for three-dimensional reconstruction provided in the embodiments of the present application obtain a three-dimensional model and at least one frame of key frame images corresponding to the three-dimensional model; project the three-dimensional model to obtain a projection image that corresponds one-to-one to the at least one frame of key frame images; align each key frame image in the at least one frame of key frame images with the corresponding projection image to obtain at least one frame of aligned key frame images; and determine the texture information of the three-dimensional model based on the at least one frame of aligned key frame images, thereby providing a texture optimization method for three-dimensional reconstruction, improving the clarity of the texture information of the three-dimensional model, and enhancing the quality of the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0020] Figure 2 is a flowchart of an embodiment of a texture optimization method for three-dimensional reconstruction according to the present application;

[0021] Figure 3 is a schematic diagram of an application scenario of the texture optimization method for three-dimensional reconstruction according to this embodiment;

[0022] Figure 4 is a flowchart of another embodiment of a texture optimization method for three-dimensional reconstruction according to the present application;

[0023] Figure 5 is a structural diagram of an embodiment of a texture optimization device for three-dimensional reconstruction according to the present application;

[0024] Figure 6 It is a structural diagram of a computer system suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] Figure 1 An exemplary architecture 100 is shown to which the texture optimization method and apparatus for three-dimensional reconstruction of the present application can be applied.

[0028] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 constitute a topological network, and network 104 is used to provide a medium for communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0029] Terminal devices 101, 102, and 103 can be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connection, information interaction, display, processing, and other functions, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, for example, to provide distributed services, or they can be implemented as a single software or software module. No specific limitations are given here.

[0030] Server 105 can be a server that provides various services, such as a background processing server that optimizes the texture information of a three-dimensional model obtained from terminal devices 101, 102, and 103 and the key frame images corresponding to the three-dimensional model. The background processing server obtains the three-dimensional model and at least one key frame image corresponding to the three-dimensional model; projects the three-dimensional model to obtain a projection image that corresponds one-to-one with the at least one key frame image; aligns each key frame image in the at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image; and determines the texture information of the three-dimensional model based on the at least one aligned key frame image. Optionally, the background processing server can feed back the three-dimensional model with optimized texture information to the terminal device for display by the terminal device. As an example, server 105 can be a cloud server.

[0031] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0032] It should also be noted that the texture optimization method for 3D reconstruction provided by the embodiments of the present disclosure can be executed by a server, a terminal device, or a server and a terminal device in cooperation with each other. Accordingly, the various components (e.g., various units, subunits, modules, and submodules) of the texture optimization apparatus for 3D reconstruction can be all located in the server, all located in the terminal device, or separately located in the server and the terminal device.

[0033] It should be understood that Figure 1The number of terminal devices, networks, and servers in the system is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. When the electronic device on which the texture optimization method for 3D reconstruction is running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., a server or terminal device) on which the texture optimization method for 3D reconstruction is running.

[0034] Continue to refer Figure 2 , a process 200 of an embodiment of a texture optimization method for 3D reconstruction is shown, comprising the following steps:

[0035] Step 201: Acquire a three-dimensional model and at least one key frame image corresponding to the three-dimensional model.

[0036] In this embodiment, the execution body of the texture optimization method for three-dimensional reconstruction (eg Figure 1 The server or terminal device in the system can obtain the three-dimensional model and at least one key frame image corresponding to the three-dimensional model remotely or locally through a wired connection or a wireless connection.

[0037] The 3D model can be a 3D model of any object generated using 3D reconstruction technology. For example, in the e-commerce field, the 3D model can be a 3D model of an item sold by an online merchant. The at least one keyframe image corresponding to the 3D model can be, for example, an image used in the process of generating the 3D model using 3D reconstruction technology.

[0038] It can be understood that the three-dimensional model obtained in this embodiment is a three-dimensional model with relatively fuzzy texture information.

[0039] Step 202 : Project the three-dimensional model to obtain a projection image that corresponds one-to-one to at least one key frame image.

[0040] In this embodiment, the execution entity may project the three-dimensional model to obtain a projection image that corresponds one-to-one to at least one key frame image.

[0041] As an example, the execution subject may project the 3D model from 3D to 2D using Open GL (Open Graphics Library) to obtain a projection image that corresponds one-to-one to at least one key frame image.

[0042] As another example, the execution entity may project the 3D model using a projection model to obtain a projection image that corresponds one-to-one with at least one key frame image, wherein the projection model represents the correspondence between the 3D model and the projection image.

[0043] In some optional implementations of this embodiment, the execution entity may implement step 202 in the following manner:

[0044] For each key frame image in at least one key frame image, perform the following operations:

[0045] First, determine the camera pose information and camera intrinsic parameter information corresponding to the key frame image.

[0046] The camera pose information and camera intrinsic parameter information corresponding to the key frame image are the camera pose information and camera intrinsic parameter information of the camera when the key frame image was taken. The camera intrinsic parameter information can be represented by the following matrix:

[0047]

[0048] Among them, f x 、f y Characterizes the focal length of the camera, c x 、c y Represents the optical center of the camera.

[0049] The camera pose information can be represented by the following matrix:

[0050]

[0051] Among them, r0-r8 are used to represent the direction information of the camera, and t0-t2 are used to represent the position information of the camera.

[0052] Second, the projection information is determined based on the camera intrinsic parameter information.

[0053] Specifically, the above execution entity can represent the projection information through the following projection matrix:

[0054]

[0055] Among them, width and height represent the width and height of the image respectively, and far and near represent the farthest and closest distances that the camera can observe respectively.

[0056] Third, according to the camera posture information and projection information, a projection image of the three-dimensional model corresponding to the key frame image is obtained.

[0057] Specifically, the projection image can be obtained by the following formula:

[0058] Position=P*V*M

[0059] Among them, P represents the projection matrix, V represents the camera posture information, M represents the point in the three-dimensional model, and Position represents the point in the projected image obtained by projection.

[0060] Step 203 : aligning each key frame image in at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image.

[0061] In this embodiment, the execution entity aligns each key frame image in at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image.

[0062] As an example, the above-mentioned execution entity can realize the image alignment between each key frame image and the corresponding projection image through one or more methods including a registration method based on the grayscale information of the image to be registered, a registration method based on the grayscale information of the image to be registered, and a registration method based on the feature information of the image to be registered.

[0063] In some optional implementations of this embodiment, the execution entity may perform step 203 as follows:

[0064] For each key frame image in at least one key frame image, perform the following operations:

[0065] First, based on the optical flow method, the offset information between the key frame image and the projection image corresponding to the key frame image is determined.

[0066] Optical flow can represent the movement information of the target caused by the movement between two consecutive frames of images, which is reflected between the key frame image and the projected image corresponding to the key frame image, and can represent the offset information.

[0067] Second, the key frame image is adjusted according to the offset information to obtain an aligned key frame image corresponding to the key frame image.

[0068] According to the offset information, the execution entity may adjust the key frame image so that the texture information of the key frame image is consistent with the texture information of the corresponding projection image.

[0069] In some optional implementations of this embodiment, in order to improve information processing speed, the execution entity may implement image alignment based on ROI (region of interest) of the image.

[0070] Specifically, with respect to the first step, the execution entity determines the respective regions of interest of the key frame image and the projection image corresponding to the key frame image; based on the regions of interest, the key frame image and the projection image corresponding to the key frame image are cropped to obtain a cropped key frame image and a cropped projection image, respectively; and based on the optical flow method, the offset information between the cropped key frame image and the cropped projection image is determined.

[0071] Regarding the above-mentioned second step, the above-mentioned execution entity adjusts the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain the aligned cropped image; and uses the aligned cropped image as the area of ​​interest of the aligned key frame image to obtain the aligned key frame image corresponding to the key frame image.

[0072] Here, taking the aligned cropped image as the region of interest to obtain the aligned key frame image corresponding to the key frame image can be compared to the reverse process of cropping the region of interest to obtain the cropped key frame image.

[0073] Step 204 : Determine texture information of the three-dimensional model based on at least one aligned key frame image.

[0074] In this embodiment, the execution entity may determine the texture information of the three-dimensional model based on at least one aligned key frame image.

[0075] As an example, the execution subject may adjust the texture information of the three-dimensional model by aligning the key frame images based on the three-dimensional reconstruction technology to obtain the three-dimensional model with optimized texture information.

[0076] In some optional implementations of this embodiment, the execution entity may map at least one aligned key frame image to a three-dimensional model based on a back-projection relationship to determine texture information of the three-dimensional model.

[0077] Specifically, for each point d = (x, y, z) on the model, the corresponding key frame image is I i , the key frame image is I i The corresponding camera posture information is V i .

[0078] Assume that the coordinates of a point in camera space are (x c ,y c ,z c ),but:

[0079]

[0080] From this we can get the projection coordinates of the point in the key frame:

[0081]

[0082] According to this projection relationship, the coordinates of each point in the three-dimensional model in the corresponding key frame image can be obtained, thereby realizing texture mapping.

[0083] Continue to see Figure 3 , Figure 3FIG3 is a schematic diagram 300 of an application scenario of the texture optimization method for three-dimensional reconstruction according to this embodiment. Figure 3 In an application scenario, user 301 sends a 3D model 303 of a toy car and keyframe images 3041, 3042, and 3043 corresponding to the 3D model to server 308 via terminal device 302. After obtaining 3D model 303 and keyframe images 3041, 3042, and 3043 corresponding to the 3D model, server 308 projects the 3D model to obtain projection images 3051, 3052, and 3053 that correspond one-to-one with keyframe images 3041, 3042, and 3043. Server 308 aligns keyframe images 3041, 3042, and 3043 with corresponding projection images 3051, 3052, and 3053 to obtain aligned keyframe images 3061, 3062, and 3063. Based on the aligned keyframe images 3061, 3062, and 3063, server 308 determines texture information of the 3D model to obtain a 3D model 307 with optimized texture information.

[0084] The method provided by the above-mentioned embodiment of the present disclosure obtains a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; projects the three-dimensional model to obtain a projection image corresponding one-to-one to the at least one key frame image; aligns each key frame image in the at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image; and determines the texture information of the three-dimensional model based on the at least one aligned key frame image, thereby providing a texture optimization method for three-dimensional reconstruction, improving the clarity of the texture information of the three-dimensional model and enhancing the quality of the three-dimensional model. Moreover, it can be understood that the computational complexity of the texture information optimization process of the present application is low, and it can be deployed in mobile devices with limited computing power, thereby expanding the scope of application of the present application.

[0085] Continue to refer Figure 4 , shows a schematic process 400 of another embodiment of a texture optimization method for three-dimensional reconstruction according to the present application, comprising the following steps:

[0086] Step 401: Acquire a three-dimensional model and at least one key frame image corresponding to the three-dimensional model.

[0087] Step 402 : Project the three-dimensional model to obtain a projection image that corresponds one-to-one to at least one key frame image.

[0088] Step 403: For each key frame image in at least one key frame image, perform the following operations:

[0089] Step 4031: Determine the regions of interest of the key frame image and the projection image corresponding to the key frame image.

[0090] Step 4032: crop the key frame image and the projection image corresponding to the key frame image according to the region of interest, to obtain a cropped key frame image and a cropped projection image, respectively.

[0091] Step 4033: Determine the offset information between the cropped key frame image and the cropped projection image based on the optical flow method.

[0092] Step 4034 : Adjust the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain an aligned cropped image.

[0093] Step 4035 : Using the aligned cropped image as the region of interest of the aligned key frame image, obtain the aligned key frame image corresponding to the key frame image.

[0094] Step 404 : Mapping at least one aligned key frame image to the three-dimensional model according to the back-projection relationship to determine texture information of the three-dimensional model.

[0095] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiment, the process 400 of the texture optimization method for 3D reconstruction in this embodiment specifically illustrates the process of image alignment based on the region of interest. In this way, this embodiment improves the information processing speed.

[0096] Continue to refer Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a texture optimization device for three-dimensional reconstruction. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 5 As shown, the texture optimization device for three-dimensional reconstruction includes: an acquisition unit 501, configured to acquire a three-dimensional model and at least one frame of key frame images corresponding to the three-dimensional model; a projection unit 502, configured to project the three-dimensional model to obtain a projection image that corresponds one-to-one to the at least one frame of key frame images; an alignment unit 503, configured to align each key frame image in the at least one frame of key frame images with the corresponding projection image to obtain at least one frame of aligned key frame images; and a determination unit 504, configured to determine texture information of the three-dimensional model based on the at least one frame of aligned key frame images.

[0098] In some optional implementations of this embodiment, the alignment unit 503 is further configured to: perform the following operations for each key frame image in at least one key frame image: determine the offset information between the key frame image and the projected image corresponding to the key frame image based on the optical flow method; adjust the key frame image according to the offset information to obtain the aligned key frame image corresponding to the key frame image.

[0099] In some optional implementations of this embodiment, the alignment unit 503 is further configured to: determine the respective regions of interest of the key frame image and the projection image corresponding to the key frame image; crop the key frame image and the projection image corresponding to the key frame image according to the regions of interest, to obtain a cropped key frame image and a cropped projection image, respectively; determine the offset information between the cropped key frame image and the cropped projection image based on the optical flow method; adjust the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image, to obtain an aligned cropped image; use the aligned cropped image as the region of interest of the aligned key frame image, to obtain the aligned key frame image corresponding to the key frame image.

[0100] In some optional implementations of this embodiment, the projection unit 502 is configured to: perform the following operations for each key frame image in at least one key frame image: determine the camera posture information and camera intrinsic parameter information corresponding to the key frame image; determine the projection information based on the camera intrinsic parameter information; and obtain a projection image of the three-dimensional model corresponding to the key frame image based on the camera posture information and the projection information.

[0101] In some optional implementations of this embodiment, the determining unit 504 is configured to: map at least one aligned key frame image to the three-dimensional model according to the back-projection relationship, and determine texture information of the three-dimensional model.

[0102] In this embodiment, an acquisition unit in a texture optimization device for three-dimensional reconstruction acquires a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; a projection unit projects the three-dimensional model to obtain a projection image that corresponds one-to-one to the at least one key frame image; an alignment unit aligns each key frame image in the at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image; and a determination unit determines texture information of the three-dimensional model based on the at least one aligned key frame image, thereby providing a texture optimization device for three-dimensional reconstruction, improving the clarity of the texture information of the three-dimensional model, and enhancing the quality of the three-dimensional model.

[0103] Reference below Figure 6 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1Schematic diagram of the structure of the computer system 600 of the devices 101, 102, 103, 105 shown. Figure 6 The device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0104] like Figure 6 As shown, the computer system 600 includes a processor (e.g., CPU, central processing unit) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the method of the present application are performed.

[0107] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0108] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a stand-alone software package, partially on the client computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0110] The units involved in the embodiments described in the present application can be implemented by software or by hardware. The described units can also be set in a processor. For example, they can be described as: a processor comprising an acquisition unit, a projection unit, an alignment unit, and a determination unit. Among them, the names of these units do not constitute a limitation on the units themselves under certain circumstances. For example, the alignment unit can also be described as "a unit that aligns each key frame image in at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image."

[0111] As another aspect, the present application further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: obtains a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; projects the three-dimensional model to obtain a projection image that corresponds one-to-one with the at least one key frame image; aligns each key frame image in the at least one key frame image with the corresponding projection image to obtain at least one aligned key frame image; and determines texture information of the three-dimensional model based on the at least one aligned key frame image.

[0112] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A texture optimization method for three-dimensional reconstruction, comprising: Acquire a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; Projecting the three-dimensional model to obtain a projection image corresponding to the at least one key frame image, including: performing the following operations for each key frame image in the at least one key frame image: determining camera pose information and camera intrinsic parameter information corresponding to the key frame image; determining projection information based on the camera intrinsic parameter information; and performing a three-dimensional to two-dimensional projection of the three-dimensional model based on the camera pose information and the projection information to obtain a projection image of the three-dimensional model corresponding to the key frame image; Performing image alignment on each key frame image in the at least one key frame image frame with the corresponding projection image to obtain at least one aligned key frame image frame, including: performing the following operations on each key frame image in the at least one key frame image frame: determining respective regions of interest of the key frame image and the projection image corresponding to the key frame image; cropping the key frame image and the projection image corresponding to the key frame image according to the regions of interest to obtain a cropped key frame image and a cropped projection image, respectively; determining offset information between the cropped key frame image and the cropped projection image based on an optical flow method; adjusting the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain an aligned cropped image; using the aligned cropped image as the region of interest of the aligned key frame image to obtain the aligned key frame image corresponding to the key frame image; The texture information of the three-dimensional model is determined according to the at least one aligned key frame image.

2. The method according to claim 1, wherein Determining the texture information of the three-dimensional model according to the at least one aligned key frame image includes: According to the back-projection relationship, the at least one aligned key frame image is mapped to the three-dimensional model to determine texture information of the three-dimensional model.

3. A texture optimization device for three-dimensional reconstruction, comprising: an acquisition unit configured to acquire a three-dimensional model and at least one key frame image corresponding to the three-dimensional model; The projection unit is configured to project the three-dimensional model to obtain a projection image corresponding to the at least one key frame image, including: performing the following operations for each key frame image in the at least one key frame image: determining camera pose information and camera intrinsic parameter information corresponding to the key frame image; determining projection information based on the camera intrinsic parameter information; and performing a three-dimensional to two-dimensional projection of the three-dimensional model based on the camera pose information and the projection information to obtain a projection image of the three-dimensional model corresponding to the key frame image; The alignment unit is configured to perform image alignment on each key frame image in the at least one key frame image frame with the corresponding projection image to obtain at least one aligned key frame image frame, including: performing the following operations for each key frame image in the at least one key frame image frame: determining respective regions of interest of the key frame image and the projection image corresponding to the key frame image; cropping the key frame image and the projection image corresponding to the key frame image according to the regions of interest to obtain a cropped key frame image and a cropped projection image, respectively; determining offset information between the cropped key frame image and the cropped projection image based on an optical flow method; adjusting the cropped key frame image according to the offset information between the cropped key frame image and the cropped projection image to obtain an aligned cropped image; using the aligned cropped image as the region of interest of the aligned key frame image to obtain the aligned key frame image corresponding to the key frame image; The determining unit is configured to determine the texture information of the three-dimensional model according to the at least one aligned key frame image.

4. The device according to claim 3, wherein The determining unit is configured to: According to the back-projection relationship, the at least one aligned key frame image is mapped to the three-dimensional model to determine texture information of the three-dimensional model.

5. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

6. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 2.

Images