Training method of three-dimensional reconstruction model, three-dimensional reconstruction method and device

Abstract

The present disclosure provides a training method for a three-dimensional reconstruction model, a three-dimensional reconstruction method and an apparatus. The method comprises: using an initial three-dimensional reconstruction model and an image feature sequence corresponding to a sample three-dimensional scene to determine a target prediction sequence corresponding to the sample three-dimensional scene; wherein the target prediction sequence is used to reconstruct a three-dimensional model corresponding to the sample three-dimensional scene; the image feature sequence is determined based on a two-dimensional sample image corresponding to the sample three-dimensional scene; and using the target prediction sequence and the target label sequence of the sample three-dimensional scene to adjust the initial three-dimensional reconstruction model to obtain a target three-dimensional reconstruction model. The present disclosure provides a set of general three-dimensional scene reconstruction algorithm frameworks that can be applied to images taken by different types of cameras (such as pinhole cameras, panoramic cameras, etc.), and can also be applied to different reconstruction targets (such as room structures, furniture layouts).

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of computer vision, three-dimensional reconstruction, etc. Background Art

[0002] 3D reconstruction refers to reconstructing a 3D model of a 3D scene (or 3D object) from its corresponding 2D image. 3D reconstruction is a long-standing research topic in computer vision.

[0003] However, the current existing 3D reconstruction methods design independent algorithms for images captured by different types of cameras (such as pinhole cameras, panoramic cameras, etc.) and different types of tasks (such as room structure estimation, furniture layout estimation). These algorithms lack versatility and are difficult to extend to other tasks.

[0004] Specifically, most algorithms for reconstructing indoor three-dimensional scenes from two-dimensional images decompose the problem into room structure estimation and furniture layout restoration, and then solve these two sub-problems separately.

[0005] Room structure estimation usually uses different algorithms depending on the image type: (1) When the input image is a perspective image, an object detection algorithm is usually used to achieve room structure estimation; (2) When the input image is a panoramic image, room structure estimation can usually be achieved by regressing room corner points and wall contours.

[0006] For the sub-problem of furniture layout restoration, existing technologies first detect furniture from images and then restore individual furniture using multimodal retrieval or 3D reconstruction. Summary of the Invention

[0007] The present disclosure provides a training method for a three-dimensional reconstruction model, a three-dimensional reconstruction method, and an apparatus to solve or alleviate one or more technical problems in the prior art.

[0008] In a first aspect, the present disclosure provides a method for training a three-dimensional reconstruction model, comprising:

[0009] Determining a target prediction sequence corresponding to the sample three-dimensional scene using the initial three-dimensional reconstruction model and the image feature sequence corresponding to the sample three-dimensional scene; wherein the target prediction sequence is used to reconstruct the three-dimensional model corresponding to the sample three-dimensional scene; and the image feature sequence is determined based on the two-dimensional image corresponding to the sample three-dimensional scene;

[0010] The target prediction sequence and the target label sequence of the sample 3D scene are used to adjust the initial 3D reconstruction model to obtain the target 3D reconstruction model.

[0011] In a second aspect, the present disclosure provides a three-dimensional reconstruction method, comprising:

[0012] Determining a target prediction sequence corresponding to the to-be-reconstructed three-dimensional scene using the target three-dimensional reconstruction model and an image feature sequence corresponding to the to-be-reconstructed three-dimensional scene; wherein the image feature sequence is determined based on a two-dimensional image corresponding to the to-be-reconstructed three-dimensional scene;

[0013] According to the target prediction sequence, a three-dimensional model corresponding to the to-be-reconstructed three-dimensional scene is reconstructed; wherein the target three-dimensional reconstruction model is obtained by training according to a training method of any three-dimensional reconstruction model.

[0014] In a third aspect, the present disclosure provides a training device for a three-dimensional reconstruction model, comprising:

[0015] The first determination module is configured to determine a target prediction sequence corresponding to the sample three-dimensional scene using the initial three-dimensional reconstruction model and the image feature sequence corresponding to the sample three-dimensional scene; wherein the target prediction sequence is used to reconstruct the three-dimensional model corresponding to the sample three-dimensional scene; and the image feature sequence is determined based on the two-dimensional sample image corresponding to the sample three-dimensional scene.

[0016] The adjustment module is used to adjust the initial 3D reconstruction model using the target prediction sequence and the target label sequence of the sample 3D scene to obtain the target 3D reconstruction model.

[0017] In a fourth aspect, the present disclosure provides a three-dimensional reconstruction device, comprising:

[0018] a third determination module, configured to determine a target prediction sequence corresponding to the to-be-reconstructed three-dimensional scene using the target three-dimensional reconstruction model and an image feature sequence corresponding to the to-be-reconstructed three-dimensional scene; wherein the image feature sequence is determined based on a two-dimensional image corresponding to the to-be-reconstructed three-dimensional scene;

[0019] The reconstruction module is used to reconstruct a three-dimensional model corresponding to the three-dimensional scene to be reconstructed according to the target prediction sequence; wherein the target three-dimensional reconstruction model is obtained by training according to a training device of any three-dimensional reconstruction model.

[0020] According to a fifth aspect, an electronic device is provided, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0024] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0025] In a seventh aspect, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0026] The beneficial effects of the technical solution provided by the present disclosure include at least reducing the complexity of three-dimensional reconstruction and improving the accuracy and efficiency of three-dimensional reconstruction.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0029] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0030] Figure 2 is a flowchart of an implementation method 200 for training a three-dimensional reconstruction model according to an embodiment of the present disclosure;

[0031] Figure 3A is a schematic diagram of a two-dimensional sample image according to an embodiment of the present disclosure Figure 1 ;

[0032] Figure 3B is a schematic diagram of a two-dimensional sample image according to an embodiment of the present disclosure Figure 2 ;

[0033] Figure 4 is a schematic diagram of determining a target label sequence corresponding to the sample three-dimensional scene according to an embodiment of the present disclosure;

[0034] Figure 5 is a schematic block diagram of an initial three-dimensional reconstruction model according to an embodiment of the present disclosure;

[0035] Figure 6A is a flowchart of an implementation of a 3D reconstruction method 600 according to an embodiment of the present disclosure;

[0036] Figure 6BThis is a schematic diagram of a three-dimensional model obtained based on target prediction sequence prediction according to an embodiment of the present disclosure. Figure 1 ;

[0037] Figure 6C This is a schematic diagram of a three-dimensional model obtained based on target prediction sequence prediction according to an embodiment of the present disclosure. Figure 2 ;

[0038] Figure 6D Schematic diagram 3 of a three-dimensional model predicted based on a target prediction sequence according to an embodiment of the present disclosure;

[0039] Figure 6E This is a schematic diagram of a three-dimensional model obtained based on target prediction sequence prediction according to an embodiment of the present disclosure. Figure 4 ;

[0040] Figure 7 is a structural diagram of a training device 700 for a three-dimensional reconstruction model according to an embodiment of the present disclosure;

[0041] Figure 8 8 is a schematic structural diagram of a training device 800 for a three-dimensional reconstruction model according to an embodiment of the present disclosure;

[0042] Figure 9 is a structural diagram of a three-dimensional reconstruction apparatus 900 according to an embodiment of the present disclosure;

[0043] Figure 10 FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The present disclosure will be described in further detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0045] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, circuits, etc. well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0046] The term "and / or" herein indicates that there may be three relationships. For example, A and / or B may indicate the existence of A alone, the existence of A and B at the same time, and the existence of B alone. The term "at least one" herein indicates any combination of at least two of any one or more of a plurality of. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0047] 3D reconstruction refers to the process of reconstructing a 3D model of a 3D scene using its corresponding 2D image. Specifically, taking an indoor 3D scene as an example, the existing 3D reconstruction method has the following implementation process:

[0048] Step 1: Structure estimation and layout estimation of indoor 3D scenes.

[0049] In some embodiments, the structure of the indoor 3D scene can be estimated based on the 2D image corresponding to the indoor 3D scene (e.g., estimating room outline information of the indoor 3D scene). Specifically, if the 2D image corresponding to the indoor 3D scene is a perspective image, then the structure of the indoor 3D scene can be estimated using an object detection algorithm; or, if the 2D image corresponding to the indoor 3D scene is a panoramic image, then the structure of the indoor 3D scene can be estimated using a regression algorithm.

[0050] Furthermore, the furniture layout of the 3D indoor scene can be estimated based on the 2D image corresponding to the 3D indoor scene. For example, the furniture contained in the 3D indoor scene and its attribute information (such as location information, size information, type information, etc.) can be estimated based on the 2D image corresponding to the 3D indoor scene.

[0051] Step 2: Reconstruct a 3D model corresponding to the indoor 3D scene based on the structure estimation results and layout estimation results of the indoor 3D scene.

[0052] Specifically, the structure estimation result and the layout estimation result of the indoor three-dimensional scene can be directly combined to reconstruct the three-dimensional model corresponding to the indoor three-dimensional scene.

[0053] However, the current existing 3D reconstruction methods design independent algorithms for images captured by different types of cameras (such as pinhole cameras, panoramic cameras, etc.) and different types of tasks (such as room structure estimation, furniture layout estimation). These algorithms lack versatility and are difficult to extend to other tasks.

[0054] Therefore, in order to solve the above problems, the embodiments of the present disclosure propose a training method for a three-dimensional reconstruction model, and implement a universal three-dimensional scene reconstruction algorithm framework based on a sequence prediction method. The framework can be applied to images taken by different types of cameras (such as pinhole cameras, panoramic cameras, etc.) and to different reconstruction targets (such as room structure and furniture layout). Figure 1 Schematic diagram of an application scenario according to an embodiment of the present disclosure. Figure 1 As shown, the application scenario includes: a server 110 and a terminal device 120.

[0055] The server 110 can be used to train the initial model to obtain a target three-dimensional reconstruction model; and deploy the target three-dimensional reconstruction model to the terminal device 120. The terminal device 120 can use the internally deployed target three-dimensional reconstruction model and the image feature sequence corresponding to the three-dimensional scene to be reconstructed to determine the target prediction sequence corresponding to the three-dimensional scene to be reconstructed; and use the target prediction sequence to determine the three-dimensional model corresponding to the three-dimensional scene to be reconstructed. The server 110 and the terminal device 120 can be connected via any type of wired or wireless network. In addition, in some embodiments, the server 110 can include an independent physical server, a server cluster consisting of multiple physical servers, a distributed system, and a distribution network (Content Delivery Network, CDN) that can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, and security services; the terminal device 120 can include an electronic device used by the user, such as a personal computer, mobile phone, tablet computer, notebook, e-book reader, and other computer devices with certain computing capabilities.

[0056] It should be noted that the above application scenarios are only provided to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0057] Figure 2This is a flow chart of the implementation of a training method 200 for a three-dimensional reconstruction model according to an embodiment of the present disclosure. The method can be applied to a training device for a three-dimensional reconstruction model. For example, the device can be deployed in a terminal or server or other processing device in a single machine, multi-machine or cluster system. The terminal can be a user equipment (UE, User Equipment), a mobile device, a personal digital assistant (PDA, Personal Digital Assistant) handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 2 As shown, the training method of the 3D reconstruction model includes:

[0058] S210, using the initial 3D reconstruction model and the image feature sequence corresponding to the sample 3D scene, determining a target prediction sequence corresponding to the sample 3D scene; wherein the target prediction sequence is used to reconstruct the 3D model corresponding to the sample 3D scene; and the image feature sequence is determined based on the 2D image corresponding to the sample 3D scene;

[0059] S220 , adjusting the initial 3D reconstruction model using the target prediction sequence and the target label sequence of the sample 3D scene to obtain a target 3D reconstruction model.

[0060] in, Figure 3A and Figure 3B is a schematic diagram of a sample two-dimensional image according to an embodiment of the present disclosure, such as Figure 3A and Figure 3B As shown, the two-dimensional sample image may include a two-dimensional view corresponding to the sample three-dimensional scene.

[0061] In one example, the image feature sequence corresponding to the sample 3D image is determined based on a pre-trained model and a 2D sample image. Pre-trained models include the Contrastive Language-Image Pre-training (CLIP) model, the Masked AutoEncoder (MAE), and the DINO model. Furthermore, the sample 3D scene may include an indoor 3D scene. The initial 3D reconstruction model may include a Sequence-to-Sequence (Seq2Seq) model, i.e., a 3D reconstruction model before or during training. The target 3D reconstruction model is a trained 3D reconstruction model obtained by training the initial 3D reconstruction model.

[0062] Based on the 3D reconstruction model training method proposed in the embodiments of the present disclosure, the trained target 3D reconstruction model can determine a target prediction sequence corresponding to the 3D scene based on the image feature sequence corresponding to the 3D scene, and use this target prediction sequence to reconstruct the 3D model corresponding to the 3D scene. In other words, the embodiments of the present disclosure can achieve 3D reconstruction by predicting the target sequence based on the image feature sequence using a sequence prediction method, thus being independent of the type of image or object to be reconstructed.

[0063] It can be understood that since the embodiment of the present disclosure can achieve the purpose of three-dimensional reconstruction based on the sequence prediction method, the embodiment of the present disclosure also needs to obtain the target label sequence corresponding to the sample three-dimensional scene before training the initial three-dimensional reconstruction model, so as to achieve the training of the initial three-dimensional reconstruction model. That is, before training the initial three-dimensional reconstruction model, the embodiment of the present disclosure also needs to use the sample three-dimensional scene to determine the target label sequence, wherein the target label sequence includes the real sequence corresponding to the sample three-dimensional scene, and the real sequence is used to reconstruct the real three-dimensional model corresponding to the sample three-dimensional scene.

[0064] Specifically, determining the target tag sequence according to the sample three-dimensional scene includes:

[0065] Determining an initial label sequence according to the sample three-dimensional scene; the initial label sequence includes a first initial label sequence and / or a second initial label sequence;

[0066] The initial label sequence is sequentially subjected to discretization processing and feature embedding processing to obtain a target label sequence.

[0067] Specifically, the initial label sequence can be used to represent all and / or part of the features of the sample 3D scene. In one example, the embodiment of the present disclosure can determine the target label sequence corresponding to the sample 3D scene based on the sample 3D scene and the domain-specific language (DSL) corresponding to the sample 3D scene. Specifically, taking the sample 3D scene as an indoor 3D scene as an example, the embodiment of the present disclosure can determine the initial label sequence corresponding to the indoor 3D scene based on the indoor 3D scene and the DSL corresponding to the indoor 3D scene.

[0068] The DSL may include a language specifically used to describe a specific field. Compared with common cross-domain general languages ​​(such as Java), the DSL can only be applied to a specific field, but the DSL can accurately express the characteristic information of the specific field.

[0069] Therefore, the initial label sequence determined based on the sample three-dimensional scene and the DSL corresponding to the sample three-dimensional scene proposed in the embodiment of the present disclosure can reflect most of the feature information contained in the sample three-dimensional scene, thereby improving the accuracy of the three-dimensional model subsequently obtained based on the target three-dimensional reconstruction model.

[0070] Of course, it should be noted that since DSL may include a custom language focusing on a specific field, the embodiment of the present disclosure also needs to determine the DSL corresponding to the sample three-dimensional scene before determining the initial label sequence based on the sample three-dimensional scene and the DSL corresponding to the sample three-dimensional scene.

[0071] Typically, the DSL for a certain field may include all and / or part of the characteristic information used to describe the field. Figure 3A and Figure 3B As shown, taking the sample three-dimensional scene as an indoor three-dimensional scene as an example, the structural information (such as contour information) and layout information (such as furniture information) corresponding to the indoor three-dimensional scene can be used to describe the indoor three-dimensional scene. Then, it can be determined that the DSL corresponding to the indoor three-dimensional scene can include information for describing the structure of the indoor three-dimensional scene (such as contour) and / or information for describing the layout of the indoor three-dimensional scene (such as furniture). For this reason, taking the sample three-dimensional scene as an indoor three-dimensional scene as an example, based on the sample three-dimensional scene and the sample three-dimensional scene DSL, the determined initial label sequence can include a sequence for describing the structure of the indoor three-dimensional scene (such as contour); and / or, a sequence for describing the layout corresponding to the indoor three-dimensional scene (such as furniture). That is, the embodiment of the present disclosure can use formula (1) to express the initial label sequence:

[0072] S=(g,q) Formula (1)

[0073] Among them, g can represent a sequence used to describe the structure of the indoor three-dimensional scene (such as the outline); q can represent a sequence used to describe the layout of the indoor three-dimensional scene (such as furniture); S can represent the initial label sequence corresponding to the indoor three-dimensional scene.

[0074] Of course, after determining the DSL corresponding to the sample three-dimensional scene and the content contained in the initial label sequence (that is, the sequence used to describe the structure of the indoor three-dimensional scene and / or the sequence used to describe the layout of the indoor three-dimensional scene); the embodiment of the present disclosure also needs to determine the specific content contained in the sequence used to describe the structure of the indoor three-dimensional scene and / or the sequence used to describe the layout of the indoor three-dimensional scene (such as the various elements contained in the sequence used to describe the layout of the indoor three-dimensional scene) to determine the initial label sequence.

[0075] For this reason, the embodiment of the present disclosure proposes determining an initial label sequence based on the sample three-dimensional scene, which may include:

[0076] Determine M vertices included in the sample three-dimensional scene, and determine a first initial label sequence corresponding to the sample three-dimensional scene based on the first coordinate information of the M vertices; M is a positive integer; and / or,

[0077] Determine N target objects contained in the sample three-dimensional scene, and determine target object feature sequences corresponding to the N target objects respectively; determine a second initial label sequence corresponding to the sample three-dimensional scene based on the target object feature sequences corresponding to the respective target objects; N is a positive integer.

[0078] Taking the sample three-dimensional scene including an indoor three-dimensional scene as an example, the first initial label sequence may include a sequence (i.e., g) for describing the structure (such as the outline) of the indoor three-dimensional scene; the second initial label sequence may include a sequence (i.e., q) for describing the layout (such as furniture) of the indoor three-dimensional scene; the M vertices contained in the sample three-dimensional scene may include the corners of the sample three-dimensional scene; and the N target objects contained in the sample three-dimensional scene may include the furniture contained in the sample three-dimensional scene.

[0079] This embodiment can represent the components contained in the three-dimensional scene (such as vertices and target objects) in the form of a sequence, which is conducive to reducing the training complexity of the target three-dimensional reconstruction model and improving the accuracy of the three-dimensional model obtained based on the target three-dimensional reconstruction model.

[0080] In addition, in order to better illustrate the specific steps of determining the initial label sequence (such as the first initial label sequence and / or the second initial label sequence) corresponding to the sample three-dimensional scene by using the sample three-dimensional scene and the DSL corresponding to the sample three-dimensional scene proposed in the embodiment of the present disclosure, the following content of the embodiment of the present disclosure will be combined with Figure 4 How to use the sample 3D scene and the DSL corresponding to the sample 3D scene to determine the initial label sequence corresponding to the sample 3D scene, that is, how to use the sample 3D scene to determine the first initial label sequence and the second initial label sequence is described in detail.

[0081] Figure 4 is a schematic diagram of determining the target label sequence corresponding to the sample three-dimensional scene according to an embodiment of the present disclosure, such as Figure 4 As shown, the embodiment of the present disclosure may adopt the following steps to determine the first initial label sequence corresponding to the sample three-dimensional scene:

[0082] Step 1.1: Determine the vertices contained in the sample three-dimensional scene.

[0083] Step 1.2: Determine a first initial label sequence corresponding to the sample three-dimensional scene based on the first coordinate information of the M vertices.

[0084] Specifically, determining a first initial label sequence corresponding to the sample three-dimensional scene according to the first coordinate information of the M vertices includes:

[0085] Using the first coordinate information of each vertex, determining the first arrangement order corresponding to each vertex;

[0086] A first initial label sequence is determined according to the first coordinate information of each vertex and the first arrangement order corresponding to each vertex.

[0087] The first sorting information corresponding to each vertex can be used to represent the position of each vertex in the first initial label sequence.

[0088] Taking the sample 3D scene as an indoor 3D scene as an example, the outline of the indoor 3D scene can be determined based on a polygonal wall. Then, when the outline of the indoor 3D scene is a polygonal wall, a first initial label sequence is determined based on the first coordinate information of multiple vertices (e.g., corners) contained in the indoor 3D scene, where the first coordinate information of the vertex (e.g., corner) can be determined based on the camera shooting position and orientation of the 2D sample image corresponding to the sample 3D scene.

[0089] Specifically, the first initial tag sequence can be determined based on formula (2-1):

[0090] g=(g1, g2, ..., g i ) Formula (2-1)

[0091] Among them, g can represent the first initial label sequence corresponding to the indoor three-dimensional scene; g i It can represent the first coordinate information corresponding to the i-th vertex (such as a corner) of the indoor three-dimensional scene.

[0092] Furthermore, if a right-handed coordinate system is established with the camera optical center of the two-dimensional sample image corresponding to the sample three-dimensional scene as the coordinate origin, the camera optical axis direction as the vertical axis (Y axis), and the negative direction of gravity as the vertical axis (Z axis), then in this case, (x i ,y i , z i ) represents the first coordinate information of the i-th vertex in the sample two-dimensional image, where x i Can represent the x-axis coordinate of the i-th vertex; y i Can represent the vertical coordinate of the i-th vertex; z i It can represent the z-axis coordinate of the i-th vertex.

[0093] However, it should be noted that in the training of the 3D reconstruction model, the z-axis coordinates of each vertex (such as a corner) have little impact on the training of the 3D model. Therefore, in order to reduce the training complexity and resources required for the 3D reconstruction model, the first coordinate information of each vertex proposed in the embodiment of the present disclosure may only include the x-axis coordinates and y-axis coordinates of each vertex. That is, the embodiment of the present disclosure may use formula (2-2) to represent the first coordinate information of the i-th vertex in the sample 2D image:

[0094] g i =(x i ,y i ) Formula (2-2)

[0095] In addition, the embodiment of the present disclosure may also determine the first arrangement order corresponding to each vertex (such as a corner) included in the sample three-dimensional scene according to the first coordinate information of each vertex.

[0096] For example, if the sample three-dimensional scene includes three corners, namely, a first corner G1, a second corner G2, and a third corner G3; and in a right-handed coordinate system established with the optical center of the camera of the two-dimensional sample image corresponding to the sample three-dimensional scene as the coordinate origin, the camera optical axis direction as the longitudinal axis (Y axis), and the negative gravity direction as the vertical axis (Z axis), the first coordinate information corresponding to corner G1 is g1, the first coordinate information corresponding to corner G2 is g2, and the first coordinate information corresponding to corner G3 is g3. Corners G1, G2, and G3 can first be sorted clockwise in the coordinate system to obtain a clockwise order corresponding to corners G1, G2, and G3, such as (G1, G2, G3); then, the first corner corresponding to the left boundary of the camera can be determined based on the camera center position and / or orientation, and the initial arrangement order can be adjusted based on the first corner corresponding to the left boundary of the camera to obtain a first arrangement order. For example, if the first corner corresponding to the left boundary of the camera is determined to be G2 based on the center position and / or orientation of the camera, then the first arrangement order corresponding to the corner G2 can be determined as follows: the corner G2 is in the first position in the first initial label sequence; and based on the positional relationship between the corner G2 and the corners G1 and G3 in the clockwise sequence, the first arrangement order corresponding to the corners G1 and G3 is determined, so that the first initial label sequence can be determined based on the first arrangement order corresponding to the corners G1, G2 and G3, such as (g2, g3, g1).

[0097] The method of determining the first initial label sequence using the coordinate information of each vertex can, to a certain extent, unify the method of determining the first initial label sequence corresponding to multiple sample three-dimensional scene images, thereby avoiding the increased complexity of the training process of the target three-dimensional reconstruction model due to the different methods of determining the first initial label sequences corresponding to multiple sample three-dimensional scene images.

[0098] In addition, the embodiment of the present disclosure may not only need to determine the first initial label sequence corresponding to the sample three-dimensional scene, but may also need to determine the second initial label sequence corresponding to the sample three-dimensional scene. Figure 4 As shown, the embodiment of the present disclosure proposes that the embodiment of the present disclosure can use the following steps to determine the second initial label sequence corresponding to the sample three-dimensional scene:

[0099] Step 2.1: Determine the target object contained in the sample three-dimensional scene.

[0100] Step 2.2: Determine the target object feature sequence corresponding to the target object.

[0101] Specifically, determining the target object feature sequence corresponding to the target object includes:

[0102] Determining attribute information of the target object; the attribute information includes at least one of the second coordinate information, category, size information, angle information, and identification information of the target object, where the identification information is used to characterize the target object;

[0103] The target object feature sequence corresponding to the target object is determined using the attribute information of the target object.

[0104] Each target object may correspond to unique identification information, and the identification information may be used to indicate the three-dimensional model corresponding to the target object.

[0105] Taking an indoor 3D scene as an example, the feature sequence of each target object corresponding to the indoor 3D scene can be determined based on the attribute information of the target object (such as furniture). For example, if the attribute information can include at least one of the target object's second coordinate information p, category c, size information l, angle information r, and identification information f, then formula (3) can be used to express the determination of the target object feature sequence corresponding to any target object Q in the sample 3D scene:

[0106] q i =(c i , l i , p i , r i , f i ) Formula (3)

[0107] Among them, q iIt can represent the target object feature sequence corresponding to the i-th target object Q in the indoor three-dimensional scene; c i It can represent the category of the i-th target object Q in the indoor three-dimensional scene; i It can represent the size information corresponding to the i-th target object Q in the indoor three-dimensional scene; i It can represent the second coordinate information corresponding to the i-th target object Q in the indoor three-dimensional scene; r i It can represent the angle information corresponding to the i-th target object Q in the indoor three-dimensional scene; f i It can represent the identification information corresponding to the i-th target object Q in the indoor three-dimensional scene.

[0108] Furthermore, if a right-handed coordinate system is established with the optical center of the camera corresponding to the two-dimensional sample image of the sample three-dimensional scene as the coordinate origin, the camera optical axis direction as the longitudinal axis (Y axis), and the negative direction of gravity as the vertical axis (Z axis), then in this case, the second coordinate information p corresponding to any target object Q can be expressed as (x Q ,y Q , z Q ) means (where x Q It can represent the x-axis coordinate of the target object Q; y Q It can represent the y-axis coordinate of the target object Q; z Q The z-axis coordinate of the target object Q may be represented. In addition, the size information l corresponding to any target object Q may include at least one of the length, height, and width of the target object Q; the angle information r corresponding to any target object Q may include the rotation angle of the target object Q relative to a horizontal plane (such as the ground); the category c to which any target object Q belongs may include doors, windows, beds, wardrobes, lamps, bedside tables, curtains, decorative paintings, carpets, desks, dressing tables, chairs, stools, and TV cabinets; and the identification information f corresponding to any target object Q may be used to characterize a three-dimensional model corresponding to the target object Q.

[0109] Determining the target object feature sequence by using the attribute information of each target object can, to a certain extent, avoid the negative impact on the subsequent training of the target three-dimensional model caused by the target object feature sequence containing too little information.

[0110] Step 2.3: Determine a second initial label sequence corresponding to the sample three-dimensional scene based on the target object feature sequences corresponding to the N target objects.

[0111] Specifically, determining a second initial label sequence corresponding to the sample three-dimensional scene according to target object feature sequences corresponding to the N target objects includes:

[0112] Determine a second arrangement order corresponding to each target object by using at least one of the second coordinate information, the category to which it belongs, and the frequency of occurrence of each target object in the sample three-dimensional scene;

[0113] According to the second arrangement order corresponding to each target object and the target object feature sequence, a second initial label sequence is determined.

[0114] The second arrangement order may be used to represent the position of the target feature sequence corresponding to each target object in the second initial label sequence.

[0115] Taking the sample 3D scene as an indoor 3D scene as an example, the layout information of the indoor 3D scene can be determined based on multiple target objects (such as furniture) in the indoor 3D scene. Then, the second initial label sequence can be determined based on the multiple target objects (such as furniture) contained in the indoor 3D scene and the target object feature sequence corresponding to each target object. For example, if the indoor 3D scene includes i pieces of furniture, then the second initial label sequence can be expressed by formula (4):

[0116] q=(q1,q2,...,q i ) Formula (4)

[0117] Wherein, q can represent the second initial label sequence corresponding to the indoor three-dimensional scene; q i It can represent the target object feature sequence corresponding to the i-th target object Q in the indoor three-dimensional scene.

[0118] In addition, the embodiment of the present disclosure can also determine the second arrangement order corresponding to each target object (such as furniture) contained in the sample three-dimensional scene based on at least one of the second coordinate information, category and occurrence frequency of each target object (such as furniture).

[0119] For example, if the sample three-dimensional scene contains furniture Q1, furniture Q2, and furniture Q3, where furniture Q1 belongs to the category of bed, furniture Q2 belongs to the category of door, and furniture Q3 belongs to the category of carpet. Then, if the second arrangement order corresponding to each target object is predetermined to include: the position of the target object feature sequence corresponding to the furniture belonging to the category of bed in the second initial label sequence is before the target object feature sequence corresponding to the furniture belonging to the category of door, the position of the target object feature sequence corresponding to the furniture belonging to the category of bed in the second initial label sequence is before the target object feature sequence corresponding to the furniture belonging to the category of carpet, and the position of the target object feature sequence corresponding to the furniture belonging to the category of door in the second initial label sequence is before the position of the furniture belonging to the category of carpet, then the second initial label sequence can be determined to be (q1, q2, q3).

[0120] Alternatively, if the sample three-dimensional scene includes furniture Q1, furniture Q2, and furniture Q3, where the frequency of occurrence corresponding to furniture Q1 is w1, the frequency of occurrence corresponding to furniture Q2 is w2, and the frequency of occurrence corresponding to furniture Q3 is w3; then, by comparing the frequencies of occurrence corresponding to furniture Q1, furniture Q2, and furniture Q3, the second arrangement order corresponding to each piece of furniture can be determined, and based on the second arrangement order corresponding to each piece of furniture, the second initial label sequence can be determined. Specifically, the second arrangement order corresponding to each piece of furniture can be determined based on the ascending (or descending) order of the frequencies of occurrence corresponding to furniture Q1, furniture Q2, and furniture Q3. For example, if the ascending order of the occurrence frequencies of furniture Q1, furniture Q2 and furniture Q3 is "w1, w2, w3", then it can be determined that the corresponding second arrangement order of furniture Q1, furniture Q2 and furniture Q3 may include: in the second initial label sequence, furniture Q1 is located before the position of furniture Q2; in the second initial label sequence, furniture Q2 is located before the position of furniture Q3; and, in the second initial label sequence, furniture Q1 is located before the position of furniture Q3, then the second initial label sequence can be determined to be (q1, q2, q3).

[0121] Alternatively, if the sample three-dimensional scene includes furniture Q1, furniture Q2 and furniture Q3, and a right-handed coordinate system is established with the camera optical center of the two-dimensional sample image corresponding to the sample three-dimensional scene as the coordinate origin, the camera optical axis direction as the longitudinal axis (Y axis), and the negative direction of gravity as the vertical axis (Z axis), the second coordinate information corresponding to the furniture Q1 is p1 = (x1, y1, z1) (that is, the x-axis coordinate corresponding to the furniture Q1 is x1), the second coordinate information corresponding to the furniture Q2 is p2 = (x2, y2, z2) (that is, the x-axis coordinate corresponding to the furniture Q2 is x2), and the second coordinate information corresponding to the furniture Q3 includes p3 = (x3, y3, z3) (that is, the x-axis coordinate corresponding to the furniture Q3 is x3). At this point, the second arrangement order corresponding to each piece of furniture can be determined by comparing the x-axis coordinates corresponding to furniture Q1, furniture Q2, and furniture Q3 (or by comparing the y-axis coordinates corresponding to furniture Q1, furniture Q2, and furniture Q3, and the z-axis coordinates corresponding to furniture Q1, furniture Q2, and furniture Q3). Based on the second arrangement order corresponding to each piece of furniture, the second initial label sequence can be determined. Specifically, the position of each piece of furniture in the second initial label sequence can be determined based on the ascending (or descending) order of the x-axis coordinates corresponding to furniture Q1, furniture Q2, and furniture Q3. For example, if the x-axis coordinates corresponding to furniture Q1, furniture Q2, and furniture Q3 are arranged in ascending order as "x1, x2, x3", then the second arrangement order corresponding to furniture Q1, furniture Q2, and furniture Q3 can be determined to include: in the second initial label sequence, the position of furniture Q1 is before the position of furniture Q2; in the second initial label sequence, the position of furniture Q2 is before the position of furniture Q3; and, in the second initial label sequence, the position of furniture Q1 is before the position of furniture Q3. In this case, the second initial label sequence can be determined to be (q1, q2, q3).

[0122] The method of determining the second initial label sequence using at least one of the second coordinate information, category and occurrence frequency corresponding to each target object can achieve the purpose of sorting the feature sequences of each target object in the second initial label sequence, thereby reducing the complexity, required resources and time required for three-dimensional reconstruction model training.

[0123] Of course, it should be noted that the above order of determining the first initial tag sequence and the second initial tag sequence is only an example, and the present disclosure does not limit the specific order of determining the first initial tag sequence and the second initial tag sequence.

[0124] Of course, it should be noted that, in some implementations, after determining the initial tag sequence, the embodiments of the present disclosure also need to use the initial tag sequence to determine the target tag sequence.

[0125] Specifically, as Figure 4 As shown, the embodiment of the present disclosure can adopt the following steps to determine the target tag sequence:

[0126] Step 3.1: Discretize the initial tag sequence S (for example, the first initial tag sequence or the second initial tag sequence) to obtain a tag sequence S′ after discretization. The elements s′ contained in the discretized sequence S′ are i All are discrete values.

[0127] Specifically, we can first determine the elements s contained in the initial tag sequence S i Whether it is a discrete value; the element s contained in the initial label sequence S i It can be divided into the following six categories: g contained in the first initial tag sequence i , c contained in the second initial tag sequence i , l contained in the second initial tag sequence i , p contained in the second initial tag sequence i , r contained in the second initial tag sequence i and f contained in the second initial tag sequence i wait.

[0128] Then, if any element s contained in the initial tag sequence S i is a discrete value, then the element s can be directly i Determine the element s' in the sequence S' i , and the position of the element in the initial label sequence S is the same as the position of the sequence S′; or, if any element in the initial label sequence S is a continuous value, then the element s i Perform discretization to obtain the discretized element s′ i , and the discretized element s′ i Save to sequence S', and element s i The position of S in the initial label sequence corresponds to the discretized element s' i The positions in the sequence S' are the same.

[0129] Step 3.2: The discretized label sequence S′ is subjected to feature embedding processing to obtain the target label sequence E(S′).

[0130] Specifically, the formula (5-1) can be used to convert each element s′ in the discrete label sequence S′ into i Perform word embedding processing to obtain each element s′ i The corresponding feature embedding E(s′ i), and use formula (5-2) to determine the target tag sequence E(S'):

[0131] E(s′ i )=E value (s′ i )+E type (s′ i )+E pos (s′ i ) Formula (5-1)

[0132] E(S′)=[E(s′1),E(s′2),...,E(s′ i )] Formula (5-2)

[0133] Among them, E value Can represent element s′ i The corresponding value embedding; E type Can represent element s′ i Type embedding of E pos Represents element s′ i The relative position in its sequence (such as the first initial tag sequence or the second initial tag sequence).

[0134] In one example, in the element s′ i is the g contained in the first initial tag sequence i , c contained in the second initial tag sequence i , l contained in the second initial tag sequence i , p contained in the second initial tag sequence i , r contained in the second initial tag sequence i When the E value According to the element s′ i The corresponding eigenvector is determined; in this element s′ i is the f contained in the second initial tag sequence i Then, according to the element s′ i The corresponding image feature determines E value , where the acquired element s′ i The corresponding image features may include: inputting the two-dimensional rendering corresponding to the three-dimensional furniture model into a pre-trained model, such as a contrastive language-image pre-training (CLIP) model, and the CLIP model outputs the element s′ i The corresponding image features. type According to the element s′ i The six categories corresponding to the first initial label sequence are determined, that is, the g i, c contained in the second initial tag sequence i , l contained in the second initial tag sequence i , p contained in the second initial tag sequence i , r contained in the second initial tag sequence i and f contained in the second initial tag sequence i These six categories. pos According to the element s′ i The position in the sequence S' is determined.

[0135] The above content briefly describes how the disclosed embodiments determine the target label sequence and image feature sequence corresponding to the sample 3D scene. Of course, it should be noted that after determining the target label sequence and image feature sequence, the disclosed embodiments also need to use the target label sequence and image feature sequence to train the initial 3D model to obtain the target 3D model.

[0136] Therefore, how the initial 3D model is trained based on the target label sequence and the image feature sequence is also a problem that needs to be considered in the embodiments of the present disclosure. Specifically, the following content of the embodiments of the present disclosure will explain in detail how to train the initial 3D model using the target label sequence and the image feature sequence.

[0137] Figure 5 is a schematic block diagram of an initial three-dimensional reconstruction model according to an embodiment of the present disclosure, such as Figure 5 As shown, the initial 3D reconstruction model includes a transformer encoding layer and a transformer decoding layer;

[0138] The Transformer encoding layer is used to encode the image feature sequence to obtain the semantic features corresponding to the image feature sequence;

[0139] The Transformer decoding layer is used to determine a first prediction sequence using semantic features and historical prediction sequences corresponding to the sample three-dimensional scene; the first prediction sequence is used to determine a target prediction sequence.

[0140] The Transformer encoding layer and the Transformer decoding layer included in the initial 3D reconstruction model can both be composed of 6 Transformer modules.

[0141] In one example, the target prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene at a first moment;

[0142] The historical prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene in the first time sequence;

[0143] The first moment is after the first time series.

[0144] For example, if the first time series includes moments within the time period [tk, t), such as moments tk, t-k+1, t-k+2, ..., t-1; then, the first moment may include moments after the first time series, such as moment t.

[0145] Based on this, if the image feature sequence is input into the initial 3D reconstruction model, the decoding layer of the initial 3D reconstruction model can receive the semantic features of the image feature sequence output by the encoding layer at time t, as well as the output sequence (i.e., historical prediction sequence) E(s) of the decoding layer for the image feature sequence at the previous time t-1. ＜t ); The encoding layer of the initial 3D reconstruction model receives the semantic features of the image feature sequence at time t and the output sequence (ie, historical prediction sequence) E(s before time t-1) ＜t ) can be used to calculate the semantic features of the image feature sequence at time t and the output sequence (i.e., historical prediction sequence) E(s ＜t ), determine the first prediction sequence corresponding to the sample three-dimensional scene, that is, the hidden feature h corresponding to the sample three-dimensional scene t , and based on the first prediction feature, determine the target prediction feature.

[0146] Transformer has the ability to model long-range dependencies, which enables the initial three-dimensional reconstruction model to take global information into account at the same time. Therefore, when the initial three-dimensional reconstruction model includes a Transformer encoding layer and a Transformer decoding layer, the present disclosure can improve the efficiency of training the initial three-dimensional reconstruction model, and enable the initial three-dimensional reconstruction model to have the advantages of powerful expression ability, parallel computing, long-range dependency modeling, multi-head attention mechanism and interpretability, thereby improving the performance and accuracy of the initial three-dimensional reconstruction model. In addition, since the target prediction sequence is determined by using multiple intermediate values ​​predicted by the initial three-dimensional reconstruction model (i.e., historical prediction sequence), the error caused by a single prediction can be reduced based on the embodiment of the present disclosure, thereby improving the prediction accuracy of the target three-dimensional reconstruction model.

[0147] Further, as Figure 5 As shown, the initial 3D reconstruction model also includes a linear layer and a normalization layer; wherein,

[0148] The linear layer is used to adjust the dimension of the first prediction sequence to obtain an adjusted first prediction sequence;

[0149] The normalization layer is used to normalize the adjusted first prediction sequence according to the image feature sequence corresponding to the sample three-dimensional scene to obtain a second prediction sequence; wherein the second prediction sequence is used to determine the target prediction sequence.

[0150] For example, still taking the first time series including the moments in the time period [tk, t), and the first moment including the moment t as an example, if the first prediction sequence corresponding to the sample three-dimensional scene is obtained, that is, the hidden feature h corresponding to the sample three-dimensional scene t Afterwards, the embodiment of the present disclosure can use formula (6) to determine the second prediction sequence s corresponding to the sample three-dimensional scene t :

[0151] p(s t |s ＜t , I) = p(E(s t )|E(s ＜t ), F) = softmax(linear(h t )) Formula (6)

[0152] Among them, F can represent the image feature sequence corresponding to the sample two-dimensional image I; E(s t ) can represent the second prediction sequence corresponding to the sample three-dimensional image.

[0153] That is, the embodiment of the present disclosure can first use the linear layer of the initial 3D reconstruction model to adjust the hidden feature h t dimension to obtain the adjusted hidden features; then, the hidden features after the adjusted dimension can be processed using the normalization layer of the initial three-dimensional reconstruction model to obtain a second prediction sequence corresponding to the sample three-dimensional image and the occurrence probability corresponding to the second prediction sequence.

[0154] Furthermore, after determining the second prediction sequence corresponding to the sample three-dimensional image, the target prediction sequence can be determined using formula (7):

[0155] p(S′|I)=П t p(s t |s ＜t , I)=П t p(E(s t )|E(s ＜t ), F) Formula (7)

[0156] Among them, I represents the two-dimensional image corresponding to the sample three-dimensional scene, F represents the image feature sequence corresponding to the two-dimensional image I; S′ can represent the target prediction sequence corresponding to the sample three-dimensional scene.

[0157] Of course, it should be noted that after determining the target prediction sequence corresponding to the sample three-dimensional scene, the embodiment of the present disclosure can also use the target prediction sequence and the target label sequence of the sample three-dimensional scene to adjust the initial three-dimensional reconstruction model to obtain the target three-dimensional reconstruction model.

[0158] Specifically, the embodiment of the present disclosure can use the cross-entropy loss function according to the target prediction sequence and target label sequence corresponding to the sample three-dimensional scene, and then adjust the initial three-dimensional reconstruction model based on the loss function to obtain the target three-dimensional reconstruction model.

[0159] Among them, the value of the loss function is related to the difference between the target prediction sequence and the target label sequence corresponding to the sample three-dimensional scene. The larger the difference between the two, the larger the corresponding loss function. Determining the loss function in this way can accelerate the convergence of the model and improve the training speed of the model.

[0160] The above loss function determination method is only an example. The embodiment of the present disclosure may also adopt other loss function determination methods, and the embodiment of the present disclosure does not limit the loss function determination method.

[0161] Figure 6A FIG. 6 is a flowchart of a 3D reconstruction method 600 according to an embodiment of the present disclosure, comprising:

[0162] S610: Determine a target prediction sequence corresponding to the 3D scene to be reconstructed using the target 3D reconstruction model and the image feature sequence corresponding to the 3D scene to be reconstructed; wherein the image feature sequence is determined based on the 2D image corresponding to the 3D scene to be reconstructed;

[0163] S620: Reconstruct a three-dimensional model corresponding to the to-be-reconstructed three-dimensional scene according to the target prediction sequence; wherein the target three-dimensional reconstruction model is obtained by training according to any three-dimensional reconstruction model training method.

[0164] The two-dimensional image may include a panoramic image and / or a perspective image. Figure 6B and Figure 6C As shown, in the case where the two-dimensional image includes a panoramic image, the image feature sequence corresponding to the to-be-reconstructed three-dimensional scene is input into the target three-dimensional reconstruction model, and the target three-dimensional reconstruction model can output a target prediction sequence for the to-be-reconstructed three-dimensional scene, and use the target prediction sequence to determine the three-dimensional model corresponding to the three-dimensional reconstruction; and Figure 6D and Figure 6E As shown, in the case where the two-dimensional image includes a perspective view, the image feature sequence corresponding to the three-dimensional scene to be reconstructed is input into the target three-dimensional reconstruction model, and the target three-dimensional reconstruction model can output a target prediction sequence for the three-dimensional scene to be reconstructed, and use the target prediction sequence to determine the three-dimensional model corresponding to the three-dimensional reconstruction.

[0165] The three-dimensional reconstruction method proposed in the embodiment of the present disclosure can achieve the purpose of three-dimensional prediction based on the sequence prediction method, thereby reducing the complexity of three-dimensional reconstruction, improving the efficiency of three-dimensional reconstruction, and improving the accuracy of the reconstructed three-dimensional model.

[0166] The target 3D reconstruction model includes a Transformer encoding layer and a Transformer decoding layer;

[0167] The Transformer encoding layer is used to process the image feature sequence to obtain the semantic features corresponding to the image feature sequence;

[0168] The Transformer decoding layer is used to determine a first prediction sequence using semantic features and a historical prediction sequence corresponding to the 3D scene to be reconstructed; the first prediction sequence is used to determine a target prediction sequence.

[0169] The Transformer's ability to model long-range dependencies enables the target 3D reconstruction model to simultaneously consider global information. Therefore, since the target 3D reconstruction model includes both the Transformer encoding and decoding layers, it improves the prediction efficiency of the target 3D reconstruction model and provides it with advantages such as powerful expressiveness, parallel computing, long-range dependency modeling, multi-head attention mechanisms, and interpretability, thereby improving the performance and accuracy of the target 3D reconstruction model.

[0170] The target 3D reconstruction model also includes a linear layer and a normalization layer; among them,

[0171] The linear layer is used to adjust the dimension of the first prediction sequence to obtain an adjusted first prediction sequence;

[0172] The normalization layer is used to perform normalization processing on the adjusted first prediction sequence to obtain a second prediction sequence; wherein the second prediction sequence is used to determine the target prediction sequence.

[0173] The target prediction sequence includes a prediction sequence of the target 3D reconstruction model for the 3D scene to be reconstructed at the first moment;

[0174] The historical prediction sequence includes a prediction sequence of the target three-dimensional reconstruction model for the to-be-reconstructed three-dimensional scene in the second time sequence;

[0175] The first time series follows the second time series.

[0176] For example, if the first time series includes moments in a time period of [tk, t], such as moments tk, t-k+1, t-k+2, ..., t-1; then the first moment may be moment t.

[0177] The present disclosure also provides a training device for a three-dimensional reconstruction model. Figure 7 FIG. 7 is a schematic structural diagram of a 3D reconstruction model training device 700 according to an embodiment of the present disclosure, comprising:

[0178] The first determination module 710 is configured to determine a target prediction sequence corresponding to the sample 3D scene using the initial 3D reconstruction model and the image feature sequence corresponding to the sample 3D scene; wherein the target prediction sequence is used to reconstruct the 3D model corresponding to the sample 3D scene; and the image feature sequence is determined based on the 2D sample image corresponding to the sample 3D scene.

[0179] The adjustment module 720 is configured to adjust the initial 3D reconstruction model using the target prediction sequence and the target label sequence of the sample 3D scene to obtain a target 3D reconstruction model.

[0180] Figure 8 FIG. 8 is a schematic diagram of a training device 800 for a three-dimensional reconstruction model according to an embodiment of the present disclosure. Figure 8 As shown, the 3D reconstruction model training device 800 further includes:

[0181] The second determination module 830 is used to determine the target label sequence based on the sample three-dimensional scene, wherein the target label sequence includes a real sequence corresponding to the sample three-dimensional scene, and the real sequence is used to reconstruct a real three-dimensional model corresponding to the sample three-dimensional scene.

[0182] In some embodiments, the second determining module 830 is configured to determine an initial label sequence based on the sample three-dimensional scene; the initial label sequence includes a first initial label sequence and / or a second initial label sequence;

[0183] The initial label sequence is sequentially subjected to discretization processing and feature embedding processing to obtain a target label sequence.

[0184] In some implementations, the second determining module 830 includes:

[0185] The first determination submodule 831 is configured to determine M vertices included in the sample three-dimensional scene, and determine a first initial label sequence corresponding to the sample three-dimensional scene based on the first coordinate information of the M vertices; M is a positive integer; and / or,

[0186] The second determination submodule 832 is used to determine the N target objects contained in the sample three-dimensional scene, and respectively determine the target object feature sequences corresponding to the N target objects, and determine the second initial label sequence corresponding to the sample three-dimensional scene based on the target object feature sequences corresponding to the N target objects; N is a positive integer.

[0187] In some implementations, the first determining submodule 831 is configured to:

[0188] Using the first coordinate information of each vertex, determine the position of each vertex in the first initial label sequence;

[0189] The first initial label sequence is determined according to the first coordinate information of each vertex in the sample three-dimensional scene and the position of each vertex in the first initial label sequence.

[0190] In some implementations, the second determining submodule 832 is configured to:

[0191] Determine the position of the target object feature sequence corresponding to each target object in the second initial label sequence using at least one of the second coordinate information, the category, and the occurrence frequency of each target object in the sample three-dimensional scene;

[0192] The second initial label sequence is determined according to each target object feature sequence and the position of each target object feature sequence in the second initial label sequence.

[0193] In some implementations, the second determining submodule 832 is configured to:

[0194] Determining attribute information of the target object; the attribute information includes at least one of second coordinate information, category, size information, angle information, and identification information of the target object, where the identification information corresponds to the target object;

[0195] The target object feature sequence corresponding to the target object is determined using the attribute information of the target object.

[0196] In some embodiments, the initial 3D reconstruction model includes a Transformer encoding layer and a Transformer decoding layer; wherein,

[0197] The Transformer encoding layer is used to process the image feature sequence to obtain the semantic features corresponding to the image feature sequence;

[0198] The Transformer decoding layer is used to determine a first prediction sequence using semantic features and historical prediction sequences corresponding to the sample three-dimensional scene; the first prediction sequence is used to determine a target prediction sequence.

[0199] In some embodiments, the initial three-dimensional reconstruction model further includes a linear layer and a normalization layer; wherein,

[0200] The linear layer is used to adjust the dimension of the first prediction sequence to obtain an adjusted first prediction sequence;

[0201] The normalization layer is used to perform normalization processing on the adjusted first prediction sequence to obtain a second prediction sequence; wherein the second prediction sequence is used to determine the target prediction sequence.

[0202] In some embodiments, the target prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene in a first time sequence;

[0203] The historical prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene in the second time series;

[0204] The first time series follows the second time series.

[0205] The present disclosure also provides a three-dimensional reconstruction device. Figure 9 FIG. 9 is a schematic structural diagram of a 3D reconstruction apparatus 900 according to an embodiment of the present disclosure, comprising:

[0206] A third determination module 910 is configured to determine a target prediction sequence corresponding to the to-be-reconstructed 3D scene using the target 3D reconstruction model and the image feature sequence corresponding to the to-be-reconstructed 3D scene; wherein the image feature sequence is determined based on the 2D image corresponding to the to-be-reconstructed 3D scene;

[0207] The reconstruction module 920 is used to reconstruct a 3D model corresponding to the 3D scene to be reconstructed according to the target prediction sequence; wherein the target 3D reconstruction model is trained according to any 3D reconstruction model training device.

[0208] In some embodiments, the target 3D reconstruction model includes a Transformer encoding layer and a Transformer decoding layer; wherein,

[0209] The initial 3D reconstruction model includes the Transformer encoding layer and the Transformer decoding layer;

[0210] The Transformer encoding layer is used to process the image feature sequence to obtain the semantic features corresponding to the image feature sequence;

[0211] The Transformer decoding layer is used to determine a first prediction sequence using semantic features and a historical prediction sequence corresponding to the 3D scene to be reconstructed; the first prediction sequence is used to determine a target prediction sequence.

[0212] In some embodiments, the target three-dimensional reconstruction model further includes a linear layer and a normalization layer; wherein,

[0213] The linear layer is used to adjust the dimension of the first prediction sequence to obtain an adjusted first prediction sequence;

[0214] The normalization layer is used to perform normalization processing on the adjusted first prediction sequence to obtain a second prediction sequence; wherein the second prediction sequence is used to determine the target prediction sequence.

[0215] In some embodiments, the target prediction sequence includes a prediction sequence of the target three-dimensional reconstruction model for the three-dimensional scene to be reconstructed in a first time sequence;

[0216] The historical prediction sequence includes a prediction sequence of the target three-dimensional reconstruction model for the to-be-reconstructed three-dimensional scene in the second time sequence;

[0217] The first time series follows the second time series.

[0218] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0219] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0220] Figure 10 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 10 As shown, the electronic device includes: a memory 1010 and a processor 1020. The memory 1010 stores a computer program that can be executed on the processor 1020. The number of memories 1010 and processors 1020 can be one or more. The memory 1010 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device performs the method provided by the above method embodiment. The electronic device may also include: a communication interface 1030 for communicating with external devices and performing data exchange.

[0221] If the memory 1010, the processor 1020, and the communication interface 1030 are implemented independently, the memory 1010, the processor 1020, and the communication interface 1030 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0222] Optionally, in a specific implementation, if the memory 1010, the processor 1020 and the communication interface 1030 are integrated on a chip, the memory 1010, the processor 1020 and the communication interface 1030 can communicate with each other through an internal interface.

[0223] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0224] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0225] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)). It is worth noting that the computer-readable storage media mentioned in the present disclosure may be non-volatile storage media, in other words, non-transitory storage media.

[0226] Those skilled in the art will understand that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or by programs instructing related hardware to accomplish the steps. The programs may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0227] In the description of the embodiments of the present disclosure, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0228] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or. For example, A / B can mean A or B. "And / or" in this document is only a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0229] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0230] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for training a three-dimensional reconstruction model, comprising: Determining a target prediction sequence corresponding to the sample three-dimensional scene using an initial three-dimensional reconstruction model and an image feature sequence corresponding to the sample three-dimensional scene; wherein the target prediction sequence is used to reconstruct a three-dimensional model corresponding to the sample three-dimensional scene; and the image feature sequence is determined based on a two-dimensional image corresponding to the sample three-dimensional scene; Adjusting the initial 3D reconstruction model using the target prediction sequence and the target label sequence of the sample 3D scene to obtain a target 3D reconstruction model; wherein the target label sequence is determined based on an initial label sequence corresponding to the sample 3D scene; and the initial label sequence includes a first initial label sequence and / or a second initial label sequence; The method for determining the initial tag sequence includes: Determine M vertices included in the sample three-dimensional scene, and determine a first initial label sequence corresponding to the sample three-dimensional scene based on first coordinate information of the M vertices; M is a positive integer; and / or, Determine N target objects contained in the sample three-dimensional scene, and respectively determine target object feature sequences corresponding to the N target objects; determine a second initial label sequence corresponding to the sample three-dimensional scene based on the target object feature sequences corresponding to each of the target objects; N is a positive integer.

2. The method according to claim 1, wherein The target label sequence includes a real sequence corresponding to the sample three-dimensional scene, and the real sequence is used to reconstruct a real three-dimensional model corresponding to the sample three-dimensional scene.

3. The method according to claim 2, wherein: Determining the target label sequence according to the initial label sequence corresponding to the sample three-dimensional scene includes: sequentially performing discretization processing and feature embedding processing on the initial label sequence to obtain the target label sequence.

4. The method according to claim 3, wherein: The determining, based on the first coordinate information of the M vertices, a first initial label sequence corresponding to the sample three-dimensional scene includes: Using the first coordinate information of each of the vertices, determining a first arrangement order corresponding to each of the vertices; The first initial label sequence is determined according to the first coordinate information of each of the vertices and the first arrangement order corresponding to each of the vertices.

5. The method according to claim 3, wherein The determining, based on the target object feature sequences corresponding to the target objects, a second initial label sequence corresponding to the sample three-dimensional scene includes: Determine a second arrangement order corresponding to each target object by using at least one of the second coordinate information, the category, and the occurrence frequency of each target object in the sample three-dimensional scene; According to the second arrangement order corresponding to each of the target objects and the target object feature sequence, the second initial label sequence is determined.

6. The method according to claim 5, wherein: Determining a target object feature sequence corresponding to the target object includes: Determining attribute information of the target object; the attribute information includes at least one of second coordinate information, category, size information, angle information, and identification information of the target object, wherein the identification information is used to characterize the target object; The target object feature sequence corresponding to the target object is determined using the attribute information of the target object.

7. The method according to claim 6, wherein: The initial 3D reconstruction model includes a transformer encoding layer and a transformer decoding layer; wherein, The Transformer encoding layer is used to process the image feature sequence to obtain semantic features corresponding to the image feature sequence; The Transformer decoding layer is used to determine a first prediction sequence using the semantic features and a historical prediction sequence corresponding to the sample three-dimensional scene; the first prediction sequence is used to determine the target prediction sequence.

8. The method according to claim 7, wherein: The initial three-dimensional reconstruction model also includes a linear layer and a normalization layer; wherein, The linear layer is used to adjust the feature dimension of the first prediction sequence to obtain an adjusted first prediction sequence; The normalization layer is used to perform normalization processing on the adjusted first prediction sequence to obtain a second prediction sequence; wherein the second prediction sequence is used to determine the target prediction sequence.

9. The method according to claim 7, wherein: The target prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene at a first moment; The historical prediction sequence includes a prediction sequence of the initial three-dimensional reconstruction model for the sample three-dimensional scene in a first time sequence; The first moment is after the first time sequence.

10. A three-dimensional reconstruction method, comprising: Determining a target prediction sequence corresponding to the to-be-reconstructed three-dimensional scene using a target three-dimensional reconstruction model and an image feature sequence corresponding to the to-be-reconstructed three-dimensional scene; wherein the image feature sequence is determined based on a two-dimensional image corresponding to the to-be-reconstructed three-dimensional scene; Reconstruct a three-dimensional model corresponding to the three-dimensional scene to be reconstructed according to the target prediction sequence; wherein the target three-dimensional reconstruction model is trained according to the three-dimensional reconstruction model training method according to any one of claims 1-9.

11. A training device for a three-dimensional reconstruction model, comprising: A first determination module is configured to determine a target prediction sequence corresponding to the sample three-dimensional scene using an initial three-dimensional reconstruction model and an image feature sequence corresponding to the sample three-dimensional scene; in; The target prediction sequence is used to reconstruct a three-dimensional model corresponding to the sample three-dimensional scene; the image feature sequence is determined based on the two-dimensional sample image corresponding to the sample three-dimensional scene; A second determination module is configured to determine a target label sequence based on an initial label sequence corresponding to the sample three-dimensional scene; the initial label sequence includes a first initial label sequence and / or a second initial label sequence; an adjustment module, configured to adjust the initial 3D reconstruction model using the target prediction sequence and the target label sequence of the sample 3D scene to obtain a target 3D reconstruction model; The second determination module is configured to determine M vertices included in the sample three-dimensional scene, and determine a first initial label sequence corresponding to the sample three-dimensional scene based on first coordinate information of the M vertices; M is a positive integer; and / or, Determine N target objects contained in the sample three-dimensional scene, and respectively determine target object feature sequences corresponding to the N target objects; determine a second initial label sequence corresponding to the sample three-dimensional scene based on the target object feature sequences corresponding to each of the target objects; N is a positive integer.

12. The device according to claim 11, wherein The second determination module is used to perform discretization processing and feature embedding processing on the initial label sequence in sequence to obtain a target label sequence.

13. A three-dimensional reconstruction device, comprising: a third determination module, configured to determine a target prediction sequence corresponding to the 3D scene to be reconstructed using the target 3D reconstruction model and an image feature sequence corresponding to the 3D scene to be reconstructed; wherein the image feature sequence is determined based on a 2D image corresponding to the 3D scene to be reconstructed; A reconstruction module is used to reconstruct a three-dimensional model corresponding to the three-dimensional scene to be reconstructed according to the target prediction sequence; wherein, the target three-dimensional reconstruction model is trained according to the three-dimensional reconstruction model training device according to any one of claims 11-12.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images