Three-dimensional scene reconstruction, online home furnishing and commodity acquisition method, device and medium

Abstract

Embodiments of the present application provide a three-dimensional scene reconstruction, online home decoration and commodity acquisition method, device and medium. In the embodiments of the present application, based on the image corresponding to the first space object, 2D visual information such as the boundary line and the vanishing point existing in the image is detected, based on the 2D visual information, the constraint relationship existing between the boundary lines and between the vanishing points is combined, 3D structure information corresponding to the first space, that is, a target three-dimensional model, is obtained, in which the real scene information corresponding to the first space object can be embodied, which is beneficial to improve the application effect based on the three-dimensional model, for example, in the home decoration scene, the home decoration collocation effect based on the three-dimensional model can be improved. Moreover, the stability of the three-dimensional model is higher, the interpretability is stronger, and the effect is more robust; at the same time, combined with the constraint relationship between the boundary lines and the vanishing points, the fitting degree between the main body structures of the three-dimensional model generated is higher.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method, device and medium for 3D scene reconstruction, online home decoration and product acquisition. Background Technology

[0002] With the development of internet applications, users can perform various operations online, such as purchasing goods and online home decoration. Taking online home decoration as an example, users can use home decoration apps to select existing 3D model rooms that are most similar to their house layouts, place various furniture or decorations in the 3D model rooms to view the home decoration effect, and then select furniture based on the effect for offline actual home decoration.

[0003] However, existing 3D model rooms are created using 3D modeling technology to produce a three-dimensional house model, lacking information about the user's actual home environment. Therefore, the results are not ideal for home decoration needs that require matching with a real home environment. Thus, there is an urgent need for a solution that can reflect real-world home scene information within the 3D house model to improve the matching effect of online home decoration. Summary of the Invention

[0004] This application provides a method, device, and medium for three-dimensional scene reconstruction, online home decoration, and product acquisition, which are used to represent real three-dimensional scene information in a three-dimensional scene model and improve the application effect based on the three-dimensional scene model, such as the effect of home decoration matching.

[0005] This application provides a three-dimensional scene reconstruction method, comprising: acquiring a target image corresponding to a first spatial object, wherein the first spatial object is at least a portion of spatial objects in a target physical space; detecting multiple boundary lines and at least two vanishing points on orthogonal principal directions in the target image, wherein the boundary lines are the lines of intersection between adjacent physical main structures in the first spatial object; determining the camera intra-parameters of a target camera and the gravity direction in the camera coordinate system based on the vanishing points on the at least two orthogonal principal directions, wherein the target camera refers to the camera used to capture the target image; reconstructing an initial three-dimensional model corresponding to the first spatial object based on the multiple boundary lines and the gravity direction in the camera coordinate system, wherein the initial three-dimensional model includes adjacent model main structures corresponding to the adjacent physical main structures; and optimizing the initial three-dimensional model based on the constraint relationships between the multiple boundary lines, the constraint relationships between the vanishing points on the at least two orthogonal principal directions, and the camera intra-parameters to obtain a target three-dimensional model.

[0006] This application also provides an online home decoration method, including: responding to an image upload operation to obtain a target image corresponding to a first spatial object, wherein the first spatial object is at least a portion of spatial objects in a target physical space; responding to a placement operation of a target home decoration object on the target image to merge the target home decoration object into a target 3D model corresponding to the first spatial object to obtain a target 3D model fused with the target home decoration object; and projecting the target 3D model fused with the target home decoration object onto the target image to obtain a home decoration rendering containing the target home decoration object; wherein the target 3D model is constructed according to the steps in the 3D scene reconstruction method provided in this application.

[0007] This application embodiment also provides a product selection method, including: responding to a selection operation on a product page, determining a target product to be selected, the target product having a three-dimensional product model; responding to a matching effect viewing operation, selecting a target image corresponding to a first spatial object to be matched with the target product; adding the product's three-dimensional model to the target three-dimensional model corresponding to the first spatial object to obtain a target three-dimensional model fused with the target product; projecting the target three-dimensional model fused with the target product onto the target image to obtain a matching effect diagram of the target product and the first spatial object; wherein, the target three-dimensional model is constructed according to the steps in the three-dimensional scene reconstruction method provided in this application embodiment.

[0008] This application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the three-dimensional scene reconstruction method, online home decoration method, or product selection method provided in this application.

[0009] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the three-dimensional scene reconstruction method, online home decoration method, or product selection method provided in this application.

[0010] In this embodiment, based on the image corresponding to the first spatial object, 2D visual information such as boundary lines and disappearance points in the image is detected. Based on this 2D visual information and the constraint relationship between the boundary lines and disappearance points, the 3D structural information corresponding to the first space is obtained, namely the target 3D model. The target 3D model can reflect the real scene information corresponding to the first spatial object, which is conducive to improving the application effect based on the 3D model. For example, in the home decoration scene, it can improve the home decoration matching effect based on the 3D model.

[0011] Furthermore, in this embodiment, 2D visual information such as boundary lines and vanishing points are directly detected from the image, and then 3D structural information is generated using this 2D visual information and related constraint relationships, instead of directly regressing 3D structural information from the image. Therefore, the 3D model generated in this embodiment is more stable and more interpretable, and is more robust to situations with large shooting angle tilt and cluttered scenes. At the same time, combined with the constraint relationships between boundary lines and vanishing points, the fit between the main structures of the generated 3D model is higher. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0013] Figure 1a A schematic diagram of the structure of a three-dimensional scene reconstruction system provided as an exemplary embodiment of this application;

[0014] Figure 1b A flowchart illustrating a three-dimensional scene reconstruction method provided for an exemplary embodiment of this application;

[0015] Figure 2a A schematic diagram of a boundary line in a first spatial object provided for an exemplary embodiment of this application;

[0016] Figure 2b A schematic diagram of the structure of an anonymization point provided for an exemplary embodiment of this application;

[0017] Figure 3a A schematic diagram of the structure of a boundary line detection model based on Hough transform, provided for an exemplary embodiment of this application;

[0018] Figure 3b A schematic diagram of the structure of an annulment point detection model based on Hough transform, provided for an exemplary embodiment of this application;

[0019] Figure 3c A schematic diagram showing the location of the intersection point between boundary lines, provided for an exemplary embodiment of this application;

[0020] Figure 3d A schematic diagram of the main structure of a reference model provided for an exemplary embodiment of this application;

[0021] Figure 3e A schematic diagram of another model main structure provided for an exemplary embodiment of this application;

[0022] Figure 4a A flowchart illustrating an online home decoration method provided for an exemplary embodiment of this application;

[0023] Figure 4b A schematic diagram illustrating a three-dimensional scene reconstruction process provided for an exemplary embodiment of this application;

[0024] Figure 4c An online home decoration effect diagram based on a target 3D model is provided as an exemplary embodiment of this application;

[0025] Figure 4d Another online home decoration effect diagram based on a target 3D model is provided as an exemplary embodiment of this application;

[0026] Figure 4e A flowchart illustrating a product selection method provided for an exemplary embodiment of this application;

[0027] Figure 5 A schematic diagram of the structure of a three-dimensional scene reconstruction device provided for an exemplary embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To address the technical problem of poor home decoration matching effects caused by the lack of real-world home scene information in existing online home decoration scenarios, this application provides a 3D scene reconstruction method. In this method, based on the image corresponding to a first spatial object, 2D visual information such as boundary lines and disappearance points are detected in the image. Based on this 2D visual information and the constraint relationships between boundary lines and disappearance points, the 3D structural information corresponding to the first space is obtained, i.e., the target 3D model. This target 3D model can reflect the real-world scene information corresponding to the first spatial object, which is beneficial to improving the application effect based on the 3D model. For example, in home decoration scenarios, it can improve the home decoration matching effect based on the 3D model; and in online shopping scenarios, the matching effect between the 3D model and the products can improve the shopping experience and reduce the probability of returns and exchanges.

[0031] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0032] Figure 1a This is a schematic diagram of the structure of a three-dimensional scene reconstruction system provided as an exemplary embodiment of this application. Figure 1a As shown, the system includes: a terminal device 10 and a server device 20. The terminal device 10 and the server device 20 are communicatively connected.

[0033] In this embodiment, the terminal device 10 can be a mobile phone, laptop computer, or desktop computer, etc., and the server device 20 can be a physical server, cloud server, or server array, etc. Figure 1a The illustration uses a smartphone as an example of a terminal device 10 and a physical server as an example, but it is not limited to these examples.

[0034] Among them, the terminal device 10 can acquire the target image corresponding to the first spatial object. For example, the terminal device has a built-in camera, and the terminal device can capture the target image in the first spatial object through the built-in camera; or, for example, the target image corresponding to the first spatial object can be captured by a camera independent of the terminal device 10, and the captured target image can be provided to the terminal device 10 by the camera independent of the terminal device 10.

[0035] In this embodiment, the terminal device 10 can provide the target image corresponding to the first spatial object to the server device 20, and the server device 20 can generate a target 3D model corresponding to the first spatial object. The process of the server device 20 generating the target 3D model corresponding to the first spatial object includes: detecting multiple boundary lines and at least two vanishing points in the target image, where the boundary lines are the lines of intersection between adjacent physical main structures in the first spatial object; determining the camera's intra-camera parameters and the gravity direction in the camera coordinate system based on the vanishing points in the at least two orthogonal main directions, where the target camera refers to the camera used to capture the target image; reconstructing an initial 3D model corresponding to the first spatial object based on the multiple boundary lines and the gravity direction in the camera coordinate system, where the initial 3D model includes adjacent model main structures corresponding to adjacent physical main structures; and optimizing the initial 3D model based on the constraint relationships between the multiple boundary lines, the constraint relationships between the vanishing points in the at least two orthogonal main directions, and the camera's intra-camera parameters to obtain the target 3D model. Detailed descriptions of each operation can be found in subsequent embodiments and will not be elaborated here.

[0036] Based on the target 3D model, users can also add home decoration items such as sofas, paintings, office chairs, or desks to the target image on the terminal device 10. The terminal device 10 can obtain the location range information of the target items in the target image and provide the target item information and the location range information of the target items in the target image to the server device 20. According to the location range information of the target items in the target image, the server device 20 merges the 3D model of the target item into the target 3D model corresponding to the first spatial object, and projects the merged target 3D model onto the target image to obtain a target image containing the target items. The server device 20 returns the target image containing the target items to the terminal device 10, which then displays it to the user to realize the online home decoration matching effect display.

[0037] It should be noted that the three-dimensional scene reconstruction method provided in this application embodiment can be applied not only to Figure 1a The 3D scene reconstruction system shown can be completed by the cooperation of terminal devices and server devices, or it can be implemented independently by the terminal devices. In the case of independent implementation by the terminal devices, actions originally performed by the server devices can be performed by the terminal devices in this embodiment. Other aspects are the same as... Figure 1a Similar or identical elements in the illustrated system will not be described further here. For details regarding the 3D scene reconstruction process, please refer to the description in the following method embodiments.

[0038] Figure 1b This is a flowchart illustrating a three-dimensional scene reconstruction method provided as an exemplary embodiment of this application. Figure 1b As shown, the method includes:

[0039] 101. Obtain the target image corresponding to the first spatial object, where the first spatial object is at least a portion of the spatial objects in the target physical space;

[0040] 102. Detect multiple boundary lines and at least two disappearing points in orthogonal principal directions in the target image. The boundary lines are the intersection lines between adjacent physical main structures in the first spatial object.

[0041] 103. Based on at least two vanishing points along the orthogonal principal directions, determine the camera's intrinsic parameters and the direction of gravity in the camera coordinate system. The target camera refers to the camera used to capture the target image.

[0042] 104. Based on multiple boundary lines and the direction of gravity in the camera coordinate system, reconstruct the initial three-dimensional model corresponding to the first spatial object. The initial three-dimensional model includes the adjacent model main structure corresponding to the adjacent physical main structure.

[0043] 105. Based on the constraint relationships between multiple boundary lines, the constraint relationships between the disappearance points on at least two orthogonal principal directions, and the camera's intrinsic parameters, optimize the initial 3D model to obtain the target 3D model.

[0044] In this embodiment, the target physical space refers to a spatial area in a specific scenario. For example, the target physical space can be a shopping mall, supermarket, airport, or house, or other areas with spatial concepts. The target physical space contains at least one spatial object; in other words, at least one spatial object constitutes the target physical space. For example, the target physical space can be a physical house, which includes multiple spatial objects such as a kitchen, bedroom, living room, and bathroom. Another example is a shopping mall, which includes multiple floors with different merchants on each floor; each merchant can be considered a spatial object. For ease of distinction and description, at least some spatial objects in the target physical space are referred to as the first spatial object. For example, the first spatial object can be one spatial object in the target physical space, or it can be multiple spatial objects in the target physical space, such as two or three; yet another example is that the first spatial object can be a partial space within a certain spatial object. Taking a physical house as the target physical space as an example, the first spatial object can be the entire physical house, or it can be an independent spatial object such as the master bedroom, secondary bedroom or living room, or it can be a part of an independent spatial object such as the master bedroom, secondary bedroom or living room, or it can be a part of multiple independent spatial objects at the same time, such as a part of the living room space and a part of the balcony space at the same time.

[0045] In this embodiment, a target image corresponding to the first spatial object can be obtained. For example, an image of the first spatial object can be captured by a camera, and the target image corresponding to the first spatial object can be obtained from the camera. Alternatively, a camera can be installed on the terminal device, and the target image corresponding to the first spatial object can be captured by the camera on the terminal device. The target image is an environmental image of the first spatial object, containing real environmental information about the first spatial object.

[0046] In this embodiment, the target image contains multiple boundary lines. These boundary lines are the lines of intersection between adjacent physical structures within the first spatial object. The physical structures may include, but are not limited to, walls, ceilings, or floors. Specifically, the line of intersection between walls can be called a wall line, the line of intersection between a wall and the floor can be called a ground line, and the line of intersection between a wall and the ceiling can be called a ceiling line. Figure 2a As shown in the figure, the first space object is a part of the living room. The first space object includes: ceiling, floor and walls. The extracted floor line is C1, the extracted wall line is C2 and the extracted ceiling line is C3. Figure 2aThe ground line C1, wall line C2, and ceiling line C3 are examples of boundary lines in the embodiments of this application, but are not limited thereto.

[0047] In this embodiment, the first spatial object can conform to a Manhattan structure, which includes three mutually orthogonal principal directions. For example, the three orthogonal principal directions are the gravity direction, the direction facing the main wall, and the direction facing the side wall. The gravity direction can be considered as the direction perpendicular to the ground or ceiling. The main wall refers to the wall with the smallest angle to the optical axis of the target camera (the camera used to capture the target image), and the optical axis of the target camera is the line passing through the center of the lens. The direction facing the main wall is perpendicular to the main wall, i.e., the direction of the normal vector of the main wall. The side wall is relative to the main wall and mainly refers to the wall adjacent to the main wall. The direction facing the side wall is perpendicular to the side wall. Typically, there can be multiple side walls in the target image, for example, two. These multiple side walls are opposite or parallel, and the directions corresponding to the multiple side walls are the same, i.e., the direction of the normal vector of the side wall. This explains that the target image may simultaneously contain physical structures corresponding to three orthogonal principal directions, such as the main wall, side walls, and the floor or ceiling; or it may only contain physical structures corresponding to two orthogonal principal directions, such as only the main wall and floor, only the side walls and ceiling, or only the main wall, ceiling, and floor. Whether the target image contains physical structures corresponding to three or two orthogonal principal directions depends on the shooting angle of the target image. Simply put, the target image must contain at least two physical structures corresponding to two orthogonal principal directions.

[0048] In this embodiment, after perspective transformation, parallel lines in three-dimensional space intersect at a single point in the corresponding two-dimensional image. This point is called a vanishing point (VP), and the direction of the vanishing point represents the direction of the parallel lines in three-dimensional space that intersect at that point. Multiple vanishing points exist in the target image, including those located along at least two orthogonal principal directions. For a vanishing point located along a certain orthogonal principal direction, the parallel lines in three-dimensional space that intersect at that point can be perpendicular to the physical structure in three-dimensional space corresponding to that orthogonal principal direction. The physical structures in three-dimensional space conform to the Manhattan structure, and each physical structure corresponds to a principal direction. That is, the direction of a vanishing point in the target image located along a certain orthogonal principal direction corresponds to a principal direction in three-dimensional space; specifically, the direction of a vanishing point along a certain orthogonal principal direction represents the direction of the normal vector of a certain physical structure. The presence of at least two physical structures corresponding to orthogonal principal directions in the target image means that the target image contains vanishing points along at least two orthogonal principal directions. Figure 2b As shown, line 1 is perpendicular to the right wall (i.e. Figure 2b The direction of the vanishing point D1 obtained by the intersection of line 1 and line 1 (which is parallel to the wall where the painting is hung) represents the left side of the wall (i.e., the wall where the painting is hung). Figure 2b The direction of the normal vector of the wall with the window in the middle; line 2 and the left wall (i.e., Figure 2b The direction of the vanishing point D2, obtained by the intersection of line 2 and line 2 (parallel to the wall with the window), represents the direction of the right wall (i.e., the wall with the window in the middle). Figure 2b The direction of the normal vector of the wall on which the painting is hung; line 3 is parallel to the ground (or ceiling), and the direction of the disappearance point D3 obtained by the intersection of line 3 represents the direction of the normal vector of the ground (or ceiling), that is, the direction parallel to the direction of gravity.

[0049] In this embodiment, multiple boundary lines and at least two disappearance points in orthogonal principal directions can be detected in the target image. Furthermore, when detecting boundary lines and disappearance points, there is no need to segment the target image; detection is performed directly on the target image, thus avoiding interference from image segmentation. Therefore, the accuracy and precision of the detected boundary lines and disappearance points are both high.

[0050] In this embodiment, the position of the annihilation point in the target image is determined by the intrinsic parameters of the target camera and the 3D direction corresponding to the annihilation point (i.e., the principal direction where the annihilation point is located). The intrinsic parameters of the target camera are parameters related to the characteristics of the target camera itself, such as the focal length, pixel size, and principal point position (i.e., the position of the camera's optical center). Conversely, given that at least two annihilation points in the target image orthogonal principal directions are known, the camera's intrinsic parameters and the direction of gravity in the camera coordinate system can be determined based on these at least two orthogonal principal directions. Specifically, the camera's intrinsic parameters can be determined based on the annihilation points in the at least two orthogonal principal directions, and the direction of gravity in the camera coordinate system can be determined based on the camera's intrinsic parameters and the annihilation point in the direction of gravity.

[0051] Based on the gravity direction in the camera coordinate system and the multiple boundary lines in the target image, an initial 3D model corresponding to the first spatial object can be reconstructed according to these boundary lines and the gravity direction in the camera coordinate system. This initial 3D model includes adjacent model main structures corresponding to adjacent physical main structures. The physical main structure is the main structure constituting the first spatial object, and the model main structure is the main structure constituting the initial 3D model; the model main structure is the manifestation of the physical main structure in the initial 3D model. Specifically, the initial 3D model corresponding to the first spatial object is constructed based on the planar equations of each physical main structure. For example, a 3D model of a house can be constructed based on the planar equations of each wall, floor, or ceiling contained within the house. The planar equation of a physical main structure can uniquely represent that physical main structure. Optionally, it can be represented by the normal vector of the physical main structure and its distance to the camera's optical center, but it is not limited to this. After obtaining the initial 3D model, the model main structure in this initial 3D model also has planar equations. Theoretically, the planar equation of the model main structure should have the same planar direction as its corresponding physical main structure.

[0052] The general principle of reconstructing the initial 3D model corresponding to the first spatial object based on multiple boundary lines in the target image and the gravity direction in the camera coordinate system is as follows: Based on the gravity direction in the camera coordinate system, a ground plane in 3D space is constructed, which is perpendicular to the gravity direction in the camera coordinate system; Based on the multiple boundary lines in the target image, the boundary formed by the first spatial object on the ground plane and the main structure of the model on the boundary can be determined; Further, the height of the camera optical center and the ground can be assumed, i.e., the camera height. Based on the assumed camera height and the pre-assumed scaling ratio, the height of the main structure of the model (such as the ceiling) is determined, thereby obtaining the initial 3D model corresponding to the first spatial object.

[0053] In this embodiment, the initial 3D model corresponding to the first spatial object is constructed based on the boundary lines in the target image. When there are errors in the boundary lines, the perspective relationships in the initial 3D model will be incorrect. For example, there will be no perpendicular relationship between adjacent walls. Furthermore, assuming the camera height and scaling ratio, the generated wall height will also be inaccurate. Based on this, in this embodiment, after obtaining the initial 3D model, the initial 3D model will be optimized according to the constraint relationships between multiple boundary lines and the constraint relationships between at least two orthogonal principal directions, combined with the camera's intra-parameters, to obtain the target 3D model. The constraint relationships between multiple boundary lines are as follows: adjacent physical structures forming the boundary lines are perpendicular to each other. This constraint relationship can also be called the Manhattan constraint, that is, walls, floors, and ceilings that are adjacent in the first spatial object are perpendicular to each other. The constraint relationships between at least two orthogonal principal directions are as follows: the physical structures corresponding to any two orthogonal principal directions are perpendicular to each other, and the normal vector direction of the physical structure corresponding to the same orthogonal principal direction is parallel to the direction of the parallel line represented by the orthogonal principal direction. Based on these constraints, the normal vector direction and distance to the camera optical center of the main structure of each initial 3D model can be optimized to obtain a main structure of the model whose positional relationship, height, etc. are highly consistent with the physical main structure.

[0054] In this embodiment, based on the image corresponding to the first spatial object, 2D visual information such as boundary lines and vanishing points are detected in the image. Based on this 2D visual information and the constraint relationships between boundary lines and vanishing points, 3D structural information corresponding to the first space is obtained, i.e., the target 3D model. This target 3D model can reflect the real scene information corresponding to the first spatial object, which is beneficial to improving the application effect based on the 3D model. For example, in a home decoration scene, it can improve the home decoration matching effect based on the 3D model. Furthermore, in this embodiment, 2D visual information such as boundary lines and vanishing points are directly detected from the image, and then 3D structural information is generated using this 2D visual information and related constraint relationships, rather than directly regressing 3D structural information from the image. Therefore, the target 3D model generated in this embodiment has higher stability and stronger interpretability, and is more robust to situations with large shooting angle tilt and cluttered scenes. At the same time, combined with the constraint relationships between boundary lines and vanishing points, the fit between the main structures of the generated target 3D model is higher.

[0055] In an optional embodiment of this application, when detecting boundary lines and disappearance points as described above, multiple boundary lines and at least two disappearance points in orthogonal principal directions in the target image can be detected based on the Hough transform. The Hough transform is a feature extraction technique widely used in image analysis, computer vision, and digital image processing. In this embodiment, the Hough transform can be used to detect multiple boundary lines in the target image, such as ground lines, wall lines, and ceiling lines. Simultaneously, the Hough transform can also be used to detect at least two disappearance points in orthogonal principal directions within the target image.

[0056] Optionally, when detecting multiple boundary lines and at least two vanishing points along orthogonal principal directions in a target image based on Hough transform, a deep neural network model based on Hough transform can be used to detect these boundary lines and vanishing points. Optionally, the deep neural network model based on Hough transform includes a boundary line detection model and a vanishing point detection model. The boundary line detection model based on Hough transform can be any neural network model capable of detecting boundary lines from an image, and the vanishing point detection model based on Hough transform can be any neural network model capable of detecting vanishing points from an image. Based on this, on the one hand, the target image can be input into the boundary line detection model based on Hough transform to detect boundary lines and obtain multiple boundary lines present in the target image; on the other hand, the target image can be input into the vanishing point detection model based on Hough transform to detect vanishing points and obtain vanishing points along at least two orthogonal principal directions in the target image. The architecture and detection process of the neural network models for boundary line detection and vanishing point detection are illustrated below.

[0057] Boundary line detection based on Hough transform:

[0058] In an alternative embodiment, such as Figure 3aAs shown, the boundary detection model based on Hough transform includes: a first feature extraction network, which is a neural network incorporating skip connections and multi-scale features. Multi-scale features refer to the technique of extracting feature maps at multiple different scales from a target image. Feature maps at different scales contain different feature information; the smaller the feature map size, the greater the depth, belonging to deep network features; conversely, the larger the feature map size, the smaller the depth, belonging to shallow network features. Deep network features have a larger receptive field, lower feature map resolution, strong semantic information representation ability, and weaker geometric detail representation ability; shallow network features have a smaller receptive field, higher feature map resolution, weaker semantic information representation ability, and strong geometric detail representation ability. The receptive field is the size of the region mapped on the input image by the pixels of the feature map output by each layer of the convolutional neural network. Skip connections are a simple and effective operation for fusing deep and shallow network features, allowing element-wise addition of shallow network features with deep network features of the same scale. Based on this, the target image can be input into a boundary detection model based on Hough transform. A first feature extraction network, which integrates skip connections and multi-scale features, is then used to extract features from the target image, resulting in feature maps at multiple scales. For ease of description and distinction, the multi-scale feature maps finally output by this first feature extraction network are referred to as the multi-scale first target feature maps.

[0059] Among them, the first target feature maps at multiple scales belong to the features in the image pixel space. To facilitate boundary line extraction, the features in the image pixel space can be converted into features in the Hough space. The features in the Hough space are then binary-classified, that is, distinguishing whether the features in the Hough space correspond to the boundary lines in the image pixel space. Based on this, as... Figure 3a As shown, the boundary line detection model based on Hough transform also includes a Hough transform network. Based on this Hough transform network, a polar coordinate-based Hough transform can be used to perform a Hough transform on the first target feature map at multiple scales, mapping straight lines in the target image to points in Hough space. The transformed points are then binary-classified in Hough space, for example, distinguishing which points in Hough space are points formed by boundary lines in the image pixel space and which are not.

[0060] In this context, the first target feature map at any scale belongs to the image space, meaning it uses a pixel coordinate system. The pixel coordinate system has the center of the target image as the origin, the horizontal x-axis, and the vertical y-axis, with a unit length of pixels. Unlike the pixel coordinate system, Hough space uses a polar coordinate system. In polar coordinates, a line in the target image can be represented by (ρ, θ), where ρ represents the distance from the end of the line to the origin of the pixel coordinate system, and θ represents the angle between the line and the x-axis of the pixel coordinate system. The coordinates of the first target feature map in Hough space are (ρ, θ). Therefore, representing a line in the target image using polar coordinates transforms it into a point. All straight lines in the target image are mapped to points in Hough space. These points include those mapped from boundary lines (matching the boundary line features) and those mapped from non-boundary lines (not matching the boundary line features). For ease of distinction and description, points in Hough space that match the boundary line features are called target points. Based on this, multiple target points matching the boundary line features can be selected in the Hough space, and these target points can be remapped into the image space to obtain multiple boundary lines existing in the target image. For example... Figure 3a As shown, multiple target points can be remapped into the image space using the Reverse Hough transform (RHT) to obtain multiple boundary lines existing in the target image.

[0061] Further optional, such as Figure 3a As shown, the first target feature extraction network includes multiple downsampling modules, multiple upsampling modules, and skip connection modules. In an optional embodiment, feature extraction is performed on the target image to obtain an initial feature map. This initial feature map is used as the first intermediate feature map at the largest scale. The downsampling modules perform N downsampling operations on the first intermediate feature map at the largest scale to obtain first intermediate feature maps at other scales, where N is a positive integer, for example, N can be 2, 3, 4, or 6, etc. The first intermediate feature map at the smallest scale is used as the first target feature map at the smallest scale. The upsampling modules perform N upsampling operations on the first target feature map at the smallest scale. In each upsampling operation, skip connections are made between the first intermediate feature map at the same scale obtained from the downsampling operation to obtain first target feature maps at other scales.

[0062] like Figure 3aAs shown, assuming the scale of the first intermediate feature map with the largest scale is 512*512*3, and N=3, after three downsampling processes, first intermediate feature maps with scales of 256*256*3, 128*128*3, and 64*64*3 can be obtained. Then, the first intermediate feature map with a scale of 64*64*3 is used as the first target feature map with the smallest scale. The first target feature map is then subjected to three upsampling processes. Specifically, after one upsampling process, a second intermediate feature map with a scale of 128*128*3 can be obtained. This second intermediate feature map is then connected to the first intermediate feature map with a scale of 128*128*3 to obtain the first target feature map with a scale of 128*128*3. This process continues... The first target feature map with a size of 128*128*3 is upsampled once to obtain a second intermediate feature map with a size of 256*256*3. The second intermediate feature map with a size of 256*256*3 and the first intermediate feature map with a size of 256*256*3 are then connected by a skip connection to obtain the first target feature map with a size of 256*256*3. The first target feature map with a size of 256*256*3 is then upsampled once to obtain a second intermediate feature map with a size of 512*512*3. The second intermediate feature map with a size of 512*512*3 and the first intermediate feature map with a size of 512*512*3 are then connected by a skip connection to obtain the first target feature map with a size of 512*512*3.

[0063] Further as Figure 3a As shown, the Hough transform network includes a Hough transform module and a feature fusion module. Based on this, an implementation method using polar coordinate-based Hough transform to perform Hough transform on first target feature maps at multiple scales, mapping straight lines in the target image to points in Hough space, includes: performing Hough transform on first target feature maps X at multiple scales using the Hough transform module to obtain second target feature maps Y at multiple scales in polar coordinate-based Hough space, wherein Hough transform can be performed separately on the first target feature map X at each scale to obtain the second target feature map Y in polar coordinate-based Hough space at that scale; performing scale transformation on the second target feature maps at multiple scales using the feature fusion module to obtain multiple feature maps of the same scale, and concatenating the multiple feature maps of the same scale to obtain a third target feature map Z; performing convolutional dimensionality reduction on the third target feature map to obtain a two-dimensional image in Hough space, the two-dimensional image using polar coordinates, the two-dimensional image containing multiple points, each point having coordinates (ρ, θ), and each point corresponding to a straight line in the target image.

[0064] In this embodiment, the scale of the third target feature map is not limited. For example, the scale of the third target feature map can be any one of the multiple scales corresponding to the second target feature map. Preferably, the scale of the third target feature map is the largest scale among the multiple scales corresponding to the second target feature map. Based on this, when performing scale transformation on the second target feature map of multiple scales, the second target feature map of non-maximum scale can be upsampled to make its size the largest scale.

[0065] like Figure 3a As shown, the first target feature map, the second target feature map, the third target feature map, and the two-dimensional image in Hough space are displayed. Furthermore, in this embodiment, the Hough transform is performed based on the depth feature map, hence it is called the Depth Hough transform (DHT). Figure 3a In the diagram, the feature fusion module is represented by a circled 'c'. This feature fusion module further includes an upsampling unit and a concat unit. The upsampling unit is used to scale the second target feature maps at multiple scales to obtain multiple feature maps of the same scale. The concat unit is used to concatenate the multiple feature maps of the same scale to obtain the third target feature map Z.

[0066] It should be noted that before using the boundary detection model based on Hough transform, the model can be trained in advance to obtain the boundary detection model based on Hough transform. In this embodiment, a novel combination of deep Hough transform and neural network model is applied to the detection of boundary lines in images. Specifically, a large number of images of spatial objects (such as indoor scenes) are acquired, and wall lines, ground lines, and ceiling lines in the images are labeled to obtain a sample dataset. This sample dataset is then used to train a basic neural network based on Hough transform. This basic neural network includes a first feature extraction network and a Hough transform network. The first feature extraction network includes an upsampling module, a downsampling module, and a skip connection module to extract features from the sample images in pixel space to obtain multi-scale sample feature maps. The Hough transform network includes a Hough transform module and a feature fusion module to transform the multi-scale sample feature maps into Hough space and obtain sample images in Hough space. Then, a binary classification method is used to classify the sample images in Hough space, specifically dividing the points in the sample images into a first class of points corresponding to boundary lines and a second class of points corresponding to other straight lines. Finally, a loss function is generated based on the labeling and classification results. Optionally, a Binary CrossEntropy loss function can be used. The loss function (BCELoss) is used to continue training even if the loss function is not up to standard, until the loss function meets the standard or the model training time or number of iterations reaches the set time or number of iterations, thus obtaining the boundary detection model based on Hough transform. It should be noted that... Figure 3a The section demonstrating the loss function calculation based on the annotation and classification results is used in the model training phase only; additionally, in Figure 3a The paper also demonstrates the process of remapping multiple target points into the image space using the inverse Hough transform to obtain multiple boundary lines existing in the target image. This process is used in the model inference stage.

[0067] Hough transform-based disappearance point detection:

[0068] In an alternative embodiment, such as Figure 3bAs shown, the Hough transform-based disappearance point detection model includes a second feature extraction network. This second feature extraction network can be any network capable of feature extraction, such as a backbone network in image classification, like UNet or the Stacked Hourglass model. UNet is a variant of Fully Convolutional Networks (FCNs) with a symmetrical network structure resembling the letter U, hence its name. The Stacked Hourglass model is a network structure that utilizes multi-scale features to identify pose. The target image can be input into the Hough transform-based disappearance point detection model, and the second feature extraction network extracts features from the target image to obtain a fourth target feature map. Figure 3b As shown, taking a target image of [512x512]x3 as an example, where [512x512] is the scale of the target image and 3 is the number of channels; and taking a fourth target feature map of [128x128]x3 as an example, but not limited to this. The fourth target feature map can be mapped to a Gaussian spherical space. For example, projecting straight lines in the target image onto a Gaussian spherical space centered at the target camera's center, in the Gaussian spherical space, straight lines intersecting at the same point in the target image will have the strongest response at that point. Therefore, the annihilation points located on at least two orthogonal principal directions can be obtained based on the response values ​​and angles of each point in the Gaussian spherical space. Further, as... Figure 3b As shown, the Hough transform-based disappearance point detection model also includes a Gaussian spherical Hough transform network. Based on this, the fourth target feature map is fed into the Gaussian spherical Hough transform network. In this network, the fourth target feature map is subjected to a Hough transform using a Gaussian spherical Hough transform to map the straight lines in the target image to points in Gaussian spherical space. Then, at least two points in Gaussian spherical space that satisfy the probability and angle requirements are selected as disappearance points along at least two orthogonal principal directions, and remapped into the image space to obtain disappearance points along at least two orthogonal principal directions in the target image. The Gaussian sphere space includes multiple points, each derived from a straight line in the target image. Each point has two attributes: a brightness value and an angle value. The brightness value represents the probability that the corresponding point is a point obtained by direct intersection (i.e., an annulment point). The greater the brightness of the corresponding point on the Gaussian sphere, the greater the probability that the point is an annulment point. The angle value represents the angle between the line corresponding to the point and the x-axis in the image space. Based on this angle value, it can be determined whether the lines corresponding to each point on the Gaussian sphere are perpendicular to each other. Therefore, at least two points in the Gaussian sphere space that meet the requirements for probability and angle values ​​can be selected as annulment points along at least two orthogonal principal directions.

[0069] Optionally, the Hough transform network based on the Gaussian sphere includes a Hough transform module and a Gaussian sphere transform module. Based on this, after extracting features from the target image to obtain the fourth target feature map, the fourth target feature map can be transformed sequentially to Hough space and Gaussian sphere space. In Hough space, straight lines in the target image are mapped to a point (referred to as a Hough point), but it is impossible to directly distinguish whether a Hough point is formed by the intersection of multiple straight lines. Furthermore, the Hough points are mapped to Gaussian sphere space. In Gaussian sphere space, the response value (i.e., brightness value) of each point (referred to as a Gaussian point) varies depending on the number of straight lines corresponding to the Gaussian point. The response value of a Gaussian point formed by a single straight line is less than that formed by the intersection of multiple straight lines. Therefore, the Gaussian points with the strongest response values ​​can be selected as cancellation points. Specifically, the fourth target feature map is input into the Hough transform module. Inside this module, the fourth target feature map is subjected to a Hough transform to obtain the fifth target feature map in the polar coordinate-based Hough space, such as... Figure 3b As shown, the illustration uses the fifth target feature map with dimensions of [184x180]x128 as an example, but it is not limited to this. Here, 184 is the maximum value in the distance dimension represented by ρ, 180 is 180° in the θ angle dimension, and 128 is the number of channels. Figure 3b In this context, HT denotes the Hough space, and (ρ, θ) are the coordinates in the Hough space. Further, as... Figure 3b As shown, a Hough convolution is performed on the fifth target feature map in the Hough space to obtain the sixth target feature map. The sixth target feature map retains the same dimension as the fifth target feature map. Figure 3b In this context, the Hough convolution is represented as HT Conv. Then, as... Figure 3b As shown, the sixth target feature map is input into the Gaussian spherical transformation module. Inside this module, a spherical Gaussian spherical transformation is performed on the sixth target feature map to obtain the seventh target feature map in Gaussian spherical space. Figure 3b In this diagram, (α, β) are variables in the Gaussian spherical space. Taking the seventh target feature map with dimensions

[32768] x 128 as an example, 128 represents the number of channels.

[32768] indicates that the Gaussian sphere is discretized into 32768 points. Through this spherical Gaussian transformation, the points in the Hough space can be determined to correspond to discrete points in the Gaussian spherical space. A spherical convolution is performed on the seventh target feature map in the Gaussian spherical space to obtain the eighth target feature map. The eighth target feature map includes multiple Gaussian points, each corresponding to a straight line in the target image. Figure 3b The diagram uses spherical convolution (ShpericalConv) as an example, with the eighth target feature map having dimensions of

[32768] x128, but it is not limited to this.

[0070] Optionally, based on the probability values ​​of each Gaussian point in the eighth target feature map, points whose probability values ​​meet set requirements are selected as extinction points from multiple Gaussian points. For example, points whose probability values ​​exceed a set probability threshold are selected as extinction points, such as 80%, 90%, or 95%. At least two extinction points with angles greater than a set angle are selected from the extinction points as extinction points along at least two orthogonal principal directions. This can be explained by selecting the two or three extinction points with the largest angles as extinction points along at least two orthogonal principal directions. Figure 3b The Gaussian sphere shown is illustrated using three points of elimination as an example.

[0071] In this embodiment, determining the camera intrinsic parameters of the target camera and the gravity direction in the camera coordinate system based on at least two orthogonal principal directions includes: selecting two target extinction points from the at least two finite extinction points when the extinction points in the at least two orthogonal principal directions include at least two finite extinction points; determining the camera intrinsic parameters of the target camera based on the constraint relationship between the two target extinction points and the camera intrinsic parameters; and transforming the extinction points located in the gravity direction in the at least two orthogonal principal directions into the camera coordinate system based on the camera intrinsic parameters to obtain the gravity direction in the camera coordinate system.

[0072] When the target camera acquires a target image, two parallel straight lines in three-dimensional space intersect in the target image after perspective transformation, forming vanishing points. These vanishing points may or may not intersect at infinity. In this embodiment, infinity is defined in a certain way. For example, if the distance from the pixel coordinate of the vanishing point to the origin of the pixel coordinate system is greater than a set multiple (such as 10 times) of the diagonal length of the target image, the vanishing point is considered to intersect at infinity. For ease of distinction and description, vanishing points intersecting at infinity are called infinite vanishing points, and vanishing points not intersecting at infinity are called finite vanishing points. Based on this, the vanishing points in the at least two orthogonal principal directions in this embodiment may include infinite vanishing points or finite vanishing points. Specifically, when the vanishing points in the at least two orthogonal principal directions include at least two finite vanishing points, two vanishing points are selected as the target vanishing points. The method of selecting two target annihilation points from at least two finite annihilation points is not limited. For example, two annihilation points can be randomly selected from at least two finite annihilation points as target annihilation points; or, for example, the two annihilation points closest to the camera's optical center can be selected from at least two finite annihilation points as target annihilation points. Regardless of which two target annihilation points are selected, these two target annihilation points may or may not include annihilation points located in the direction of gravity, and this is not limited.

[0073] After selecting two target extinction points, the camera's intra-camera parameters can be determined based on the constraint relationship between these two points and the camera's intra-camera parameters. The constraint relationship between the two target extinction points and the camera's intra-camera parameters is: (K... -1 VP1)·(K -1 VP2) = 0, this constraint means that the dot product of the corresponding straight lines of the two target annihilation points in 3D space is 0. Simply put, the corresponding straight lines of the two target annihilation points in 3D space are perpendicular to each other. VP1: represents the 3D vector of the first target annihilation point. This 3D vector is formed by adding one dimension to the image coordinates of the first target annihilation point. For example, adding 1 to the image coordinates can form the 3D vector of the first target annihilation point. K: represents the intrinsic parameters of the target camera, a 3x3 matrix including the focal length and principal point position. The principal point position is the optical center position of the camera. For example, the position at image coordinates (0.5, 0.5) can be taken as the optical center position, but it is not limited to this. The focal length is to be solved, and the principal point position is known. -1 : Represents the inverse of the camera's intrinsic parameters; VP2: is the three-dimensional vector of the second target vanishing point. This three-dimensional vector is formed by adding one dimension to the image coordinates of the second target vanishing point. For example, 1 can be added to the image coordinates to form the three-dimensional vector of the second target vanishing point.

[0074] Since VP1 and VP2 are known, the camera's intrinsic parameters can be obtained by solving for the constraints between the two target disappearance points and the camera's intrinsic parameters. Then, using the camera's intrinsic parameters and the disappearance points located in the direction of gravity, the gravity direction *dir* in the camera coordinate system can be calculated. y = K -1 VP y Where y represents the direction of gravity, and VP y dir represents a three-dimensional vector representing the point of disappearance in the image space along the direction of gravity. y This indicates the direction of gravity in the camera coordinate system.

[0075] Furthermore, the vanishing point constrains the perspective relationship in the corresponding principal direction. Assuming the normal direction of a physical structural element (such as a wall) is N... i N i For the unknown quantity, the extinction point corresponding to this physical entity structure is VP. i Then, it is necessary to satisfy the constraint that the normal direction of the physical main structure and the principal direction of the parallel line that intersects to obtain the disappearance point are parallel to each other, that is, N i · (K) -1 VP i=1.

[0076] In an optional embodiment, when reconstructing the initial three-dimensional model corresponding to the first spatial object, the reference physical main structure contained in the first spatial object can be determined. The reference physical main structure can be the ground or the ceiling, depending on the physical main structure contained in the target image. For example, if the target image contains the ground, the ground is used as the reference physical main structure; if the target image does not contain the ground but contains the ceiling, the ceiling is used as the reference physical main structure. Based on the established reference physical structure, a reference plane in three-dimensional space is constructed in the camera coordinate system according to the direction of gravity. This reference plane corresponds to the reference physical structure. Multiple reference boundary lines intersecting with the reference physical structure are identified from among multiple boundary lines. If the reference physical structure is the ground, these multiple boundary lines can be multiple ground lines obtained by the intersection of multiple walls with the ground; if the reference physical structure is a ceiling, these multiple boundary lines can be multiple ceiling lines obtained by the intersection of multiple walls with the ceiling. Based on the intersection points of these multiple boundary lines, a reference model structure corresponding to the reference physical structure is constructed on the reference plane. Based on the preset height from the camera optical center to the reference plane and the intersection points of the multiple boundary lines, other model structures corresponding to other physical structures are constructed on the reference model structure to obtain the initial three-dimensional model corresponding to the first spatial object. If the reference physical structure is the ground, the other physical structures can be walls, ceilings, etc. Figure 3e The diagram shows an exemplary representation of the main structure of the baseline model and other main structures of models. For example, in... Figure 3e In this context, the main structure of the baseline model refers to the ground in the initial 3D model, while the main structure of other models refers to the walls in the initial 3D model.

[0077] Optionally, one implementation method for constructing a reference model body structure corresponding to the reference physical body structure on the reference plane based on the intersection positions between multiple reference boundary lines includes: selecting valid reference boundary lines from multiple reference boundary lines, wherein there are multiple valid reference boundary lines; sorting the valid reference boundary lines according to the angle between the valid reference boundary lines and the x-axis in the image coordinate system to obtain the adjacency relationship between the valid reference boundary lines; determining the intersection positions between the valid reference boundary lines based on the adjacency relationship between the valid reference boundary lines; and dividing the intersection positions into first intersection positions that intersect with the boundary of the target image and second intersection positions that do not intersect with the boundary of the target image. Figure 3c The example illustrates the positions of the first and second intersection points, but is not limited to these. Based on the positions of the first and second intersection points, the model boundary corresponding to the benchmark physical main structure is drawn on the benchmark plane to obtain the benchmark model main structure, such as... Figure 3dAs shown. In Figure 3d In the diagram, the mesh region serves as the reference plane, and the white areas within the mesh region represent the reference physical main structure, formed by the model boundary corresponding to the reference model main structure. This continues from... Figure 3c The intersection point shown is... Figure 3d The main structure of the baseline model is the ground formed by the white area.

[0078] Alternatively, the embodiments of this application do not limit the method of selecting a valid reference boundary line from multiple reference boundary lines, as illustrated below.

[0079] Example B1: Based on the angles between multiple reference boundary lines and the x-axis in the image coordinate system, a first reference boundary line is selected from these reference boundary lines. For example, the reference boundary line with an angle less than a set angle threshold is selected as the first reference boundary line. The angle threshold can be 10 degrees, 15 degrees, or 20 degrees, etc. Furthermore, if there are multiple boundary lines with angles less than the angle threshold, the boundary line with the smallest angle can be selected as the first reference boundary line. Alternatively, a boundary line can be randomly selected from the boundary lines with angles less than the angle threshold as the first reference boundary line. Then, some falsely detected reference boundary lines are eliminated based on the first reference boundary line. Specifically, based on the angles between other reference boundary lines and the first reference boundary line, reference boundary lines with angles less than a set angle threshold are eliminated. The angle threshold can be 3 degrees, 5 degrees, or 10 degrees, etc. The remaining reference boundary lines and the first reference boundary line are considered as valid reference boundary lines.

[0080] Example B2: Based on the lengths of multiple reference boundary lines, a first reference boundary line is selected from them. For example, a reference boundary line with a length greater than a set length threshold is selected from multiple reference boundary lines as the first reference boundary line. The length threshold is not limited and depends on the size of the target image. Then, some falsely detected reference boundary lines are removed based on the first reference boundary line. Specifically, based on the angle between other reference boundary lines and the first reference boundary line, reference boundary lines with an angle less than a set angle threshold are removed. The reference boundary lines that are not removed and the first reference boundary line are taken as valid reference boundary lines.

[0081] Example B3:Based on the angles between multiple reference boundary lines and the x-axis in the image coordinate system, and the lengths of the multiple reference boundary lines, a first reference boundary line is selected from the multiple reference boundary lines. For example, candidate reference boundary lines with angles less than a set angle threshold are selected from the multiple reference boundary lines, and reference boundary lines with lengths greater than a set length threshold are selected from the candidate reference boundary lines as the first reference boundary line. Based on the angles between other reference boundary lines and the first reference boundary line, reference boundary lines with angles less than a set angle threshold are eliminated, and the reference boundary lines that are not eliminated and the first reference boundary line are taken as valid reference boundary lines.

[0082] Optionally, when the intersection points between effective reference boundary lines are divided into first intersection points and second intersection points, one implementation method involves constructing other model main structures corresponding to other physical main structures on the reference model main structure based on a preset height from the camera optical center to the reference plane and the intersection points between multiple reference boundary lines to obtain an initial 3D model corresponding to the first spatial object. This includes: determining adjacent model boundaries intersecting at the second intersection point on the reference model main structure based on the second intersection point; determining the initial height of other model main structures based on a preset height from the camera optical center to the reference plane and a preset scaling ratio; and constructing other model main structures on the adjacent model boundaries intersecting at the second intersection point based on the initial height of the other model main structures to obtain an initial 3D model corresponding to the first spatial object. For example, if the reference physical main structure is the ground, constructing other model main structures on the adjacent model boundaries intersecting at the second intersection point specifically involves constructing walls on the adjacent model boundaries intersecting at the second intersection point, and further, adding a ceiling above the adjacent walls to obtain an initial 3D model corresponding to the first spatial object.

[0083] In one optional embodiment, the above-mentioned optimization of the initial 3D model based on the constraint relationships between multiple boundary lines, the constraint relationships between the vanishing points in at least two orthogonal principal directions, and the camera's intra-camera parameters to obtain the target 3D model includes: constructing an optimization function with the position parameters and / or height parameters of each model's main structure as optimization variables based on the constraint relationships between multiple boundary lines, the constraint relationships between the vanishing points in at least two orthogonal principal directions, and the camera's intra-camera parameters; solving the optimization function using a least squares algorithm to obtain the optimized position parameters and / or height parameters of each model's main structure; and adjusting the position of each model's main structure according to the optimized position parameters and / or height parameters to obtain the target 3D model.

[0084] In this embodiment, the reference model main structure and other model main structures may have position parameters and / or height parameters. The position parameters include the normal vector of the model main structure and the distance from the model main structure to the camera optical center. These two pieces of information can uniquely determine the position of the model main structure in the initial 3D model and its relative position with other model main structures. The height parameters are the height information of the model main structure from the reference plane, such as the height information of the ceiling, the height information of the wall, and the height information of the ground. If the ground is taken as the reference plane, the height information of the ground is 0.

[0085] Optionally, when constructing an optimization function with the position and / or height parameters of each model's main structure as optimization variables, at least one of three optimization terms (first, second, and third types) can be constructed based on the constraint relationships between multiple boundary lines, the constraint relationships between disappearance points on at least two orthogonal principal directions, and the camera's intrinsic parameters. The optimization function is then generated based on at least one optimization term. Preferably, the optimization function can be generated simultaneously based on the first, second, and third optimization terms.

[0086] For each main structure of the model, a reference normal vector of the main structure in the camera coordinate system is generated based on the camera's intrinsic parameters and the disappearance point perpendicular to the main structure. Theoretically, the reference normal vector and the normal vector of the main structure should be parallel. Therefore, the first type of optimization term can be constructed by taking the dot product of the normal vector of the main structure and the reference normal vector as 1 as the optimization objective.

[0087] Among them, for any adjacent model main structure, the normal vector of any adjacent model main structure should be perpendicular. For example, the ground and the wall are perpendicular, the wall and the ceiling are perpendicular, and adjacent walls are perpendicular to each other. Therefore, the second type of optimization term can be constructed by taking the dot product of the normal vectors between any adjacent model main structures as 0 as the optimization objective.

[0088] Specifically, for any adjacent model main structure, the boundary line of any adjacent model main structure in the image coordinate system is generated based on the camera intrinsic parameters, the normal vector of any adjacent model main structure, and the distance from any adjacent model main structure to the camera optical center. In addition, the boundary line between corresponding adjacent physical main structures in the target image is detected based on the Hough transform. Theoretically, the boundary line of any adjacent model main structure in the image coordinate system (here, the boundary line can be regarded as the projection of the boundary line between any adjacent model main structures in the initial 3D model in the image space) should be the same as the boundary line between corresponding adjacent physical main structures detected from the target image (here, the boundary line can be regarded as the projection of the boundary line between adjacent physical main structures in the first spatial object in the image space). Therefore, the third type of optimization term can be constructed with the goal of making the boundary line of any adjacent model main structure in the image coordinate system the same as the boundary line between corresponding adjacent physical main structures in the target image.

[0089] The following example illustrates the construction process of the above optimization terms by combining the plane equations corresponding to the main structure of the model. The main structure of the model to be optimized consists of walls, ceiling, and floor. Assume the wall equation to be optimized is... The first point estimated by the disappearance point i The normal vector of the wall is The first detected from the image i The boundary line between the wall and the floor is marked as... The first detected from the image i The wall and the first j The boundary line of the wall is recorded as The first detected from the image i The boundary line between the wall and the ceiling is marked as... ;in, Represents the norm, It is the first i The normal vector of each wall is a quantity to be optimized. This normal vector has an initial value (i.e., a value determined in the initial 3D model). It is the first i The distance from the wall to the camera's optical center, and the height of the ceiling. These are values ​​assumed when constructing the initial 3D model and require further optimization. Therefore, the optimization variables are the position parameters of the walls (such as the wall equation) and the height parameters of the ceiling. An exemplary optimization function can be represented as follows:

[0090]

[0091] Here, T represents the set of walls appearing in the image. The least squares algorithm is used to solve the optimization function, with the objective of minimizing the sum of all terms in the function, to obtain the position parameters of the walls (such as the wall equations) and the height parameters of the ceiling. The optimization function contains six terms: the first term belongs to the first type of optimization term, the second and third terms belong to the second type of optimization term, and the fourth, fifth, and sixth terms belong to the third type of optimization term. A detailed explanation follows:

[0092] First item, the i normal vector of the wall And the normal vector of the wall obtained from the disappearance point. The directions of the two should be consistent, their dot product should be 1, and after subtracting 1, it should be 0. This is the amount to be optimized.

[0093] The second item, and The normal vectors of adjacent walls are perpendicular, and the dot product of their normal vectors is 0. and All of these are quantities that need to be optimized.

[0094] The third item, and These represent the wall and the floor, respectively, and the wall and the floor are perpendicular to each other. and The dot product is 0. It is known, that is, the direction of gravity in the camera coordinate system. This is the amount to be optimized.

[0095] The fourth item, Indicates the first i The boundary line between the wall and the floor. This indicates that the boundary line is projected onto the image coordinate system, and the boundary line projected onto the image coordinate system is the same as the boundary line detected from the image. The difference is 0. It is known that the boundary between the wall and the ground in the target image is detected based on the Hough transform.

[0096] Fifth, the ceiling boundary line and the ceiling line detected from the image. The distance between them is 0.

[0097] The sixth item, the adjacent first item i The wall and the first j The boundary line between the walls, and the first wall detected from the image. i The wall and the first j The boundary line between the walls The difference between them is 0.

[0098] In this embodiment, after the above optimization, a target 3D model corresponding to the first spatial object can be obtained. In this embodiment, instead of directly detecting 3D structural information from the target image, 2D visual information such as boundary lines and vanishing points are directly detected. Then, 3D structural information is generated using this 2D visual information and Manhattan constraints and vanishing point constraints of the indoor scene. Compared to methods that directly regress 3D structural information based on the target image, this embodiment has higher stability, stronger interpretability, and is more robust to situations with large user shooting angles and cluttered scenes. Furthermore, in this embodiment, boundary lines such as wall lines, ground lines, and ceiling lines are directly detected from the target image, and these boundary lines are used as constraints to optimize the 3D equations of the walls. The resulting target 3D model has a higher degree of fit with the actual wall, ground, and ceiling dividing lines, resulting in higher model quality. Optionally, other real-world scene information, such as textures from the 2D image, can also be reflected in the target 3D model.

[0099] After obtaining the target 3D model corresponding to the first spatial object, various applications can be developed based on it. For example, the target 3D model can be displayed to users so they can view or understand the 3D structure of the first spatial object. Another example is applying the target 3D model to an online home decoration scenario to view the matching effect of home decoration objects (such as paintings, sofas, etc.) with the real environment of the first spatial object, and selecting a home decoration plan based on the matching effect. Yet another example is using the target 3D model for product selection, to view the matching effect of the product to be purchased with the real environment of the first spatial object, and making a purchase based on the matching effect. The following embodiments will use online home decoration and online shopping scenarios as examples for illustration.

[0100] Figure 4a This is a flowchart illustrating an online home decoration method provided as an exemplary embodiment of this application. Figure 4a As shown, the method includes:

[0101] 401a. In response to the image upload operation, obtain the target image corresponding to the first spatial object, wherein the first spatial object is at least a portion of the spatial objects in the target physical space;

[0102] 402a. In response to the placement operation of the target home decoration object on the target image, the target home decoration object is merged into the target 3D model corresponding to the first spatial object to obtain the target 3D model of the merged target home decoration object;

[0103] 403a. Project the target 3D model of the target home decoration object onto the target image to obtain a home decoration rendering containing the target home decoration object. In this embodiment, the online home decoration method can be implemented on a terminal device or by a combination of a terminal device and a server device; there is no limitation on this. The following description uses a combination of a terminal device and a server device as an example.

[0104] Users can open a home decoration app on their terminal device to access an online home decoration page. This page offers various decoration options, such as those based on model rooms or real background images. Users can choose the real background image option, which triggers a page to add real background images. Users can capture a target image of a first spatial object via their terminal device or directly select it from a gallery. The home decoration app, responding to the image capture or selection operation, can obtain the target image corresponding to the first spatial object. The first spatial object can be at least a portion of a spatial object within a target physical space. For example, the target physical space is the user's house to be decorated, which includes bedrooms, living rooms, kitchens, or bathrooms. The first spatial object can be the bedroom and living room, or a portion of the living room; there are no limitations on this. Further details can be found in the aforementioned embodiments and will not be elaborated upon here.

[0105] In this embodiment, after the home decoration app obtains the target image corresponding to the first spatial object, it can upload the target image to the server device. The server device uses the 3D scene reconstruction method described in the aforementioned method embodiments to generate the target 3D model corresponding to the first spatial object. The method for generating the target 3D model can be found in the aforementioned embodiments and will not be repeated here. It should be noted that the process of generating the target 3D model is imperceptible to the user. Figure 4b This is a schematic diagram illustrating the target image and the process from the target image to the generation of a target 3D model, as shown in an embodiment of this application. Figure 4b The text illustrates, from the perspective of model generation principles, the generation of a colored intermediate model and a model with both color and texture from a target image to a target 3D model. However, in practice, intermediate models are usually not generated. It should be noted that the intermediate model and the target 3D model have different perspectives, but are not limited to this. Furthermore, Figure 4b The colors in the image are not explicitly shown; they are only represented by grayscale.

[0106] After a user selects a target image, the home improvement app can display various home improvement objects in the associated area of ​​the target image. These objects can include appliances, furniture, or decorative items. Figure 4c and Figure 4dThe example shown is displaying a sofa, coffee table, rug, armchair, and wall painting in the lower area of ​​the target image. Figure 4c and Figure 4d In the app, users can select a target home decoration object from a variety of options and place it on a target image. For example, they can drag the selected target image to the corresponding position on the target image and then release it. The home decoration app responds to this placement operation by providing the target object's identification information (such as name, product ID, or image) and its positional range on the target image to the server device. The server device then uses the target object's identification information to obtain its corresponding 3D model. Based on the target object's positional range on the target image, it merges the target object's 3D model into the target 3D model and projects the merged model onto the target image to obtain a home decoration rendering containing the target object. This rendering is then provided to the terminal device, which displays the rendering. Figure 4c In the example below, a user selects a painting and places it in the center of the main wall. Figure 4c The home decoration renderings shown have accurate perspective distortion and limit the movement of paintings to the wall surface, demonstrating high precision. Figure 4d In this example, the user selects a sofa and a carpet, places the sofa on the center line between the main wall and the floor, and then lays the carpet on the floor. Figure 4d The perspective in the home decoration renderings shown is correct, and the placement is reasonable.

[0107] In the above embodiments, the online home decoration process is implemented by the cooperation of terminal devices and server devices, but it is not limited to this. Of course, it can also be implemented independently by the terminal device. Specifically, after obtaining the target image corresponding to the first spatial object, the home decoration app can use the three-dimensional scene reconstruction method described in the aforementioned method embodiments to generate the target three-dimensional model corresponding to the first spatial object; at the same time, the home decoration app can display a variety of home decoration objects in the associated area of ​​the target image, and the home decoration app can respond to the placement operation of the target home decoration object on the target image, obtain the identification information of the target home decoration object and the position range information of the target home decoration object on the target image; according to the identification information of the target home decoration object, obtain the three-dimensional model corresponding to the target home decoration, according to the position range information of the target home decoration object on the target image, merge the three-dimensional model of the target home decoration object into the target three-dimensional model, and project the target three-dimensional model of the merged target home decoration object onto the target image to obtain a home decoration rendering containing the target home decoration object and display the home decoration rendering.

[0108] It should be noted that the above embodiments are illustrated by taking the real-time construction of a target 3D model as an example, but are not limited to this. For example, a target 3D model can be constructed when the target image is uploaded for the first time, and the pre-constructed target 3D model can be used directly in subsequent online home decoration.

[0109] Figure 4e This is a flowchart illustrating a product selection method provided for an exemplary embodiment of this application. Figure 4e As shown, the method includes:

[0110] 401b: Respond to the selection operation on the product page, determine the selected target product, and the target product has a 3D model of the product;

[0111] 402b. To view the matching effect, select the target image corresponding to the first spatial object to be matched with the target product;

[0112] 403b. Add the product's 3D model to the target 3D model corresponding to the first spatial object to obtain a target 3D model that integrates the target product;

[0113] 404b. Project the target 3D model of the target product onto the target image to obtain a matching effect diagram of the target product and the first spatial object.

[0114] In this embodiment, the product selection method can be implemented on the terminal device, or it can be implemented in conjunction with the terminal device and the server device; there is no limitation on this. The following description uses the implementation of the product selection method on the terminal device as an example, but it is not limited to this.

[0115] In this embodiment, during a shopping scenario, when a user purchases furniture, appliances, or decorative items, to facilitate the selection of goods that better suit the actual environment, the user can add the selected items to the target image corresponding to a first spatial object and view the matching effect after addition. The first spatial object can be any area with a spatial concept, such as a home environment, a shopping mall environment, or an office scene. The goods can be any items that need to be placed in the first spatial object; for example, in a home environment, goods can be wardrobes, paintings, or tables and chairs, while in a shopping mall environment, goods can be storage cabinets, shelves, or racks.

[0116] In this embodiment, a product page can be displayed on the terminal device. The product page displays a variety of products, and the user can select a target product on the product page. The terminal device can respond to the selection operation on the product page, determine the selected target product, and the target product has a corresponding 3D model.

[0117] In this embodiment, a matching effect viewing control can be added to the shopping cart page, order page, or product details page. Users can use this control to view the product matching effect. In response to this operation, the e-commerce app allows the user to select the target image corresponding to the first spatial object. Users can acquire the target image of the first spatial object through a terminal device or directly select it from the image library. After acquiring the target image, the e-commerce app can use the 3D scene reconstruction method provided in the above embodiment to construct the target 3D model corresponding to the first spatial object, and add the 3D model of the target product selected by the user to the target 3D model corresponding to the first spatial object to obtain a fused target 3D model of the target product. The fused target 3D model of the target product is projected onto the target image to obtain a matching effect diagram of the target product and the first spatial object, and the matching effect diagram is displayed. Users can determine whether to purchase the target product through this matching effect diagram. Further optionally, the matching effect diagram can be automatically scored and a matching score can be given according to preset matching rules or strategies to assist users in determining whether to purchase the target product, which helps improve the user's shopping experience, increases the probability of users successfully purchasing the desired product, and thus reduces the probability of returns and exchanges.

[0118] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0119] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0120] Figure 5 A schematic diagram of a three-dimensional scene reconstruction device provided for an exemplary embodiment of this application, as shown below. Figure 5 As shown, the device includes: an acquisition module 51, a detection module 52, a determination module 53, a reconstruction module 54, and an optimization module 55.

[0121] The acquisition module 51 is used to acquire the target image corresponding to the first spatial object, wherein the first spatial object is at least a portion of the spatial objects in the target physical space;

[0122] Detection module 52 is used to detect multiple boundary lines and at least two orthogonal principal directions of disappearance points in the target image. The boundary lines are the intersection lines between adjacent physical main structures in the first spatial object.

[0123] The determination module 53 is used to determine the camera intrinsic parameters of the target camera and the direction of gravity in the camera coordinate system based on at least two extinction points in the orthogonal principal directions. The target camera refers to the camera used to capture the target image.

[0124] Reconstruction module 54 is used to reconstruct the initial three-dimensional model corresponding to the first spatial object based on multiple boundary lines and the gravity direction in the camera coordinate system. The initial three-dimensional model includes the adjacent model main structure corresponding to the adjacent physical main structure.

[0125] The optimization module 55 is used to optimize the initial 3D model based on the constraint relationship between multiple boundary lines, the constraint relationship between the disappearance points in at least two orthogonal principal directions, and the camera's intrinsic parameters, so as to obtain the target 3D model.

[0126] In an optional embodiment, the detection module 52 is specifically used to: input the target image into a boundary line detection model based on Hough transform to detect the boundary lines, so as to obtain multiple boundary lines existing in the target image; input the target image into an annulment point detection model based on Hough transform to detect annulment points, so as to obtain at least two annulment points in the target image along two orthogonal principal directions.

[0127] In an optional embodiment, the detection module 52 is specifically used to: input the target image into a boundary line detection model based on Hough transform, in which a first feature extraction network that integrates skip connections and multi-scale features is used to extract features from the target image to obtain a first target feature map at multiple scales; perform a Hough transform on the first target feature map at multiple scales using a polar coordinate-based Hough transform to map the straight lines in the target image to points in Hough space; select multiple target points that match the boundary line features in the Hough space, and remap the multiple target points to the image space to obtain multiple boundary lines in the target image.

[0128] In an optional embodiment, the detection module 52 is specifically used to: extract features from the target image to obtain a first intermediate feature map of the largest scale; perform N downsampling operations on the first intermediate feature map of the largest scale to obtain multiple first intermediate feature maps of other scales; use the first intermediate feature map of the smallest scale as the first target feature map of the smallest scale; perform N upsampling operations on the first target feature map of the smallest scale; and perform skip connections with the first intermediate feature map of the same scale in each upsampling operation to obtain first target feature maps of other scales.

[0129] In an optional embodiment, the detection module 52 is specifically configured to: perform Hough transform on the first target feature map at multiple scales to obtain a second target feature map at multiple scales in the Hough space based on polar coordinates; perform scale transformation on the second target feature map at multiple scales to obtain multiple feature maps of the same scale; concatenate the multiple feature maps of the same scale to obtain a third target feature map; and perform convolution dimensionality reduction on the third target feature map to obtain a two-dimensional image in the Hough space, wherein the two-dimensional image contains multiple points, and each point corresponds to a straight line existing in the target image.

[0130] In an optional embodiment, the detection module 52 is specifically used to: input the target image into an annulment point detection model based on Hough transform, in which a second feature extraction network is used to extract features from the target image to obtain a fourth target feature map; perform a Hough transform on the fourth target feature map using a Gaussian sphere-based Hough transform to map the straight lines in the target image to points in Gaussian sphere space; select at least two annulment points in the Gaussian sphere space whose probability values ​​and angle values ​​meet the requirements, and remap them to the image space to obtain at least two annulment points in the target image in the orthogonal principal directions.

[0131] In an optional embodiment, the detection module 52 is specifically configured to: perform a Hough transform on the fourth target feature map to obtain a fifth target feature map in a polar coordinate-based Hough space; perform a Hough convolution on the fifth target feature map in the Hough space to obtain a sixth target feature map; perform a spherical Gaussian spherical transform on the sixth target feature map to obtain a seventh target feature map in a Gaussian spherical space; and perform a spherical convolution on the seventh target feature map in the Gaussian spherical space to obtain an eighth target feature map, wherein the eighth target feature map includes multiple points, each point corresponding to a straight line existing in the target image.

[0132] In an optional embodiment, the detection module 52 is specifically used to: select points whose probability values ​​meet the set requirements from multiple points as disappearance points based on the probability values ​​of each point in the eighth target feature map; and select at least two disappearance points whose angle is greater than the set angle from the disappearance points as disappearance points in at least two orthogonal principal directions.

[0133] In an optional embodiment, the determining module 53 is specifically configured to: select the two target annihilation points closest to the camera optical center from the at least two finite annihilation points when there are at least two annihilation points among the at least two orthogonal principal directions; determine the camera intrinsic parameters of the target camera based on the constraint relationship between the two target annihilation points and the camera intrinsic parameters; and transform the annihilation points located in the gravity direction among the at least two orthogonal principal directions into the camera coordinate system based on the camera intrinsic parameters to obtain the gravity direction in the camera coordinate system.

[0134] In an optional embodiment, the reconstruction module 54 is specifically used to: construct a reference plane in the camera coordinate system according to the gravity direction in the camera coordinate system, the reference plane corresponding to the reference physical main structure contained in the first spatial object; identify multiple reference boundary lines intersecting with the reference physical main structure from multiple boundary lines, and construct a reference model main structure corresponding to the reference physical main structure on the reference plane according to the intersection position between the multiple reference boundary lines; and construct other model main structures corresponding to other physical main structures on the reference model main structure according to the preset height from the camera optical center to the reference plane and the intersection position between the multiple reference boundary lines, so as to obtain the initial three-dimensional model corresponding to the first spatial object.

[0135] In an optional embodiment, the reconstruction module 54 is specifically used to: select effective reference boundary lines from multiple reference boundary lines; sort the effective reference boundary lines according to the angle between the effective reference boundary lines and the x-axis in the image coordinate system to obtain the adjacency relationship between the effective reference boundary lines; determine the intersection position between the effective reference boundary lines according to the adjacency relationship between the effective reference boundary lines, and divide the intersection position into a first intersection position that intersects with the boundary of the target image and a second intersection position that does not intersect with the boundary of the target image; and draw the model boundary corresponding to the reference physical main structure on the reference plane according to the first intersection position and the second intersection position to obtain the reference model main structure.

[0136] In an optional embodiment, the reconstruction module 54 is specifically used to: select a first reference boundary line from multiple reference boundary lines based on the angle between multiple reference boundary lines and the x-axis in the image coordinate system and / or the length of multiple reference boundary lines; and discard reference boundary lines whose angles are less than a set angle threshold based on the angles between other reference boundary lines and the first reference boundary line, and use the remaining reference boundary lines and the first reference boundary line as valid reference boundary lines.

[0137] In an optional embodiment, the reconstruction module 54 is specifically used to: determine the adjacent model boundaries that intersect at the second intersection point on the main structure of the reference model based on the second intersection point position; determine the initial height of other main structures of the model based on the preset height from the camera optical center to the reference plane and the preset scaling ratio; and construct other main structures of the model on the adjacent model boundaries that intersect at the second intersection point based on the initial height, so as to obtain the initial three-dimensional model corresponding to the first spatial object.

[0138] In an optional embodiment, the optimization module 55 is specifically used to: construct an optimization function with the position parameters and / or height parameters of each model's main structure as optimization variables, based on the constraint relationships between multiple boundary lines, the constraint relationships between at least two orthogonal principal directions and the camera's intra-camera parameters; the position parameters include the normal vector of the model's main structure and the distance to the camera's optical center; solve the optimization function using a least squares algorithm to obtain the optimized position parameters and / or height parameters of each model's main structure; and adjust the position of each model's main structure according to the optimized position parameters and / or height parameters to obtain the target 3D model.

[0139] In an optional embodiment, the optimization module 55 is specifically used for: for each main model structure, generating a reference normal vector of the main model structure in the camera coordinate system based on the camera's intra-camera parameters and the disappearance point perpendicular to the main model structure; constructing a first type of optimization term with the dot product of the main model structure's normal vector and the reference normal vector being 1 as the optimization objective; constructing a second type of optimization term with the dot product of the normal vectors between any two adjacent main model structures being 0 as the optimization objective; generating a boundary line of any two adjacent main model structures in the image coordinate system based on the camera's intra-camera parameters, the normal vectors of any two adjacent main model structures, and the distance to the camera's optical center; constructing a third type of optimization term with the boundary line of any two adjacent main model structures in the image coordinate system being the same as the boundary line between corresponding adjacent physical main structures in the target image as the optimization objective; and generating an optimization function based on the first, second, and third type of optimization terms. Detailed implementation methods for each of the above operations can be found in the foregoing embodiments and will not be repeated here.

[0140] This application also provides an online home decoration device, which includes an acquisition module, a fusion module, and a projection module. The acquisition module is used to acquire a target image corresponding to a first spatial object in response to an image upload operation. The first spatial object is at least a portion of the spatial objects in the target physical space. The fusion module is used to merge the target home decoration object into a target 3D model corresponding to the first spatial object in response to a placement operation of the target home decoration object on the target image, thereby obtaining a target 3D model of the merged target home decoration object. The projection module is used to project the target 3D model of the merged target home decoration object onto the target image to obtain a home decoration rendering containing the target home decoration object; wherein the target 3D model is constructed according to the steps in the 3D scene reconstruction method provided in this application. Detailed implementation methods for each of the above operations can be found in the foregoing embodiments and will not be repeated here.

[0141] This application also provides a product selection device, which includes: a determining module, a selecting module, an adding module, and a projection module. The determining module is used to determine the selected target product in response to a selection operation on a product page. The target product has a three-dimensional model. The selecting module is used to select a target image corresponding to a first spatial object to be paired with the target product in response to a matching effect viewing operation. The adding module is used to add the product's three-dimensional model to the target three-dimensional model corresponding to the first spatial object to obtain a target three-dimensional model of the merged target product. The merging module is used to project the merged target three-dimensional model of the target product onto the target image to obtain a matching effect diagram of the target product and the first spatial object; wherein, the target three-dimensional model is constructed according to the steps in the three-dimensional scene reconstruction method provided in this application. Detailed implementation methods of the above operations can be found in the foregoing embodiments and will not be repeated here.

[0142] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 6 As shown, the device includes a memory 64 and a processor 65.

[0143] Memory 64 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method used to operate on the computing platform.

[0144] The memory 64 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0145] Processor 65, coupled to memory 64, is used to execute a computer program in memory 64 for: acquiring a target image corresponding to a first spatial object, the first spatial object being at least a portion of the spatial objects in the target physical space; detecting multiple boundary lines and at least two vanishing points on orthogonal principal directions in the target image, the boundary lines being the lines of intersection between adjacent physical main structures in the first spatial object; determining the camera intrinsic parameters of a target camera and the gravity direction in the camera coordinate system based on the vanishing points on the at least two orthogonal principal directions, the target camera referring to the camera used to capture the target image; reconstructing an initial three-dimensional model corresponding to the first spatial object based on the multiple boundary lines and the gravity direction in the camera coordinate system, the initial three-dimensional model including adjacent model main structures corresponding to adjacent physical main structures; and optimizing the initial three-dimensional model based on the constraint relationships between the multiple boundary lines, the constraint relationships between the vanishing points on the at least two orthogonal principal directions, and the camera intrinsic parameters to obtain a target three-dimensional model.

[0146] In an optional embodiment, when the processor 65 detects multiple boundary lines and at least two annulment points in the target image based on the Hough transform, it specifically performs the following steps: inputting the target image into a boundary line detection model based on the Hough transform to detect the boundary lines, thereby obtaining multiple boundary lines in the target image; and inputting the target image into an annulment point detection model based on the Hough transform to detect annulment points, thereby obtaining at least two annulment points in the target image in the orthogonal principal directions.

[0147] In an optional embodiment, when the processor 65 inputs the target image into a boundary detection model based on Hough transform to detect boundary lines and obtain multiple boundary lines existing in the target image, it specifically performs the following steps: inputting the target image into the boundary detection model based on Hough transform, in which a first feature extraction network fusing skip connections and multi-scale features is used to extract features from the target image to obtain first target feature maps at multiple scales; performing a Hough transform on the first target feature maps at multiple scales using a polar coordinate-based Hough transform to map the straight lines existing in the target image to points in Hough space; selecting multiple target points that match the boundary line features in the Hough space, and remapping the multiple target points back to the image space to obtain multiple boundary lines existing in the target image.

[0148] In an optional embodiment, when the processor 65 extracts features from the target image using a first feature extraction network that integrates skip connections and multi-scale features to obtain first target feature maps at multiple scales, it specifically performs the following steps: extracts features from the target image to obtain a first intermediate feature map at the largest scale; performs N downsampling operations on the first intermediate feature map at the largest scale to obtain first intermediate feature maps at other scales; uses the first intermediate feature map at the smallest scale as the first target feature map at the smallest scale; performs N upsampling operations on the first target feature map at the smallest scale; and performs skip connections with the first intermediate feature map at the same scale in each upsampling operation to obtain first target feature maps at other scales.

[0149] In an optional embodiment, when the processor 65 performs a Hough transform on the first target feature maps at multiple scales using a polar coordinate-based Hough transform to map the straight lines in the target image to points in the Hough space, it specifically performs the following: performs a Hough transform on the first target feature maps at multiple scales to obtain second target feature maps at multiple scales in the polar coordinate-based Hough space; performs a scale transformation on the second target feature maps at multiple scales to obtain multiple feature maps of the same scale; concatenates the multiple feature maps of the same scale to obtain a third target feature map; and performs convolutional dimensionality reduction on the third target feature map to obtain a two-dimensional image in the Hough space, wherein the two-dimensional image contains multiple points, each point corresponding to a straight line in the target image.

[0150] In an optional embodiment, when the processor 65 inputs the target image into the Hough transform-based cancellation point detection model for cancellation point detection to obtain cancellation points in at least two orthogonal principal directions in the target image, it specifically performs the following steps: inputting the target image into the Hough transform-based cancellation point detection model, in which a second feature extraction network is used to extract features from the target image to obtain a fourth target feature map; performing a Hough transform on the fourth target feature map using a Gaussian sphere-based Hough transform to map the straight lines present in the target image to points in a Gaussian sphere space; selecting cancellation points in at least two orthogonal principal directions whose probability values ​​and angle values ​​meet the requirements in the Gaussian sphere space, and remapping them to the image space to obtain cancellation points in at least two orthogonal principal directions in the target image.

[0151] In an optional embodiment, when the processor 65 performs a Hough transform on the fourth target feature map using a Gaussian spherical Hough transform to map the straight lines in the target image to points in a Gaussian spherical space, it specifically performs the following: performs a Hough transform on the fourth target feature map to obtain a fifth target feature map in a polar coordinate-based Hough space; performs a Hough convolution on the fifth target feature map in the Hough space to obtain a sixth target feature map; performs a spherical Gaussian spherical transform on the sixth target feature map to obtain a seventh target feature map in a Gaussian spherical space; and performs a spherical convolution on the seventh target feature map in the Gaussian spherical space to obtain an eighth target feature map, wherein the eighth target feature map includes multiple points, each point corresponding to a straight line in the target image.

[0152] In an optional embodiment, when the processor 65 selects at least two orthogonal principal directions in the Gaussian spherical space where the probability value and angle value meet the requirements, it is specifically used to: select points from multiple points where the probability value meets the set requirements as disappearance points based on the probability value of each point in the eighth target feature map; and select at least two disappearance points from the disappearance points where the angle is greater than the set angle as disappearance points in at least two orthogonal principal directions.

[0153] In an optional embodiment, when the processor 65 determines the camera intrinsic parameters of the target camera and the gravity direction in the camera coordinate system based on the ablation points in at least two orthogonal principal directions, it specifically performs the following: when the ablation points in the at least two orthogonal principal directions include at least two finite ablation points, it selects the two target ablation points closest to the camera optical center from the at least two finite ablation points; it determines the camera intrinsic parameters of the target camera based on the constraint relationship between the two target ablation points and the camera intrinsic parameters; and it transforms the ablation points located in the gravity direction in the at least two orthogonal principal directions to the camera coordinate system based on the camera intrinsic parameters to obtain the gravity direction in the camera coordinate system.

[0154] In an optional embodiment, when the processor 65 reconstructs the initial three-dimensional model corresponding to the first spatial object based on multiple boundary lines and the gravity direction in the camera coordinate system, it specifically performs the following steps: Based on the gravity direction in the camera coordinate system, it constructs a reference plane in the three-dimensional space, the reference plane corresponding to the reference physical main structure contained in the first spatial object; it identifies multiple reference boundary lines intersecting with the reference physical main structure from among the multiple boundary lines; based on the intersection positions between the multiple reference boundary lines, it constructs a reference model main structure corresponding to the reference physical main structure on the reference plane; based on the preset height from the camera optical center to the reference plane and the intersection positions between the multiple reference boundary lines, it constructs other model main structures corresponding to other physical main structures on the reference model main structure to obtain the initial three-dimensional model corresponding to the first spatial object.

[0155] In an optional embodiment, when the processor 65 constructs a reference model body structure corresponding to the reference physical body structure on the reference plane based on the intersection positions between multiple reference boundary lines, it specifically performs the following steps: selecting effective reference boundary lines from the multiple reference boundary lines; sorting the effective reference boundary lines according to the angle between the effective reference boundary lines and the x-axis in the image coordinate system to obtain the adjacency relationship between the effective reference boundary lines; determining the intersection positions between the effective reference boundary lines based on the adjacency relationship between the effective reference boundary lines, and dividing the intersection positions into a first intersection position that intersects with the boundary of the target image and a second intersection position that does not intersect with the boundary of the target image; and drawing the model boundary corresponding to the reference physical body structure on the reference plane based on the first intersection position and the second intersection position to obtain the reference model body structure.

[0156] In an optional embodiment, when the processor 65 selects a valid reference boundary line from multiple reference boundary lines, it specifically performs the following steps: selects a first reference boundary line from multiple reference boundary lines based on the angle between the multiple reference boundary lines and the x-axis in the image coordinate system and / or the length of the multiple reference boundary lines; and, based on the angle between other reference boundary lines and the first reference boundary line, discards reference boundary lines whose angle is less than a set angle threshold, and uses the remaining reference boundary lines and the first reference boundary line as valid reference boundary lines.

[0157] In an optional embodiment, when the processor 65 constructs other model main structures corresponding to other physical main structures on the main structure of the reference model based on the preset height from the camera optical center to the reference plane and the intersection positions between multiple reference boundary lines to obtain the initial three-dimensional model corresponding to the first spatial object, it is specifically used to: determine the adjacent model boundaries intersecting at the second intersection position on the main structure of the reference model based on the second intersection position; determine the initial height of the other model main structures based on the preset height from the camera optical center to the reference plane and the preset scaling ratio; and construct other model main structures on the adjacent model boundaries intersecting at the second intersection position based on the initial height to obtain the initial three-dimensional model corresponding to the first spatial object.

[0158] In an optional embodiment, when the processor 65 optimizes the initial 3D model based on the constraint relationships between multiple boundary lines, the constraint relationships between the annihilation points on at least two orthogonal principal directions, and the camera's intra-camera parameters to obtain the target 3D model, specifically, it performs the following steps: Based on the constraint relationships between multiple boundary lines, the constraint relationships between the annihilation points on at least two orthogonal principal directions, and the camera's intra-camera parameters, it constructs an optimization function with the position parameters and / or height parameters of each model's main structure as optimization variables. The position parameters include the normal vector of the model's main structure and its distance to the camera's optical center. It then solves the optimization function using a least-squares algorithm to obtain the optimized position parameters and / or height parameters of each model's main structure, and adjusts the position of each model's main structure based on the optimized position parameters and / or height parameters to obtain the target 3D model.

[0159] In an optional embodiment, when the processor 65 constructs an optimization function with the position parameters and / or height parameters of each model's main structure as optimization variables based on the constraint relationships between multiple boundary lines, the constraint relationships between at least two orthogonal principal directions, and camera intra-camera parameters, it specifically performs the following: for each model's main structure, based on the camera intra-camera parameters and the disappearance points perpendicular to the model's main structure, it generates a reference normal vector of the model's main structure in the camera coordinate system, and constructs a first type of optimization term with the dot product of the model's main structure's normal vector and the reference normal vector being 1 as the optimization objective; for any adjacent model... The first type of optimization term is constructed with the objective that the dot product of the normal vectors between any two adjacent model main structures is zero. For any two adjacent model main structures, the boundary line in the image coordinate system is generated based on the camera's intrinsic parameters, the normal vectors of any two adjacent model main structures, and their distances to the camera's optical center. The third type of optimization term is constructed with the objective that the boundary line in the image coordinate system of any two adjacent model main structures is the same as the boundary line between corresponding adjacent physical main structures in the target image. An optimization function is generated based on the first, second, and third types of optimization terms.

[0160] Furthermore, such as Figure 6 As shown, the electronic device also includes other components such as a communication component 66, a display 67, a power supply component 68, and an audio component 69. Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components shown.

[0161] This application also provides an electronic device, the structure of which is similar to... Figure 6 The electronic devices shown have the same or similar structures; see details below. Figure 6 The structure of the electronic device shown is implemented in this embodiment, and the electronic device provided in this embodiment is similar to... Figure 6The main difference between the electronic devices in the illustrated embodiments lies in the different functions implemented by the computer programs stored in the memory executed by the processor. For the electronic device provided in this embodiment, the processor executing the computer programs stored in the memory can be used to: respond to an image upload operation to obtain a target image corresponding to a first spatial object, where the first spatial object is at least a portion of the spatial objects in the target physical space; respond to a placement operation of a target home decoration object on the target image to merge the target home decoration object into the target 3D model corresponding to the first spatial object, thereby obtaining a target 3D model fused with the target home decoration object; and project the target 3D model fused with the target home decoration object onto the target image to obtain a home decoration rendering containing the target home decoration object; wherein, the target 3D model is constructed according to the steps in the 3D scene reconstruction method provided in this application embodiment.

[0162] This application also provides an electronic device, the structure of which is similar to... Figure 6 The electronic devices shown have the same or similar structures; see details below. Figure 6 The structure of the electronic device shown is implemented in this embodiment, and the electronic device provided in this embodiment is similar to... Figure 6 The main difference between the electronic devices in the illustrated embodiments lies in the different functions implemented by the computer programs stored in the memory executed by the processor. For the electronic device provided in this embodiment, the processor executing the computer programs stored in the memory can be used to: respond to a selection operation on a product page, determine the selected target product, which has a 3D model; respond to a matching effect viewing operation, select the target image corresponding to the first spatial object to be matched with the target product; add the 3D model of the product to the target 3D model corresponding to the first spatial object to obtain a target 3D model of the merged target product; project the target 3D model of the merged target product onto the target image to obtain a matching effect diagram of the target product and the first spatial object; wherein, the target 3D model is constructed according to the steps in the 3D scene reconstruction method provided in this application embodiment.

[0163] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the above-described functions. Figure 1b , Figure 4a and Figure 4e The steps in the method shown.

[0164] The above Figure 6The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0165] The above Figure 6 The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0166] The above Figure 6 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0167] The above Figure 6 The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0174] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0176] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, include: Obtain the target image corresponding to the first spatial object, wherein the first spatial object is at least a portion of the spatial objects in the target physical space; Detect multiple boundary lines in the target image and detect at least two disappearing points in the target image along two orthogonal principal directions. The boundary lines are the intersection lines between adjacent physical main structures in the first spatial object. Based on the at least two orthogonal principal directions of the disappearance point, determine the camera intrinsic parameters of the target camera and the direction of gravity in the camera coordinate system, wherein the target camera refers to the camera used to capture the target image; Based on the multiple boundary lines and the gravity direction in the camera coordinate system, the initial three-dimensional model corresponding to the first spatial object is reconstructed. The initial three-dimensional model includes the adjacent model main structure corresponding to the adjacent physical main structure. Based on the constraint relationships between the multiple boundary lines, the constraint relationships between the disappearance points in the at least two orthogonal principal directions, and the camera intra-parameters, the initial 3D model is optimized to obtain the target 3D model. Detecting multiple boundary lines in the target image includes: in a boundary line detection model based on Hough transform, using a first feature extraction network that integrates skip connections and multi-scale features to extract features from the target image to obtain first target feature maps at multiple scales; and using a polar coordinate-based Hough transform to perform a Hough transform on the first target feature maps at multiple scales to map the straight lines in the target image to points in Hough space. Multiple target points matching the boundary line features are selected in the Hough space, and these multiple target points are remapped into the image space to obtain multiple boundary lines existing in the target image.

2. The method according to claim 1, characterized in that, Detecting at least two disappearance points in the target image along two orthogonal principal directions includes: The target image is input into an annulment detection model based on Hough transform to detect annulment points, thereby obtaining at least two annulment points in the target image along orthogonal principal directions.

3. The method according to claim 2, characterized in that, Performing a Hough transform on the first target feature maps at multiple scales using a polar coordinate-based Hough transform to map straight lines in the target image to points in Hough space includes: A Hough transform is performed on the first target feature maps at multiple scales to obtain second target feature maps at multiple scales in the Hough space based on polar coordinates. The second target feature maps at multiple scales are scaled to obtain multiple feature maps of the same scale. The multiple feature maps of the same scale are then concatenated to obtain a third target feature map. The third target feature map is subjected to convolutional dimensionality reduction to obtain a two-dimensional image in Hough space. The two-dimensional image contains multiple points, each of which corresponds to a straight line in the target image.

4. The method according to claim 2, characterized in that, The target image is input into an annulment detection model based on Hough transform for annulment detection to obtain at least two annulment points in the target image along orthogonal principal directions, including: In the Hough transform-based disappearance point detection model, the second feature extraction network is used to extract features from the target image to obtain a fourth target feature map; The Hough transform based on the Gaussian sphere is used to perform a Hough transform on the fourth target feature map so that the straight lines in the target image are mapped to points in the Gaussian sphere space. In the Gaussian spherical space, at least two orthogonal principal directions with probability and angle values ​​that meet the requirements are selected for extinction points, and these points are remapped into the image space to obtain at least two orthogonal principal directions in the target image.

5. The method according to claim 4, characterized in that, The fourth target feature map is subjected to a Hough transform based on a Gaussian sphere to map straight lines in the target image to points in a Gaussian sphere space, including: Perform a Hough transform on the fourth target feature map to obtain a fifth target feature map in Hough space based on polar coordinates; The fifth target feature map is convolved in the Hough space to obtain the sixth target feature map; The sixth target feature map is subjected to Gaussian spherical transformation to obtain the seventh target feature map in Gaussian spherical space; The seventh target feature map is convolved in the Gaussian spherical space to obtain the eighth target feature map, which includes multiple points, each point corresponding to a straight line in the target image.

6. The method according to claim 1, characterized in that, Based on the at least two orthogonal principal directions, determine the target camera's intrinsic parameters and the direction of gravity in the camera coordinate system, including: If at least two finite annulus points are included among the at least two orthogonal principal directions, select the two target annulus points that are closest to the camera optical center from the at least two finite annulus points; Based on the constraint relationship between the two target disappearance points and the camera's intrinsic parameters, the camera's intrinsic parameters are determined. Based on the camera's intrinsic parameters, the disappearance points located in the gravity direction among the at least two orthogonal principal directions are transformed into the camera coordinate system to obtain the gravity direction in the camera coordinate system.

7. The method according to claim 1, characterized in that, Based on the multiple boundary lines and the gravity direction in the camera coordinate system, the initial 3D model corresponding to the first spatial object is reconstructed, including: Based on the direction of gravity in the camera coordinate system, a reference plane for three-dimensional space is constructed in the camera coordinate system. The reference plane corresponds to the reference physical main structure contained in the first spatial object. Identify multiple baseline boundary lines that intersect with the baseline physical main structure from the multiple boundary lines, and construct a baseline model main structure corresponding to the baseline physical main structure on the baseline plane based on the intersection points between the multiple baseline boundary lines; Based on the preset height of the camera optical center to the reference plane and the intersection position between the multiple reference boundary lines, other model main structures corresponding to other physical main structures are constructed on the reference model main structure to obtain the initial three-dimensional model corresponding to the first spatial object.

8. The method according to claim 7, characterized in that, Based on the intersection points of the multiple benchmark boundary lines, a benchmark model main structure corresponding to the benchmark physical main structure is constructed on the benchmark plane, including: Select a valid reference boundary line from the plurality of reference boundary lines, and sort the valid reference boundary lines according to the angle between the valid reference boundary line and the x-axis in the image coordinate system to obtain the adjacency relationship between the valid reference boundary lines; Based on the adjacency relationship between the effective reference boundary lines, the intersection positions between the effective reference boundary lines are determined, and the intersection positions are divided into a first intersection position that intersects with the boundary of the target image and a second intersection position that does not intersect with the boundary of the target image. Based on the first intersection point position and the second intersection point position, the model boundary corresponding to the reference physical main structure is drawn on the reference plane to obtain the reference model main structure.

9. The method according to claim 8, characterized in that, Based on the preset height of the camera optical center to the reference plane and the intersection points between the multiple reference boundary lines, other model main structures corresponding to other physical main structures are constructed on the reference model main structure to obtain the initial three-dimensional model corresponding to the first spatial object, including: Based on the location of the second intersection point, determine the adjacent model boundaries on the main structure of the reference model that intersect at the location of the second intersection point; The initial height of the other main structures of the model is determined based on the preset height of the camera optical center to the reference plane and the preset scaling ratio. Based on the initial height, other main structures of the model are constructed on the adjacent model boundaries that intersect at the second intersection point to obtain the initial three-dimensional model corresponding to the first spatial object.

10. The method according to any one of claims 1-9, characterized in that, Based on the constraint relationships between the multiple boundary lines, the constraint relationships between the disappearance points on at least two orthogonal principal directions, and the camera intra-parameters, the initial 3D model is optimized to obtain the target 3D model, including: Based on the constraint relationships between the multiple boundary lines, the constraint relationships between the extinction points on the at least two orthogonal principal directions, and the camera intra-parameters, an optimization function is constructed with the position parameters and / or height parameters of each model's main structure as optimization variables. The position parameters include the normal vector of the model's main structure and the distance to the camera's optical center. The optimization function is solved using the least squares algorithm to obtain the optimized position parameters and / or height parameters of the main structure of each model. The positions of the main structures of each model are adjusted according to the optimized position parameters and / or height parameters to obtain the target three-dimensional model.

11. The method according to claim 10, characterized in that, Based on the constraint relationships between the multiple boundary lines, the constraint relationships between the disappearance points on at least two orthogonal principal directions, and the camera intra-camera parameters, an optimization function is constructed with the position parameters and / or height parameters of each model's main structure as optimization variables, including: For each main structure of the model, a reference normal vector of the main structure of the model in the camera coordinate system is generated based on the camera intra-parameters and the disappearance point perpendicular to the main structure of the model. The first type of optimization term is constructed with the dot product of the normal vector of the main structure of the model and the reference normal vector being 1 as the optimization objective. For any adjacent main structure of the model, the second type of optimization term is constructed with the goal that the dot product of the normal vectors between any adjacent main structures of the model is 0. For any adjacent model main structure, based on the camera intrinsic parameters and the normal vector and distance to the camera optical center of any adjacent model main structure, the boundary line of any adjacent model main structure in the image coordinate system is generated; and a third type of optimization term is constructed with the goal that the boundary line of any adjacent model main structure in the image coordinate system is the same as the boundary line between the corresponding adjacent physical main structures in the target image. The optimization function is generated based on the first type of optimization terms, the second type of optimization terms, and the third type of optimization terms.

12. An online home decoration method, characterized in that, include: In response to an image upload operation, obtain the target image corresponding to the first spatial object, wherein the first spatial object is at least a portion of the spatial objects in the target physical space; In response to the placement operation of the target home decoration object on the target image, the target home decoration object is merged into the target 3D model corresponding to the first spatial object to obtain the target 3D model fused with the target home decoration object; The target 3D model of the target home decoration object is projected onto the target image to obtain a home decoration rendering containing the target home decoration object; The target three-dimensional model is constructed according to the steps of the method according to any one of claims 1-11.

13. A method for selecting goods, characterized in that, include: In response to a selection operation on the product page, the selected target product is determined, and the target product has a three-dimensional product model; In response to the matching effect viewing operation, select the target image corresponding to the first spatial object to be matched with the target product; The product's 3D model is added to the target 3D model corresponding to the first spatial object to obtain a target 3D model that integrates the target product; The target 3D model of the target product is projected onto the target image to obtain a matching effect diagram of the target product and the first spatial object; The target three-dimensional model is constructed according to the steps of the method according to any one of claims 1-11.

14. An electronic device, characterized in that, include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps of the method according to any one of claims 1-11, 12 and 13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-11, 12 and 13.

Images