Product recommendation, three-dimensional scene model generation method, device and storage medium

Abstract

The embodiment of the present application provides a method, device and storage medium for product recommendation and three-dimensional scene model generation. The method includes the following steps: generating a three-dimensional scene model of the indoor space according to a panoramic view of the indoor space; the three-dimensional scene model includes a three-dimensional space structure model of the indoor space and a three-dimensional object model of the objects in the indoor space; the panoramic view is determined according to the real scene image taken in the indoor space input by the user; displaying the three-dimensional scene model and at least one recommended product; in response to the selection operation triggered by the user for the first recommended product among the at least one recommended product, adding the three-dimensional object model corresponding to the first recommended product to the three-dimensional scene model to obtain an updated three-dimensional scene model; and displaying the updated three-dimensional scene model. The product recommendation scheme provided by the embodiment of the present application has good intuitiveness.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and storage medium for product recommendation and three-dimensional scene model generation. Background Art

[0002] 3D modeling technology has played a huge role in many fields due to its good visual experience, such as real estate, home decoration design, and film and television.

[0003] At present, VR (Virtual Reality) online house viewing plays an important role in the real estate field, improving the efficiency of consumers' house viewing. Home decoration is closely related to the real estate industry. As the scale of virtual house viewing gradually increases, new user needs have spawned derivative functions such as home decoration design preview and product recommendation. Summary of the invention

[0004] The present application provides a product recommendation method, device and storage medium for improving the intuitiveness of product recommendation schemes and improving user product selection efficiency; the present application also provides a three-dimensional scene model generation method, device and storage medium for improving the restoration degree of three-dimensional scene models.

[0005] Therefore, in one embodiment of the present application, a commodity recommendation method is provided, which includes:

[0006] Generate a three-dimensional scene model of the indoor space according to the panoramic view of the indoor space; the three-dimensional scene model includes a three-dimensional spatial structure model of the indoor space and three-dimensional object models of objects in the indoor space; the panoramic view is determined according to a real scene image taken in the indoor space and input by the user;

[0007] Displaying the three-dimensional scene model and at least one recommended product;

[0008] In response to a selection operation triggered by the user on a first recommended product among the at least one recommended product, adding a three-dimensional object model corresponding to the first recommended product to the three-dimensional scene model to obtain an updated three-dimensional scene model;

[0009] The updated three-dimensional scene model is displayed.

[0010] In another embodiment of the present application, a method for generating a three-dimensional scene model is provided, which includes:

[0011] Based on the panoramic view of the indoor space, determining a three-dimensional spatial structure model of the indoor space and a three-dimensional object model of an object in the indoor space;

[0012] Determining a target structural surface contacted by the object and a first relative position of the object on the target structural surface according to a position of an object pixel area of ​​the object in the panoramic image and a position of a structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space;

[0013] Determine, according to the first relative position of the object on the target structural surface, a second relative position of the three-dimensional object model of the object on the target model structural surface in the three-dimensional spatial structure model; wherein the target model structural surface is a model structural surface in the three-dimensional spatial structure model corresponding to the target structural surface;

[0014] The three-dimensional space structure model and the three-dimensional object model of the object are combined according to the second relative position to obtain a three-dimensional scene model of the indoor space.

[0015] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0016] The memory is used to store programs;

[0017] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the above-mentioned commodity recommendation method or three-dimensional scene model generation method.

[0018] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a computer, the above-mentioned product recommendation method or three-dimensional scene model generation method can be implemented.

[0019] In the technical solution provided in the embodiment of the present application, the three-dimensional scene model displayed to the user is determined based on the real-life image captured in the indoor space and input by the user, that is, the three-dimensional scene model is more in line with the actual decoration of the user's indoor space. After the user selects a recommended product from at least one recommended product, the three-dimensional model of the selected recommended product is added to the three-dimensional scene model, and the updated three-dimensional scene model is obtained and displayed to the user. In this way, the user can intuitively feel whether the recommended product he selected matches the actual decoration of his indoor space and whether it meets his expected home improvement effect. In other words, the technical solution provided in the embodiment of the present application improves the intuitiveness of the product recommendation solution, thereby improving the efficiency of users in selecting products.

[0020] In the technical solution provided by another embodiment of the present application, the target structural surface contacted by the object and the first relative position of the object on the target structural surface are determined according to the position of the structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space in the panoramic view of the indoor space and the position of the item pixel area of ​​the item in the indoor space; the second relative position of the three-dimensional item model of the object on the structural surface of the target model in the three-dimensional spatial structure model is determined according to the first relative position of the item on the target structural surface; according to the second relative position, the three-dimensional spatial structure model and the three-dimensional item model of the item are combined to obtain the three-dimensional scene model of the indoor space. The technical solution provided by the embodiment of the present application can improve the positioning accuracy of the three-dimensional item model of the object in the three-dimensional spatial structure model, thereby improving the restoration degree of the three-dimensional scene model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flowchart of a product recommendation method provided in one embodiment of the present application;

[0023] Figure 2 A schematic diagram of a process for generating a three-dimensional scene model according to an embodiment of the present application;

[0024] Figure 3 A detection schematic diagram provided for an embodiment of the present application;

[0025] Figure 4 An example diagram of a furniture model library provided in one embodiment of the present application;

[0026] Figure 5 An example diagram of a texture library provided in an embodiment of the present application;

[0027] Figure 6 A structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In the furniture or home appliance recommendation scheme, the furniture or home appliance recommended to the user can only be imagined by the user to see whether the furniture or home appliance matches the original decoration of his or her home, which is less intuitive and the user's selection cost is high.

[0029] In order to solve or partially solve the above technical solutions, the embodiment of the present application provides a product recommendation method. Specifically, a three-dimensional scene model displayed to the user is generated based on the real-life image taken in the indoor space input by the user; when the user selects a recommended product from at least one recommended product, the three-dimensional model of the selected recommended product is added to the three-dimensional scene model to obtain an updated three-dimensional scene model and display it to the user. In this way, the user can intuitively feel whether the recommended product he selected matches the actual decoration of his indoor space. In other words, the technical solution provided by the embodiment of the present application improves the intuitiveness of the product recommendation solution, thereby improving the efficiency of users in selecting products.

[0030] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0032] Figure 1 The flowchart of the product recommendation method provided by an embodiment of the present application is shown. The execution subject of the method may be a client. The client may be a hardware with an embedded program integrated on the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a desktop computer, etc. Figure 1 As shown, the method includes:

[0033] 101. Generate a three-dimensional scene model of the indoor space according to the panoramic view of the indoor space.

[0034] 102. Display the three-dimensional scene model and at least one recommended product.

[0035] 103. In response to a selection operation triggered by the user on a first recommended product among the at least one recommended product, a three-dimensional object model corresponding to the first recommended product is added to the three-dimensional scene model to obtain an updated three-dimensional scene model.

[0036] 104. Display the updated three-dimensional scene model.

[0037] In the above 101, the 3D scene model includes a 3D spatial structure model of the indoor space and a 3D object model of the objects in the indoor space; the panoramic image is determined based on the real scene image captured in the indoor space input by the user. In this case, the objects may include furniture, furniture, decorations, etc.

[0038] In one example, the real scene image may include the panorama. The user may use a panoramic camera to obtain the panorama. In another example, the real scene image may include: multiple perspective images obtained by taking pictures at multiple viewing angles; the panorama may be obtained by stitching the multiple perspective images.

[0039] A three-dimensional spatial structure model of the indoor space and a three-dimensional object model of the objects in the indoor space can be generated according to the panoramic image; and a three-dimensional scene model of the indoor space can be obtained according to the three-dimensional spatial structure model of the indoor space and the three-dimensional object model of the objects in the indoor space.

[0040] In a feasible solution, the panoramic image may be subjected to wall line detection; and the internal three-dimensional spatial structure model may be determined based on the wall line detection result. Of course, the three-dimensional pose of the internal three-dimensional spatial structure model may also be determined based on the wall line detection result. The specific wall line detection process may refer to the prior art and will not be described in detail here. The three-dimensional pose is based on a reference coordinate system, which may be a world coordinate system.

[0041] In the above 102, displaying the 3D scene model means displaying the 3D scene model on the user interface. Specifically, a 2D image rendering may be performed on the 3D scene model from a target observation perspective to obtain a 2D image; and the 2D image is displayed on the user interface.

[0042] In one example, the at least one recommended product may be recommended based on the current hot product. For example, the current hot product may be recommended to the user as the at least one recommended product.

[0043] In another example, the at least one recommended product may be recommended based on the decoration style of the indoor space determined based on the panoramic image. In actual application, the panoramic image may be input into a trained decoration style recognition model to obtain a recognition result output by the device style recognition model; the decoration style of the indoor space may be determined based on the recognition result; and based on the decoration style, at least one recommended product matching the home decoration style may be determined.

[0044] In actual application, displaying at least one recommended product means displaying product information of at least one recommended product, and the product information may include: one or more of: product name, product image, product purchase link, product purchase control, and product selection control. The product information of the at least one recommended product may be displayed in the form of a list.

[0045] In the above 103, the selection operation may be a single-click operation, a long-press operation, a double-click operation, etc. In one example, the selection operation may be a triggering operation of a user on a product selection control corresponding to a first recommended product among at least one recommended product.

[0046] In a feasible scheme, in order to improve the rationality of the placement of the three-dimensional object model corresponding to the first recommended product, the three-dimensional pose of the three-dimensional object model corresponding to the first recommended product can be determined according to the placement characteristics of the first recommended product and the three-dimensional pose of the three-dimensional space structure model and the three-dimensional pose of the three-dimensional object model of the object in the three-dimensional scene model; and the three-dimensional pose of the three-dimensional object model corresponding to the first recommended product can be placed in the three-dimensional scene model according to the three-dimensional pose of the three-dimensional object model corresponding to the first recommended product.

[0047] In the above 104, displaying the updated 3D scene model means displaying the updated 3D scene model on the display. In practical application, the 3D scene model and the updated 3D scene model can be displayed side by side on the display; or the updated 3D scene model can be displayed on the display in a manner of replacing the 3D scene model.

[0048] In the technical solution provided in the embodiment of the present application, the three-dimensional scene model displayed to the user is determined based on the real-life image captured in the indoor space and input by the user, that is, the three-dimensional scene model is more in line with the actual decoration of the user's indoor space. After the user selects a recommended product from at least one recommended product, the three-dimensional model of the selected recommended product is added to the three-dimensional scene model, and the updated three-dimensional scene model is obtained and displayed to the user. In this way, the user can intuitively feel whether the recommended product he selected matches the actual decoration of his indoor space. In other words, the technical solution provided in the embodiment of the present application improves the intuitiveness of the product recommendation solution, thereby improving the efficiency of users in selecting products.

[0049] In an achievable solution, there are multiple items; the multiple items include the first item; and the “displaying the three-dimensional scene model and at least one recommended product” in 102 above includes:

[0050] 1021. Display the three-dimensional scene model.

[0051] 1022. In response to a replacement operation triggered by the user on the three-dimensional object model of the first object, determine the at least one recommended product according to the object category to which the first object belongs.

[0052] 1023. Display the at least one recommended product.

[0053] A three-dimensional scene model can be displayed on a user interface, and a replacement label can be displayed in association with the three-dimensional object model of each object on the user interface; in response to a click operation triggered by a user on the user interface on a replacement label corresponding to the three-dimensional object model of a first object, at least one recommended product is determined according to the object category to which the first object belongs.

[0054] Specifically, the following method can be used to associate and display the replacement labels corresponding to each item: the mapping point on the user interface of the designated point on the three-dimensional item model of each item in the three-dimensional scene model can be determined according to the coordinate conversion relationship between the reference coordinate system where the three-dimensional scene model is located and the pixel coordinate system where the user interface is located; the replacement label corresponding to each item is displayed at the mapping point on the user interface. The above-mentioned designated point can be any point on the three-dimensional item model of the item, which can be set according to actual needs, and the embodiment of the present application does not make specific restrictions on this. In one example, the above-mentioned designated point can be the center point on the three-dimensional item model of the item, so that the display area of ​​the three-dimensional item model of each item on the user interface and the display area of ​​the replacement label corresponding to each item on the user interface are more closely associated, which is convenient for user operation and improves user experience. In addition, text can be displayed in the above-mentioned replacement label, and the text can specifically include "replacement".

[0055] According to the item category to which the first item belongs, the at least one recommended item is determined. Specifically, multiple three-dimensional item models to be selected under the item category to which the first item belongs can be determined from the item model library; multiple items corresponding to the multiple three-dimensional item models to be selected under the item category to which the first item belongs are used as multiple items to be selected under the item category to which the first item belongs; and at least one recommended item is determined from the multiple items to be selected under the item category to which the first item belongs. Note: Each three-dimensional item model in the item model library corresponds to a corresponding item.

[0056] In one example, a first display area and a second display area may be divided on the user interface; a three-dimensional scene is displayed in the first display area, and at least one recommended product is displayed in the second area.

[0057] Optionally, the step of “determining the at least one recommended product according to the category to which the first product belongs” in 1022 may be implemented by one or more of the following recommendation methods:

[0058] Method 1: determining the at least one recommended product from a plurality of candidate products in the product category to which the first product belongs according to the product pixel areas of the other products except the first product among the plurality of products in the panoramic image.

[0059] Method 2: determining the at least one recommended product from a plurality of candidate products under the product category to which the first product belongs according to the product pixel areas of the plurality of products in the panoramic image.

[0060] In one feasible solution, the panoramic image can be used to identify objects and obtain object identification results; the object identification results include object categories and object pixel areas. Specifically, a target recognition model based on a neural network can be used to identify objects in the panoramic image and obtain object identification results. The object pixel area is also the object detection frame area. The target recognition model can be trained based on a sample panoramic image, wherein the sample panoramic image is marked with real object categories and real detection frames. The specific training process can refer to the existing technology and will not be described in detail here.

[0061] The pixel area of ​​each object can be segmented from the panoramic image to obtain a segmentation map of each object; and the segmentation map of each object can be converted into a perspective map of each object.

[0062] In the first embodiment, the at least one recommended commodity can be determined from a plurality of to-be-selected commodities in the commodity category to which the first commodity belongs based on perspective views of other commodities among the plurality of commodities except the first commodity.

[0063] Specifically, the multiple items to be selected include a first item to be selected; the matching degree of the first item to be selected can be determined based on the perspective views of the items other than the first item among the multiple items and at least one view of the three-dimensional item model corresponding to the first item to be selected; at least one recommended item is determined based on the matching degrees of the multiple items to be selected. For example: at least one item to be selected whose matching degree is greater than a preset matching degree threshold can be used as at least one recommended item. The above matching degree determination process can be calculated by a trained first matching degree calculation model, and this application does not make specific limitations on this. The perspective views of the items other than the first item among the multiple items and at least one view of the three-dimensional item model corresponding to the first item to be selected can be input into the first matching degree calculation model, and the first matching degree calculation model outputs the matching degree of the first item to be selected.

[0064] In the second method, the at least one recommended commodity can be determined from a plurality of to-be-selected commodities under the commodity category to which the first commodity belongs according to perspective views of a plurality of the commodities.

[0065] Specifically, the multiple items to be selected include a first item to be selected; the matching degree of the first item to be selected can be determined based on the perspective views of the multiple items and at least one view of the three-dimensional item model corresponding to the first item to be selected; at least one recommended item is determined based on the matching degree of the multiple items to be selected. For example: at least one item to be selected whose matching degree is greater than a preset matching degree threshold can be used as at least one recommended item. The above matching degree determination process can be calculated by a trained second matching degree calculation model, and this application does not make specific limitations on this. The perspective views of the multiple items and at least one view of the three-dimensional item model corresponding to the first item to be selected can be input into the second matching degree calculation model, and the second matching degree calculation model outputs the matching degree of the first item to be selected.

[0066] In the above method 1, only the pixel areas of other items in the indoor space are considered. Although the recommended products match other items, they are not necessarily similar to the first item. In the above method 3, the pixel areas of the first item and the pixel areas of other items are comprehensively considered. The recommended products not only match other surrounding items, but are also similar to the first item. Note: Both of the above recommendation methods can use retrieval technology based on image search.

[0067] In actual application, the above two recommendation methods can be provided to users for selection. Subsequently, corresponding product recommendations are performed according to the recommendation method selected by the user.

[0068] Accordingly, in the above 103, “in response to a selection operation triggered by the user on a first recommended product among the at least one recommended product, adding a three-dimensional object model corresponding to the first recommended product to the three-dimensional scene model to obtain an updated three-dimensional scene model” may specifically include:

[0069] In response to a selection operation triggered by the user for the at least one recommended product, the three-dimensional model of the selected recommended product is added to the three-dimensional scene model according to the three-dimensional pose of the three-dimensional object model of the first object in the three-dimensional scene model, and the three-dimensional object model of the first object is deleted from the three-dimensional scene model to obtain an updated three-dimensional scene model.

[0070] That is, after adding, the three-dimensional pose of the three-dimensional model of the selected recommended product is consistent with the three-dimensional pose of the three-dimensional object model of the first object.

[0071] Furthermore, the above method may also include:

[0072] 105. Determine the texture of each structural surface in the spatial structure of the indoor space based on the panoramic image.

[0073] In the above 105, the structural surface may include three surface categories: ground, wall and top. According to the wall line detection result, the surface category of each structural surface on the spatial structure and the structural surface pixel area of ​​each structural surface in the panoramic image may be determined.

[0074] The structural surface pixel area of ​​each structural surface can be segmented from the panoramic image to obtain the segmentation map of each structural surface; according to the position of the item pixel area of ​​each item in the panoramic image, the pixels of the item pixel area of ​​each item are removed from the segmentation map of each structural surface to obtain the processed segmentation map of each structural surface; according to the processed segmentation map of each structural surface and the surface category of each structural surface, matching is performed in the texture library to determine the texture of each structural surface. For example: according to the processed segmentation map of each structural surface, multiple candidate textures under the surface category to which the structural surface belongs in the texture library are matched to determine the texture of the structural surface.

[0075] Correspondingly, in the above-mentioned multiple recommendation methods, the texture of each structural surface can also be combined to make product recommendations, so that it can be ensured that the recommended products can match the texture of each structural surface.

[0076] Specifically, the above “determining the at least one recommended product from a plurality of selected products under the product category to which the first product belongs according to the product pixel areas of the plurality of products in the panoramic image” includes:

[0077] The at least one recommended product is determined from a plurality of to-be-selected products in an product category to which the first product belongs according to the object pixel areas of the plurality of objects in the panoramic image and the textures of the structural surfaces of the spatial structure.

[0078] Specifically, the at least one recommended product may be determined from a plurality of candidate products under the product category to which the first product belongs based on perspective views of a plurality of the products and texture maps corresponding to textures of each structural surface on the spatial structure.

[0079] Specifically, the multiple selected commodities include a first selected commodity; the matching degree of the first selected commodity can be determined according to the perspective views of the multiple commodities, the texture map corresponding to the texture of each structural surface on the spatial structure, and at least one view of the three-dimensional object model corresponding to the first selected commodity; at least one recommended commodity is determined according to the matching degree of the multiple selected commodities. For example: at least one selected commodity whose matching degree is greater than a preset matching degree threshold can be used as at least one recommended commodity. The above matching degree determination process can be calculated by a trained third matching degree calculation model, and this application does not make specific limitations on this. The perspective views of the multiple commodities, the texture map corresponding to the texture of each structural surface on the spatial structure, and at least one view of the three-dimensional object model corresponding to the first selected commodity can be input into the third matching degree calculation model, and the third matching degree calculation model outputs the matching degree of the first selected commodity.

[0080] In one feasible solution, the step of “generating a three-dimensional scene model of the indoor space according to the panoramic view of the indoor space” in 101 above can be implemented by the following steps:

[0081] 1011. Determine the three-dimensional spatial structure model and the three-dimensional object model of the object based on the panoramic image.

[0082] 1012. Determine the target structural surface contacted by the object and the first relative position of the object on the target structural surface according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel areas of each structural surface on the spatial structure.

[0083] 1013. Determine a second relative position of the three-dimensional object model of the object on the target model structure plane in the three-dimensional space structure model according to the first relative position of the object on the target structure plane.

[0084] The target model structure surface is a model structure surface in the three-dimensional space structure model corresponding to the target structure surface.

[0085] 1014. According to the second relative position, combine the three-dimensional space structure model and the three-dimensional object model of the object to obtain a three-dimensional scene model of the indoor space.

[0086] In the above 1011, the specific implementation of the step of determining the three-dimensional spatial structure model based on the panoramic image can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0087] In one example, the step of “determining a three-dimensional object model of the object based on the panoramic image” in 1011 can be implemented by the following steps:

[0088] S11, performing object recognition on the panoramic image to obtain an object recognition result.

[0089] The object recognition result includes the object category of the object and the object pixel area of ​​the object.

[0090] S12: According to the object pixel area of ​​the object and the object category, a target three-dimensional model is obtained by matching in an object model library.

[0091] S13: Using the target three-dimensional model as the three-dimensional object model of the object.

[0092] In the above S11, the object identification process can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0093] In the above S12, the object pixel area of ​​the object can be segmented from the panoramic image to obtain a segmentation map of the object; and the segmentation map of the object can be converted into a perspective map of the object.

[0094] According to the item category of the item, multiple candidate 3D item models under the item category can be determined in the item model library; according to the perspective view of the item, multiple candidate 3D item models under the item category are matched to obtain a target 3D item model. The similarity between the perspective view of the item and at least one view of each candidate 3D item model can be calculated, and the 3D item model with the greatest similarity is used as the target 3D item model.

[0095] In another example, the step of “determining a three-dimensional object model of the object based on the panoramic image” in 1011 can be implemented by the following steps:

[0096] S21. Determine a perspective view corresponding to the panoramic view according to the panoramic view.

[0097] The perspective image completely includes the image of the object.

[0098] S22: performing semantic segmentation on the perspective image corresponding to the panoramic image to obtain a semantic segmentation result.

[0099] S22. Determine a three-dimensional object model of the object using a monocular three-dimensional reconstruction algorithm according to the semantic segmentation result.

[0100] In the above S21, the specific implementation process of converting the panoramic image into the perspective image can be referred to the prior art, and will not be repeated here.

[0101] In the above S22, a semantic segmentation model based on a neural network may be used to perform semantic segmentation on the perspective image corresponding to the panoramic image to obtain a semantic segmentation result.

[0102] In the above S23, based on the semantic segmentation result, a monocular 3D reconstruction algorithm is used to reconstruct the 3D object model of the object. The specific reconstruction process can refer to the prior art and will not be described in detail here.

[0103] In the above 1012, “determining the target structural surface contacted by the object according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel area of ​​each structural surface on the spatial structure” can be implemented by the following steps:

[0104] S31. Determine a structural surface pixel area overlapping with the item pixel area of ​​the item in the panoramic image according to the position of the item pixel area of ​​the item in the panoramic image and the position of the structural surface pixel areas of each structural surface on the spatial structure.

[0105] S32: Determine the structural surface corresponding to the structural surface pixel area overlapping with the article pixel area of ​​the article as the target structural surface contacted by the article.

[0106] The number of the target structural surfaces may be one or more.

[0107] In the above 1012, “determining a first relative position of the object on the target structural surface according to the position of the object pixel area of ​​the object in the panoramic image and the positions of the structural surface pixel areas of each structural surface on the spatial structure” may include:

[0108] S41, determining an overlapping area between the object pixel area of ​​the object and the structural surface pixel area of ​​the target structural surface according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel area of ​​the target structural surface.

[0109] S42: Determine the first relative position of the object on the target structural surface according to the third relative position of the overlapping area on the target structural surface.

[0110] In the above S42, the first relative position of the object on the target structural surface may be determined according to the third relative position of the center point of the overlapping area on the target structural surface.

[0111] In the above 1014, the three-dimensional space structure model and the three-dimensional object model of the object are combined according to the second relative position to obtain a three-dimensional scene model of the indoor space.

[0112] The three-dimensional space structure model can be used as a reference, and the three-dimensional object model of the object can be placed in the three-dimensional space structure model according to the second relative position, so as to obtain a three-dimensional scene model of the indoor space.

[0113] In one feasible solution, the above step 1014 of “combining the three-dimensional space structure model and the three-dimensional object model of the object according to the second relative position to obtain the three-dimensional scene model of the indoor space” can be implemented by the following steps:

[0114] S51. Determine a three-dimensional position and posture of the three-dimensional object model of the object according to the second relative position and the three-dimensional position and posture of the three-dimensional spatial structure model.

[0115] S52: According to the three-dimensional pose of the three-dimensional object model of the object and the three-dimensional pose of the three-dimensional space structure model, the three-dimensional space structure model and the three-dimensional object model of the object are combined to obtain a three-dimensional scene model of the indoor space.

[0116] In the above S51, the three-dimensional position and posture of the three-dimensional spatial structure model and the three-dimensional position and posture of the three-dimensional object model of the object are both based on a reference coordinate system.

[0117] The three-dimensional position of the three-dimensional object model of the object can be determined according to the second relative position and the three-dimensional position and posture of the three-dimensional spatial structure model. The three-dimensional position and posture of the three-dimensional object model of the object can be determined according to the three-dimensional position of the three-dimensional object model of the object.

[0118] In actual application, if the three-dimensional posture of the three-dimensional object model of the object is determined arbitrarily, the object placement rules in the finally generated three-dimensional scene model will be out of touch with reality, and the user's visual experience will be poor. In order to improve the display effect of the three-dimensional scene model, the above S51 "determine the three-dimensional posture of the three-dimensional object model of the object according to the second relative position and the three-dimensional posture of the three-dimensional space structure model" can be implemented by the following steps:

[0119] Determining a relative posture of the three-dimensional object model of the object with respect to the three-dimensional spatial structure model according to the placement characteristics of the object and the surface category of the target structural surface;

[0120] The three-dimensional position and posture of the three-dimensional object model of the object is determined according to the second relative position, the relative posture of the three-dimensional object model of the object with respect to the three-dimensional spatial structure model, and the three-dimensional position and posture of the three-dimensional spatial structure model.

[0121] Placement characteristics may include: close to the ground, against the wall, and against the ceiling. Surface categories may include: ground, wall, and ceiling.

[0122] For example, if the number of target structural surfaces is two, the two target structural surfaces include the target ground and the target wall. When the placement characteristics of the object are close to the ground and close to the wall (for example, a bed or a toilet), it is determined that the normal vector of the side of the bounding box of the three-dimensional object model of the object that is close to the ground points to the target ground and is perpendicular to the target ground, and the normal vector of the side of the bounding box of the three-dimensional object model of the object that is close to the wall points to the target wall and is perpendicular to the target wall.

[0123] For another example, the number of target structural surfaces is one, and the target structural surfaces include the target top surface. When the placement characteristic of the object is to be attached to the top surface (for example, a ceiling light), the normal vector of the side surface of the bounding box of the three-dimensional object model of the object that is attached to the top points to the target top surface and is perpendicular to the target top surface.

[0124] For another example, the number of target structural surfaces is one, and the target structural surface includes the target ground. When the placement characteristic of the object is to be close to the ground (for example, a dining chair or a dining table), the normal vector of the side of the bounding box of the three-dimensional object model of the object that is close to the ground points to the target ground and is perpendicular to the target ground.

[0125] The method for determining the three-dimensional position and posture of the above-mentioned three-dimensional spatial structure model can refer to the corresponding content in the above-mentioned embodiments, which will not be repeated here.

[0126] As an option, the following method may be used to determine the three-dimensional position and posture of the three-dimensional object model of the object. Specifically,

[0127] S61. Based on the perspective view corresponding to the panoramic view, a monocular depth estimation algorithm is used to determine point cloud data corresponding to the perspective view corresponding to the panoramic view.

[0128] S62: Determine the three-dimensional pose of the object according to the point cloud data to serve as the three-dimensional pose of the three-dimensional object model of the object.

[0129] S63. According to the three-dimensional pose of the three-dimensional object model of the object and the three-dimensional pose of the internal three-dimensional space structure model, combine the internal three-dimensional space structure model and the three-dimensional object model of the object to obtain the three-dimensional scene model.

[0130] In the above S61, the above panorama is obtained based on the monocular camera and has no depth information. Therefore, the perspective view corresponding to the panorama also has no depth information. The depth information can be estimated based on the monocular depth estimation algorithm to obtain the point cloud data corresponding to the perspective view. The specific implementation of the monocular depth estimation algorithm can be referred to the prior art and will not be repeated here.

[0131] In the above S62, the object category and the three-dimensional pose of the object can be obtained by performing point cloud semantic segmentation on the point cloud data. The three-dimensional pose is based on the reference coordinate system. The specific implementation of the point cloud semantic segmentation technology can be referred to the prior art, which will not be described in detail here.

[0132] Optionally, the above method may further include:

[0133] 106. Perform collision detection in the three-dimensional scene model according to the three-dimensional position and posture of the three-dimensional spatial structure model and the three-dimensional position and posture of the three-dimensional object model of the object.

[0134] 107. Optimize the three-dimensional position and posture of the three-dimensional object model of the object according to the collision detection result.

[0135] In the above 106, the three-dimensional pose of the sub-three-dimensional model of each structural surface in the spatial structure is determined according to the three-dimensional pose of the three-dimensional spatial structure model; the bounding box of the sub-three-dimensional model of each structural surface is determined according to the three-dimensional pose of the sub-three-dimensional model of each structural surface and the size of the sub-three-dimensional model of each structural surface; the bounding box of the three-dimensional object model of each object is determined according to the three-dimensional pose of the three-dimensional object model of each object and the size of the object; collision detection is performed in the three-dimensional scene model according to the bounding box of the sub-three-dimensional model of each structural surface and the bounding box of the three-dimensional object model of each object in the multiple objects.

[0136] The specific implementation process of collision detection can refer to the existing technology and will not be repeated here.

[0137] In the above 107, if the collision detection result shows that there is an embedding between the bounding box of the three-dimensional object model of a certain object and other bounding boxes, the three-dimensional pose of the three-dimensional object model of the object is optimized to eliminate the embedding between the bounding box of the three-dimensional object model of the object and other bounding boxes.

[0138] Optionally, the above method may further include:

[0139] 108. According to the texture of each structural surface of the spatial structure in the panoramic image, set the texture of each structural surface of the model in the three-dimensional spatial structure model in the three-dimensional scene model.

[0140] In this way, the generated three-dimensional scene model is more in line with the actual situation, which helps users see the actual home decoration effect after the furniture is replaced.

[0141] Figure 2A flow chart of a three-dimensional scene model generation method provided by another embodiment of the present application is shown. The executor of the method may be a client or a server. The client may be a hardware with an embedded program integrated on a terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the present embodiment of the application. The terminal may be any terminal device including a mobile phone, a tablet computer, a vehicle-mounted terminal device, etc. The server may be a common server, a cloud or a virtual server, etc., which is not specifically limited in the present embodiment of the application. Figure 2 As shown, the method includes:

[0142] 201. Determine a three-dimensional spatial structure model of the indoor space and a three-dimensional object model of an object in the indoor space based on a panoramic view of the indoor space.

[0143] 202. Determine a target structural surface contacted by the object and a first relative position of the object on the target structural surface according to a position of an object pixel area of ​​the object in the panoramic image and a position of a structural surface pixel area of ​​each structural surface on the spatial structure.

[0144] 203. Determine a second relative position of the three-dimensional object model of the object on the target model structure plane in the three-dimensional space structure model according to the first relative position of the object on the target structure plane.

[0145] The target model structure surface is a model structure surface in the three-dimensional space structure model corresponding to the target structure surface.

[0146] 204. According to the second relative position, combine the three-dimensional space structure model and the three-dimensional object model of the object to obtain a three-dimensional scene model of the indoor space.

[0147] The specific implementation of the above steps 201 to 204 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0148] In the technical solution provided by another embodiment of the present application, the target structural surface contacted by the object and the first relative position of the object on the target structural surface are determined according to the position of the structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space in the panoramic view of the indoor space and the position of the item pixel area of ​​the item in the indoor space; the second relative position of the three-dimensional item model of the object on the structural surface of the target model in the three-dimensional spatial structure model is determined according to the first relative position of the item on the target structural surface; according to the second relative position, the three-dimensional spatial structure model and the three-dimensional item model of the item are combined to obtain the three-dimensional scene model of the indoor space. The technical solution provided by the embodiment of the present application can improve the positioning accuracy of the three-dimensional item model of the object in the three-dimensional spatial structure model, thereby improving the restoration degree of the three-dimensional scene model.

[0149] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0150] The following is an example of the technical solution provided in the embodiment of the present application:

[0151] Step 1: Spatial structure detection.

[0152] This step is to obtain spatial structure information. Input a panoramic image of the indoor space, perform wall line detection based on the image pixel clues, and then estimate the structural information of the house, including the position of the wall, ground, and ceiling. The wall line detection result is as follows: Figure 3 shown.

[0153] Step 2: Furniture inspection.

[0154] This step is to detect the furniture category and two-dimensional position of furniture in the indoor space. Input a panoramic image of the indoor space, detect the 23 defined furniture categories, and return the position of the detection box of each furniture in the panoramic image. Among them, the furniture detection box is as follows Figure 3 shown.

[0155] Step 3: Furniture model matching.

[0156] This step is to match the furniture in the panoramic image with the furniture models in the furniture model library. The specific process is: input the pixels in the furniture detection box obtained by furniture detection in the panoramic image, and obtain the perspective image of the furniture through the coordinate transformation of the panoramic image to the perspective image; according to the furniture category of the furniture, perform image search on this perspective image in the corresponding category of the furniture model library to obtain the most matching furniture model. The furniture model library here contains the two-dimensional view of each three-dimensional furniture model, such as Figure 4 As shown; the furniture model corresponds to the furniture product itself, such as tables, chairs, cabinets and beds, etc., and corresponds to the product link.

[0157] Step 4: Wall texture matching.

[0158] This step is to match the textures of the walls, floors and ceilings in the panoramic image with the material textures in the texture library. The specific process is: according to the spatial structure detection results, the panoramic image is divided into different structural surfaces to obtain two-dimensional images of different structural surfaces, including two-dimensional images of the floor, ceiling and wall respectively; the pixels belonging to the furniture area in the two-dimensional image are extracted to obtain the extracted plane texture map; in the material texture library, according to the classification of floors, ceilings and walls, image search is performed to obtain the most matching material texture; the purpose of steps 4 and 3 is to restore the furniture and structural surface textures that are consistent with the original decoration style. Here, the structural surface texture corresponds to the product itself, such as wallpaper, wall paint and floor materials, etc., corresponding to the product link. Texture libraries such as Figure 5 The matching algorithm used in this step is the same as that in step 3, except that the training data is labeled and the training strategy is different.

[0159] Step 5: Three-dimensional positioning of furniture.

[0160] This step is to estimate the 3D furniture pose from the 2D position. The specific process is: input the detected spatial structure information and furniture detection results, locate the 2D position of the furniture detection to the wall or ground according to the spatial structure information; determine the direction and position of the furniture according to whether the furniture has the characteristic of being placed along the wall; obtain the 3D boundary of the furniture, perform 3D collision detection on furniture and furniture, and furniture and house; optimize the position and size of the furniture according to the results of the collision detection.

[0161] Step 6: Reconstruct the 3D scene model.

[0162] This step is to obtain a 3D model with furniture for preview. The specific process is: convert the indoor space structure into a triangular patch structure as the basic 3D space structure model; set the texture of the floor, ceiling and wall in the 3D space structure model according to the texture obtained in step 4; perform coordinate transformation according to the 3D furniture model and its 3D posture determined in steps 3 and 5, merge the 3D furniture model with the 3D space structure model, and obtain a 3D scene model for preview. Corresponding replacement tags (also called product link tags) can also be set at the 3D center position of the wall, floor, ceiling and furniture model.

[0163] Step 7: Furniture product recommendations.

[0164] This step recommends furniture products with the same style based on the existing wall and floor textures and furniture matching. Users can specify a single piece of furniture and make recommendations for it. The 2D image sequence of the remaining furniture except the furniture can be used as the matching prediction input, and at least one suitable furniture product is recommended from the existing furniture model library according to the matching index from high to low. Users only need to select a target furniture product from the recommended furniture products to see the replacement effect.

[0165] If the user specifies different furniture, different furniture products will be recommended. The above step 7 can be executed repeatedly according to the user's specified operation.

[0166] Figure 6 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0167] The memory 1101 is used to store programs;

[0168] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the commodity recommendation and three-dimensional scene model generation methods provided by the above-mentioned method embodiments.

[0169] Further, if Figure 6As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0170] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the product recommendation and three-dimensional scene model generation methods provided in the above-mentioned method embodiments.

[0171] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A product recommendation method, wherein: include: Generating a three-dimensional scene model of the indoor space according to the panoramic view of the indoor space; The three-dimensional scene model includes a three-dimensional spatial structure model of the indoor space and a three-dimensional object model of an object in the indoor space; the panoramic image is determined according to a real scene image captured in the indoor space and input by a user; the indoor space includes a first object; displaying the three-dimensional scene model, wherein the three-dimensional scene model includes a three-dimensional object model of the first object; In response to a replacement operation triggered by the user on the three-dimensional object model of the first object, determining at least one recommended item from a plurality of to-be-selected items in an item category to which the first object belongs; displaying the at least one recommended product; In response to a selection operation triggered by the user on a first recommended product among the at least one recommended product, adding a three-dimensional object model corresponding to the first recommended product to the three-dimensional scene model to obtain an updated three-dimensional scene model; The updated three-dimensional scene model is displayed.

2. The method according to claim 1, wherein: There are multiple items; the multiple items include the first item; In response to a replacement operation triggered by the user on the three-dimensional object model of the first object, at least one recommended item is determined from a plurality of items to be selected under the item category to which the first object belongs, including: in response to a replacement operation triggered by the user on the three-dimensional object model of the first object, based on item pixel areas of the plurality of items in the panoramic image, determining the at least one recommended item from a plurality of items to be selected under the item category to which the first object belongs.

3. The method according to claim 2, wherein: Also includes: Based on the panoramic image, determining the texture of each structural surface of the spatial structure of the indoor space; Determining the at least one recommended commodity from a plurality of to-be-selected commodities in the commodity category to which the first commodity belongs according to the commodity pixel areas of the plurality of commodities in the panoramic image comprises: The at least one recommended product is determined from a plurality of to-be-selected products in an product category to which the first product belongs according to the object pixel areas of the plurality of objects in the panoramic image and the textures of the structural surfaces of the spatial structure.

4. The method according to any one of claims 1 to 3, wherein: Generating a three-dimensional scene model of the indoor space according to the panoramic view of the indoor space includes: Determine the three-dimensional spatial structure model and the three-dimensional object model of the object based on the panoramic image; Determining a target structural surface contacted by the object and a first relative position of the object on the target structural surface according to a position of an object pixel area of ​​the object in the panoramic image and a position of a structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space; Determine, according to the first relative position of the object on the target structural surface, a second relative position of the three-dimensional object model of the object on the target model structural surface in the three-dimensional spatial structure model; wherein the target model structural surface is a model structural surface in the three-dimensional spatial structure model corresponding to the target structural surface; The three-dimensional space structure model and the three-dimensional object model of the object are combined according to the second relative position to obtain a three-dimensional scene model of the indoor space.

5. The method according to claim 4, wherein: Determining a target structural surface contacted by the object according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space includes: Determining a structural surface pixel area overlapping with the item pixel area of ​​the item in the panoramic image according to the position of the item pixel area of ​​the item and the position of the structural surface pixel area of ​​each structural surface on the spatial structure; A structural surface corresponding to a structural surface pixel area overlapping with an article pixel area of ​​the article is determined as a target structural surface contacted by the article.

6. The method according to claim 4, determining the first relative position of the object on the target structural surface according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space, comprising: Determining an overlapping area between the object pixel area of ​​the object and the structural surface pixel area of ​​the target structural surface according to the position of the object pixel area of ​​the object in the panoramic image and the position of the structural surface pixel area of ​​the target structural surface; The first relative position of the object on the target structural surface is determined according to the third relative position of the overlapping area on the target structural surface.

7. The method according to claim 4, wherein: According to the second relative position, combining the three-dimensional space structure model and the three-dimensional object model of the object to obtain a three-dimensional scene model of the indoor space includes: Determining a three-dimensional pose of the three-dimensional object model of the object according to the second relative position and the three-dimensional pose of the three-dimensional spatial structure model; According to the three-dimensional pose of the three-dimensional object model of the object and the three-dimensional pose of the three-dimensional space structure model, the three-dimensional space structure model and the three-dimensional object model of the object are combined to obtain a three-dimensional scene model of the indoor space.

8. The method according to claim 7, wherein: Determining the three-dimensional pose of the three-dimensional object model of the object according to the second relative position and the three-dimensional pose of the three-dimensional spatial structure model includes: Determining a relative posture of the three-dimensional object model of the object with respect to the three-dimensional spatial structure model according to the placement characteristics of the object and the surface category of the target structural surface; The three-dimensional position and posture of the three-dimensional object model of the object is determined according to the second relative position, the relative posture of the three-dimensional object model of the object with respect to the three-dimensional spatial structure model, and the three-dimensional position and posture of the three-dimensional spatial structure model.

9. The method according to claim 7, wherein: Also includes: Performing collision detection in the three-dimensional scene model according to the three-dimensional position and posture of the three-dimensional spatial structure model and the three-dimensional position and posture of the three-dimensional object model of the object; The three-dimensional position and posture of the three-dimensional object model of the object is optimized according to the collision detection result.

10. The method according to claim 4, wherein: Also includes: According to the texture of each structural surface of the spatial structure in the panoramic image, the texture of each model structural surface in the three-dimensional spatial structure model is set in the three-dimensional scene model.

11. The method according to claim 4, wherein: Determining a three-dimensional object model of the object based on the panoramic image includes: According to the panoramic image, determining a perspective image corresponding to the panoramic image; Using a neural network-based semantic segmentation model to perform semantic segmentation on the perspective image corresponding to the panoramic image to obtain a semantic segmentation result; According to the semantic segmentation result, a monocular 3D reconstruction algorithm is used to determine the 3D object model of the object.

12. A method for generating a three-dimensional scene model, wherein: include: Based on the panoramic view of the indoor space, determining a three-dimensional spatial structure model of the indoor space and a three-dimensional object model of an object in the indoor space; Determining a target structural surface contacted by the object and a first relative position of the object on the target structural surface according to a position of an object pixel area of ​​the object in the panoramic image and a position of a structural surface pixel area of ​​each structural surface on the spatial structure of the indoor space; Determine, according to the first relative position of the object on the target structural surface, a second relative position of the three-dimensional object model of the object on the target model structural surface in the three-dimensional spatial structure model; wherein the target model structural surface is a model structural surface in the three-dimensional spatial structure model corresponding to the target structural surface; The three-dimensional space structure model and the three-dimensional object model of the object are combined according to the second relative position to obtain a three-dimensional scene model of the indoor space.

13. The method according to claim 12, wherein: According to the second relative position, combining the three-dimensional space structure model and the three-dimensional object model of the object to obtain a three-dimensional scene model of the indoor space includes: Determining a three-dimensional pose of the three-dimensional object model of the object according to the second relative position and the three-dimensional pose of the three-dimensional spatial structure model; According to the three-dimensional pose of the three-dimensional object model of the object and the three-dimensional pose of the three-dimensional space structure model, the three-dimensional space structure model and the three-dimensional object model of the object are combined to obtain a three-dimensional scene model of the indoor space.

14. An electronic device, wherein: include: A memory and a processor, wherein The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a computer, the method according to any one of claims 1 to 13 can be implemented.

Images