A method, an electronic device, and a storage medium for constructing a three-dimensional design scene portrait

Abstract

The present invention provides a method for constructing a three-dimensional design scene portrait, comprising the following steps: obtaining offline design data and real-time design data within a three-dimensional design scene, and fusing the two parts of obtained design data; preprocessing the three-dimensional space information of the three-dimensional design scene and the design material information within the scene; constructing a two-tower model, and using the two-tower model structure to map the feature representation of the three-dimensional scene and the design material representation in the same space for feature alignment; respectively performing feature characterization on the three-dimensional scene and feature extraction on the design material according to the two-tower model structure; introducing the feature vectors after feature characterization and feature extraction into the recall and ranking stages of recommendation and search, and returning personalized recommendations and search results for different three-dimensional design scenes. The method for constructing a three-dimensional design scene portrait of the present invention can perform "a thousand scenes, a thousand faces" recommendations for different three-dimensional design scenes, and improve the efficiency of users in finding design materials.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional design technology, and in particular to a three-dimensional design scene portrait construction method, electronic equipment and storage medium. Background Art

[0002] The existing mainstream portrait technology mainly builds user portraits based on user behavior data, and the portraits of the same user in different scenarios tend to be consistent. In a three-dimensional design scenario, the same user, especially a designer, will undertake different design tasks, so the portraits of different three-dimensional design scenarios will also be different. The portrait of the three-dimensional design scenario is crucial to the construction of personalized search and recommendation capabilities for the current design scenario.

[0003] In the existing three-dimensional design systems, no three-dimensional design scene portrait has been established.

[0004] Since some other 3D design systems currently do not have a portrait of the 3D design scene, they are unable to provide different recommended materials based on each 3D design scene, nor can they provide search and recommendation capabilities for “thousands of scenes and thousands of faces”. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a method and system for constructing a 3D design scene portrait, which depicts the 3D design scene portrait according to the content in the 3D design scene, including the 3D spatial relationship and the category, color, material, style, image features, 3D information, etc. of the design material. Based on the 3D design scene portrait, different 3D design scenes are recommended with "thousands of scenes and thousands of faces" and the search capability is improved, thereby improving the efficiency of users in finding design materials.

[0006] To achieve the above object, the present invention provides a method for constructing a three-dimensional design scene portrait, comprising the following steps:

[0007] Obtain offline design data and real-time design data in the 3D design scene, and fuse the two parts of the acquired design data;

[0008] Preprocessing the three-dimensional space information of the three-dimensional design scene and the design material information in the scene;

[0009] Construct a twin-tower model and use the twin-tower model structure to map the feature representation of the 3D scene and the design material representation into the same space for feature alignment;

[0010] Characterize the three-dimensional scene and extract features from the design materials based on the twin-tower model structure;

[0011] The feature vectors after feature characterization and feature extraction are introduced into the recall and ranking stages of recommendation and search, and personalized recommendations and search results are returned for different 3D design scenarios.

[0012] Further, the step of obtaining the offline design data and real-time design data in the 3D design scenario and fusing the two parts of design data obtained further includes,

[0013] The offline design data includes material-related information and 3D scene space information in the 3D design scenario;

[0014] The real-time design data includes material information in the 3D design scenario characterized by the scheme features at a specific moment;

[0015] After obtaining the offline design data and the real-time design data, a set merge of the design material content is performed according to the corresponding operations of the design material data.

[0016] Further, it further includes that the 3D scene space information includes feature extraction of the house type, the SCAN-N algorithm is used for feature extraction, the house type is Figure 2 quantified, the house type diagram is segmented, the pixel values are counted according to the segmentation, and the pixel values are formed into a one-dimensional vector.

[0017] Further, the step of preprocessing the 3D space information and the design material information in the 3D design scenario further includes,

[0018] The image is subjected to image feature extraction using a pre-trained image feature extraction model, and color classification and 3D coordinate information discretization are performed using a pre-trained color classification model.

[0019] Further, it further includes,

[0020] The image is input into the pre-trained image feature extraction model, and a one-dimensional vector is output. The extracted image features are the one-dimensional vector output by the model;

[0021] The image is input into the pre-trained color classification model, and the probability value of each classification is output. A one-dimensional vector is formed by the probability values.

[0022] Further, the step of respectively performing feature characterization on the 3D scene and feature extraction on the design material according to the dual-tower model structure further includes,

[0023] Using the 3D scene tower in the dual-tower model structure, according to the obtained 3D scene data, feature characterization of the 3D scene is performed to produce a feature vector of the 3D scene;

[0024] After the design materials are updated, the design material tower in the two-tower model structure is used to characterize the features of the design materials according to the information of the obtained design materials, and the feature vectors of the design materials are produced.

[0025] Furthermore, the step of introducing the feature vectors after feature characterization and feature extraction into the recall and ranking stages of recommendation and search, and returning personalized recommendations and search results for different 3D design scenarios, further includes

[0026] extracting 3D scene characterization features according to the designed 3D scheme, scoring the content of the material library using a vector retrieval engine, and fusing the scores with the scores of text recall for recall;

[0027] introducing the scene characterization features into the search ranking deep model for ranking and scoring and then searching.

[0028] Even further, the step of introducing the scene characterization features into the search ranking deep model for ranking and scoring and then searching further includes, after the deep ranking model is trained, inputting the features of each material into the deep ranking model, and the deep ranking model scores the materials, and sorting the list of materials participating in the ranking according to the scores of each material.

[0029] To achieve the above object, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program running on the processor, and the processor executes the steps of the above-mentioned 3D design scene portrait construction method when running the program.

[0030] To achieve the above object, the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions execute the steps of the above-mentioned 3D design scene portrait construction method when running.

[0031] The 3D design scene portrait construction method and system of the present invention have the following beneficial effects:

[0032] 1) Characterize the 3D design scene portrait according to the content in the 3D design scene, including 3D spatial relationships and categories, colors, materials, styles, image features, 3D information, etc. of the design materials. Based on this 3D design scene portrait, perform "thousands of scenes, thousands of faces" recommendations for different 3D design scenes and improve the search ability, thereby improving the efficiency of users to find design materials.

[0033] 2) Effectively establish portraits for three-dimensional design scenarios, making basic feature preparations for "thousands of scenarios with thousands of faces"; effectively utilize the three-dimensional information of the three-dimensional design scenarios and the three-dimensional information of the design materials within the scenarios; after the three-dimensional design scenarios are densely vectorized, they can be quickly applied to the recall and recommendation of search recommendations, and can be conveniently input into other complex models for various modeling.

[0034] Other features and advantages of the present invention will be described in the following specification, and partly will be obvious from the specification, or will be understood by implementing the present invention. Brief Description of the Drawings

[0035] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and together with the embodiments of the present invention, are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0036] Figure 1 It is a flowchart of a method for constructing a three-dimensional design scenario portrait according to an embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the set transformer structure adopted by the method for constructing a three-dimensional design scenario portrait according to an embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of the model structure in the model training stage of scene characterization in the construction of a three-dimensional design scenario portrait according to an embodiment of the present invention. Detailed Embodiments

[0039] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0040] Embodiment 1

[0041] Figure 1 It is a flowchart of a method for constructing a three-dimensional design scenario portrait according to the present invention. The following will refer to Figure 1 and describe in detail the method for constructing a three-dimensional design scenario portrait of the present invention.

[0042] In step 101, obtain the offline design data and real-time design data in the three-dimensional design scenario, and fuse the two parts of the obtained design data.

[0043] Preferably, the offline design data is mainly the material-related information and three-dimensional scene space information (house type information) in the three-dimensional design scenario of the previous day, and the real-time design data is mainly the material information in the three-dimensional design scenario as of the moment of scheme feature characterization on the current day.

[0044] Preferably, after obtaining the offline design data and the real-time design data, for the corresponding operations on these design material data, perform a set merge of the design material content (for the deleted design materials, remove the materials from the design material content set; for the newly added design materials, add the materials to the design material content set). After determining the set of design material content, obtain the material information of the set, including material id, category, image, color, three-dimensional coordinates, etc.

[0045] Preferably, the three-dimensional scene space information (house type information) includes a feature extraction of the house type, and this feature is the feature extracted by the SCAN-N algorithm.

[0046] Preferably, the steps of extracting features by SCAN-N can be specifically executed as follows:

[0047] 1. Quantize the house type Figure 2 value.

[0048] 2. Divide the house type diagram into blocks;

[0049] 3. Count the pixel values according to the blocks;

[0050] 4. Compose the pixel values into a one-dimensional vector.

[0051] In step 102, in the data processing stage, preprocess the three-dimensional space information of the three-dimensional design scene and the category, color, material, style, image, and three-dimensional coordinate information of the design materials in the scene.

[0052] Preferably, perform operations such as image feature extraction on the image using a pre-trained image feature extraction model (resnet), color classification using a pre-trained color classification model, and discretization of the three-dimensional coordinate information.

[0053] Preferably, Resnet is a pre-trained deep learning model. The input is an image and the output is a one-dimensional vector. The extracted image features are the one-dimensional vector output by the model.

[0054] Preferably, a classification model obtained by classification training using a pre-trained efficientnet model + softmax layer. The input is an image, and the output is the probability value of each classification. This probability value forms a one-dimensional vector.

[0055] Preferably, the steps of discretizing the three-dimensional coordinate information can be specifically executed as follows:

[0056] 1. Obtain the three-dimensional coordinates and length, width, and height information of the model;

[0057] 2. Grid the three-dimensional space and perform discretization according to the grid space where the three-dimensional coordinates x, y, and z are located.

[0058] In step 103, during the training phase of the scene characterization model, a two-tower model is constructed. The set transformer is used to perform feature representation on the three-dimensional scene, and the feature representation of the three-dimensional scene and the design material representation are mapped to the same space using the two-tower structure for feature alignment.

[0059] Preferably, as Figure 3 shown, it is the model structure in the training phase of the scene characterization model. The three-dimensional scene tower end uses the set transformer to extract features from the three-dimensional scene, ensuring that the feature extraction is independent of the input sequence order of the design materials within the three-dimensional scene. At the same time, the scene space features unique to the three-dimensional scene and the three-dimensional information of the design materials are introduced to meet the characterization of the three spatial dimensions of the three-dimensional scene; at the design material tower end, we also introduce all the features of the design materials, including three-dimensional information.

[0060] In this embodiment, the set transformer adopts the structure proposed in the 2019 ICML conference paper, Set Transformer: A framework for attention-based permutation-invariant Neural Networks, and its structure is as Figure 2 shown. This structure adopts the structure of the encoder end of the transformer, removing the positional encoding layer and the dropout layer, ensuring that it is independent of the order of the material set.

[0061] In step 104, in the feature output stage, the three-dimensional scene is characterized and the design materials are feature-extracted respectively according to the two tower model structures.

[0062] Preferably, for the three-dimensional scene, the three-dimensional scene tower is used in real time. According to the three-dimensional scene data obtained in the data acquisition stage, the feature characterization of the three-dimensional scene is performed to output the feature vector of the three-dimensional scene.

[0063] Preferably, for the design materials, when new materials are uploaded and the material information changes, the design material tower is used. According to the steps in the data acquisition stage, the information of the materials is obtained, the feature characterization of the materials is performed, and the feature vector of the design materials is output.

[0064] In this embodiment, the dimensionality of the feature vector of the three-dimensional scene is the same as that of the feature vector of the design materials, both being 512 dimensions.

[0065] In step 105, in the feature utilization stage, the feature vectors characterized in the feature output stage are introduced into the recall and ranking stages of recommendation and search, and personalized recommendations and search results are returned for different 3D design scenarios.

[0066] Preferably, in the search recall stage, according to the 3D scheme currently designed by the user, 3D scene characterization features are extracted, a vector retrieval engine is used to score the content of the material library, and the scores are fused (weighted sum) with the scores of text recall for recall.

[0067] Preferably, through the ab test framework, multiple groups of parameter settings are made and then online experiments are carried out, and finally appropriate weights are selected; according to the weights, the scores of each material can be obtained, and the materials are sorted according to the scores during recall.

[0068] Preferably, in the search ranking stage, the scene characterization features are introduced into the search ranking deep model for ranking scoring, so as to realize the search of "thousands of scenes with thousands of faces".

[0069] Preferably, the ranking models downstream of 3D scene characterization have different deep models in different scenes.

[0070] Preferably, after the deep ranking model is trained, the features of each material are input to the deep ranking model, and the ranking model will score the material. Finally, the list of materials participating in the ranking is sorted according to the scores of each material.

[0071] In this embodiment, the dense features output in the feature output stage are used to perform personalized recommendations and searches for specific 3D design scenarios, so as to build the search and recommendation ability of "thousands of scenes with thousands of faces".

[0072] In this embodiment, the feature vectors of the 3D scene and the image features of the material can be input to the wide&deep ranking model (the wide&deep model is a search ranking model proposed by Google). After the model transforms the feature vectors of the 3D scene and the image features of the material, it finally outputs the score of the model for the material. This score can be understood as the score given by the model considering the image features of the material itself and the features of the 3D scene of the current design scheme. For example, for materials that match the 3D scene style better, the scores will be higher. After scoring each of the top N materials recalled one by one, the N recalled materials can be sorted again.

[0073] In this embodiment, as Figure 3 shown, Relu is a common activation function layer in the field of deep learning, that is, a relu activation function is added to the output of the neuron; Normalize is to normalize the output vector; Similarity function is to calculate the cosine distance between two vectors.

[0074] The present invention also provides an electronic device, including a memory and a processor. A program running on the processor is stored on the memory. When the processor runs the program, it executes the steps of the above-mentioned three-dimensional design scenario portrait construction method.

[0075] The present invention also provides a computer-readable storage medium, on which computer instructions are stored. When the computer instructions run, they execute the steps of the above-mentioned three-dimensional design scenario portrait construction method. For the introduction of the three-dimensional design scenario portrait construction method, please refer to the foregoing part and will not be elaborated herein.

[0076] Those of ordinary skill in the art can understand that the foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a portrait of a 3D design scene, comprising the following steps: Obtain offline design data and real-time design data within the 3D design scene, and fuse the two parts of the obtained design data; Preprocess the 3D spatial information and the design material information within the 3D design scene; Construct a dual-tower model, and use the dual-tower model structure to map the feature representations of the 3D scene and the design material representations into the same space for feature alignment; Characterize the features of the 3D scene and extract the features of the design materials respectively according to the dual-tower model structure; Introduce the feature vectors after feature characterization and feature extraction into the recall and ranking stages of recommendation and search, and return personalized recommendations and search results for different 3D design scenes; The step of introducing the feature vectors after feature characterization and feature extraction into the recall and ranking stages of recommendation and search, and returning personalized recommendations and search results for different 3D design scenes further includes, According to the designed 3D scheme, extract the 3D scene characterization features, use a vector retrieval engine to score the content of the material library, and fuse the scores with the scores of text recall for recall; Introduce the scene characterization features into the search ranking deep model for ranking scoring and then search; The step of introducing the scene characterization features into the search ranking deep model for ranking scoring and then search further includes that after the deep ranking model is trained, input the features of each material into the deep ranking model, and the deep ranking model scores the materials, and sort the list of materials participating in the ranking according to the scores of each material.

2. The three-dimensional design scenario portrait construction method according to claim 1, characterized in that The step of obtaining offline design data and real-time design data within the 3D design scene and fusing the two parts of the obtained design data further includes, The offline design data includes material-related information and 3D scene spatial information within the 3D design scene; The real-time design data includes the material information within the 3D design scene with the feature characterization of the scheme at a specific moment; After obtaining the offline design data and the real-time design data, perform a set merge of the design material content according to the corresponding operations of the design material data.

3. The three-dimensional design scenario portrait construction method according to claim 2, wherein It further includes that the 3D scene spatial information includes the feature extraction of the house type, uses the SCAN-N algorithm for feature extraction, binarizes the house type map, divides the house type map into blocks, counts the pixel values according to the blocks, and forms a one-dimensional vector with the pixel values.

4. The three-dimensional design scenario portrait construction method according to claim 1, wherein The step of preprocessing the 3D spatial information and the design material information within the 3D design scene further includes, Use a pre-trained image feature extraction model to extract image features for the image, and use a pre-trained color classification model for color classification and 3D coordinate information discretization.

5. The three-dimensional design scenario portrait construction method according to claim 4, characterized in that It further includes, Input the image into the pre-trained image feature extraction model, output a one-dimensional vector, and the extracted image features are the one-dimensional vector output by the model; Input the image into the pre-trained color classification model, output the probability value of each classification, and form a one-dimensional vector with the probability values.

6. The method for constructing a three-dimensional design scenario portrait according to claim 1, wherein The step of characterizing the features of the 3D scene and extracting the features of the design materials respectively according to the dual-tower model structure further includes, Using the three-dimensional scene tower in the two-tower model structure, according to the obtained three-dimensional scene data, perform feature characterization of the three-dimensional scene to produce a feature vector of the three-dimensional scene; When the design material is updated, use the design material tower in the two-tower model structure. According to the information of the obtained design material, perform feature characterization of the design material to produce a feature vector of the design material.

7. An electronic device, characterized in that, It includes a memory and a processor. The memory stores a program that runs on the processor. When the processor runs the program, it executes the steps of the three-dimensional design scene portrait construction method according to any one of claims 1-6.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instructions run, they execute the steps of the three-dimensional design scene portrait construction method according to any one of claims 1-6.

Images