A method and apparatus for generating a three-dimensional memory carrier

Abstract

The present application provides a method and apparatus for generating a three-dimensional memory carrier, including: generating a first style feature vector and a first semantic feature vector based on a first scene description text input by a target user, extracting first keyword information based on a first text to be memorized input by the target user, inputting the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model, generating a first memory carrier modeling based on the first stylized keyword information, generating a first three-dimensional geometric scene modeling based on the first semantic feature vector, and generating a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling, which to a certain extent improves the matching degree between the memory carrier set modeling and the text to be memorized and improves the usability of the memory carrier geometric model.

Description

Technical Field

[0001] The present application relates to the technical field of model processing, and in particular to a method and device for generating a three-dimensional memory carrier. Background Art

[0002] The Memory Palace method is a memory method that originated in ancient Rome. Its principle is to establish associations between the memory content and imaginary buildings, rooms, and interior decorations. By memorizing the building, internal objects, and their locations, the original content can be recalled quickly and orderly. Thousands of years of practice have shown that the Memory Palace method is a very effective memory aid. In modern times, this method has a very wide range of applications. In addition to daily memory content, memory competition contestants use this method to quickly memorize a large amount of information. In addition, memory-impaired patients and the elderly can also use it for cognitive rehabilitation. However, this method has relatively high requirements for the user's imagination, making it difficult for ordinary users to master.

[0003] In related technologies, planar graphics and three-dimensional models are mainly used to assist users in reducing the difficulty of imagining three-dimensional spaces. For example, a user's own memory space can be established, and scenes and memory carriers can be selected, so that associations can be made. In addition, memory training can be provided to users, and statistical and testing functions can also be provided.

[0004] However, in this method, the memory carrier often has a large gap from the content to be memorized, and the accuracy and usability of the memory carrier are not high, and the memory carrier is not very helpful to users in memory. Summary of the Invention

[0005] In view of the above problems, embodiments of the present application provide a method, device, electronic device, and readable storage medium for generating a three-dimensional memory carrier, so as to overcome or at least partially solve the above problems.

[0006] In a first aspect of the embodiments of the present application, a method for generating a three-dimensional memory carrier is provided. The method includes:

[0007] Based on the first scene description text input by the target user, generate a first style feature vector and a first semantic feature vector, and extract first keyword information based on the first text to be memorized input by the target user;

[0008] Input the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model; wherein, the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network;

[0009] Based on the first stylized keyword information, generate a first memory carrier modeling, and based on the first semantic feature vector, generate a first three-dimensional geometric scene modeling;

[0010] Generate the first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling.

[0011] Optionally, generating the first style feature vector and the first semantic feature vector based on the first scene description text input by the target user includes:

[0012] Input the first scene description text input by the target user into the second combined neural network model to obtain the first style feature vector output by the second combined neural network model; wherein, the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model;

[0013] Input the first scene description text into the vision transformer model to obtain the first semantic feature vector output by the vision transformer model.

[0014] Optionally, extracting the first keyword information based on the first text to be memorized input by the target user includes:

[0015] Input the first text to be memorized input by the target user into the extractive summary model to obtain the first summary memory information output by the extractive summary model;

[0016] Optimize the first summary information based on the attention weights between the various word segments included in the first summary memory information to obtain the first keyword information.

[0017] Optionally, generating the first memory carrier modeling based on the first stylized keyword information includes:

[0018] Generate the first virtual model based on the first stylized keyword information;

[0019] In the case where the similarity between the first virtual model and the first real model corresponding to the first memory text and the first scene description text is greater than or equal to the first threshold, determine the first virtual model as the first memory carrier modeling.

[0020] Optionally, generating the first three-dimensional geometric scene modeling based on the first semantic feature vector includes:

[0021] Input the first semantic feature vector into the conditional generative adversarial network model to obtain the geometric scene feature vector output by the conditional generative adversarial network model;

[0022] Perform geometric constraints on the geometric scene feature vector to obtain the first three-dimensional geometric scene modeling.

[0023] Optionally, the geometric constraint on the geometric scene feature vector to obtain the first three-dimensional geometric scene modeling includes:

[0024] Input the first semantic feature vector into the conditional generative adversarial network model to obtain the confidence level output by the conditional generative adversarial network model;

[0025] When the confidence level is greater than or equal to the second threshold, perform geometric constraint on the geometric scene feature vector to obtain the first three-dimensional geometric scene modeling.

[0026] Optionally, the generation of the first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling includes:

[0027] Based on the first bounded surface where the entrance of the first three-dimensional geometric scene modeling is located, and the second bounded surfaces respectively corresponding to each of the first memory carriers, determine the first entrance connection surface included in the first three-dimensional geometric scene modeling; wherein, the first entrance connection surface represents a curve segment formed by connecting any point inside the second bounded surface (excluding the boundary) and any point inside the first bounded surface without intersection with any surface of the first three-dimensional geometric scene modeling; the connection line between any two points in the bounded surface contains all points on the same side of the bounded surface, and the boundary of the bounded surface is a closed curve;

[0028] Determine the midpoint of the connection line between the two points with the largest distance among all the first entrance connection surfaces as the rendering point for the memory carrier modeling;

[0029] Based on the first semantic feature vector, render the first memory carrier modeling at the rendering point for the memory carrier modeling in the first three-dimensional geometric scene modeling to obtain the first three-dimensional memory carrier set modeling.

[0030] Optionally, the determination of the midpoint of the connection line between the two points with the largest distance among all the first entrance connection surfaces as the rendering point for the memory carrier modeling includes:

[0031] When the number of the first entrance connection surfaces is less than the number of the first memory carrier modeling, determine the orthogonal plane corresponding to the first entrance connection surface; wherein, the orthogonal plane contains the connection line between the two points with the largest distance among the first entrance connection surfaces;

[0032] Based on the orthogonal plane, divide the first entrance connection surface to obtain a plurality of second entrance connection surfaces;

[0033] When the number of the second inlet connection surfaces is greater than or equal to the number of the first memory carrier modelings, the midpoint of the line connecting the two points with the largest distance among the respective second inlet connection surfaces is determined as the memory carrier modeling rendering point.

[0034] Optionally, rendering the first memory carrier modeling on the memory carrier modeling rendering point based on the first semantic feature vector to obtain a first three-dimensional memory carrier set modeling includes:

[0035] Determining a word vector sequence corresponding to the first semantic feature vector;

[0036] Based on the sequence order of each word vector in the word vector sequence, rendering each first memory carrier modeling corresponding to each word vector in the word vector sequence on the memory carrier modeling rendering point to obtain a first three-dimensional memory carrier set modeling.

[0037] In a second aspect, an embodiment of the present application provides a three-dimensional memory carrier generation device, and the device includes:

[0038] A generation and extraction module, configured to generate a first style feature vector and a first semantic feature vector based on a first scene description text input by a target user, and extract first keyword information based on a first text to be memorized input by the target user;

[0039] An input module, configured to input the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model; wherein, the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network;

[0040] A first generation module, configured to generate a first memory carrier modeling based on the first stylized keyword information, and generate a first three-dimensional geometric scene modeling based on the first semantic feature vector;

[0041] A second generation module, configured to generate a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling.

[0042] Optionally, the generation and extraction module includes:

[0043] A first input sub-module, configured to input a first scene description text input by a target user into a second combined neural network model to obtain a first style feature vector output by the second combined neural network model; wherein, the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model;

[0044] A second input sub-module, configured to input the first scenario description text into a vision transformer model to obtain a first semantic feature vector output by the vision transformer model.

[0045] Optionally, the generation and extraction module includes:

[0046] A third input sub-module, configured to input the first text to be memorized input by the target user into an extractive summarization model to obtain first summary memory information output by the extractive summarization model;

[0047] A fourth input sub-module, configured to optimize the first summary information based on the attention weights between the various word segments included in the first summary memory information to obtain first keyword information.

[0048] Optionally, the first generation module includes:

[0049] A first generation sub-module, configured to generate a first virtual model based on the first stylized keyword information;

[0050] A first determination sub-module, configured to determine the first virtual model as the first memory carrier modeling when the similarity between the first real model corresponding to the first memory text and the first scenario description text and the first virtual model is greater than or equal to a first threshold.

[0051] Optionally, the first generation module includes:

[0052] A fifth input sub-module, configured to input the first semantic feature vector into a conditional generative adversarial network model to obtain a geometric scene feature vector output by the conditional generative adversarial network model;

[0053] A geometric constraint sub-module, configured to perform geometric constraint on the geometric scene feature vector to obtain a first three-dimensional geometric scene modeling.

[0054] Optionally, the geometric constraint sub-module includes:

[0055] A first input unit, configured to input the first semantic feature vector into a conditional generative adversarial network model to obtain a confidence level output by the conditional generative adversarial network model;

[0056] A geometric constraint unit, configured to perform geometric constraint on the geometric scene feature vector to obtain a first three-dimensional geometric scene modeling when the confidence level is greater than or equal to a second threshold.

[0057] Optionally, the second generation module includes:

[0058] A second determination sub-module, configured to determine a first entrance connection surface included in the first three-dimensional geometric scene modeling based on a first bounded surface where an entrance of the first three-dimensional geometric scene modeling is located, and second bounded surfaces respectively corresponding to the first memory carriers; wherein, the first entrance connection surface represents a curve segment formed by connecting any point without a boundary in the second bounded surface and any point in the first bounded surface, and the curve segment has no intersection with any surface in the first three-dimensional geometric scene modeling; any line connecting two points in the bounded surface contains all points on the same side of the bounded surface, and the boundary of the bounded surface is a closed curve;

[0059] A third determination sub-module, configured to determine a midpoint of a line connecting two points with the largest distance among all the first entrance connection surfaces as a memory carrier modeling rendering point;

[0060] A rendering sub-module, configured to render the first memory carrier modeling on the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling based on the first semantic feature vector, to obtain a first three-dimensional memory carrier set modeling.

[0061] Optionally, the third determination sub-module includes:

[0062] A first determination unit, configured to determine an orthogonal plane corresponding to the first entrance connection surface when the number of the first entrance connection surfaces is less than the number of the first memory carrier modelings; wherein, the orthogonal plane contains a line connecting two points with the largest distance among the first entrance connection surfaces;

[0063] A splitting unit, configured to split the first entrance connection surface based on the orthogonal plane to obtain a plurality of second entrance connection surfaces;

[0064] A second determination unit, configured to determine a midpoint of a line connecting two points with the largest distance among all the second entrance connection surfaces as a memory carrier modeling rendering point when the number of the second entrance connection surfaces is greater than or equal to the number of the first memory carrier modelings.

[0065] Optionally, the rendering sub-module includes:

[0066] A third determination unit, configured to determine a word vector sequence corresponding to the first semantic feature vector;

[0067] A rendering unit, configured to render the first memory carrier modelings respectively corresponding to the word vectors in the word vector sequence on the memory carrier modeling rendering point based on the order of the word vectors in the word vector sequence, to obtain a first three-dimensional memory carrier set modeling.

[0068] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory. It is characterized in that the processor executes the computer program to implement the three-dimensional memory carrier generation method described in any one of the above.

[0069] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the three-dimensional memory carrier generation method described in any one of the above is implemented.

[0070] Specific beneficial effects are as follows:

[0071] In the embodiment of the present application, based on the first scenario description text input by the target user, a first style feature vector and a first semantic feature vector are generated. Based on the first text to be memorized input by the target user, first keyword information is extracted. The first style feature vector and the first keyword information are input into the first combined neural network model to obtain the first stylized keyword information output by the first combined neural network model. Among them, the first combined neural network model is obtained by combining a convolutional neural network, a recurrent neural network, and a generative adversarial network. Based on the first stylized keyword information, a first memory carrier modeling is generated. Based on the first semantic feature vector, a first three-dimensional geometric scene modeling is generated. Based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling, a first memory carrier set modeling is generated, which can generate a corresponding first memory carrier geometric model according to the first scenario description text and the first text to be memorized input by the target user to assist the user in memorizing the first text to be memorized, and to a certain extent, improves the matching degree between the memory carrier set modeling and the text to be memorized, and improves the usability of the memory carrier geometric model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0073] Figure 1 is a flowchart of a three-dimensional memory carrier generation method provided by an embodiment of the present application;

[0074] Figure 2 is a flowchart of another three-dimensional memory carrier generation method provided by an embodiment of the present application;

[0075] Figure 3 is a flowchart of a memory carrier generation method provided by an embodiment of the present application;

[0076] Figure 4 It is a schematic flowchart of a method for generating a memory carrier set modeling provided by an embodiment of the present application;

[0077] Figure 5 It is a flowchart of a specific implementation manner of a method for generating a three-dimensional memory carrier provided by an embodiment of the present application;

[0078] Figure 6 It is a logic block diagram of a three-dimensional memory carrier generation device provided by an embodiment of the present application;

[0079] Figure 7 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Specific implementation manner

[0080] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully conveyed to those skilled in the art.

[0081] Refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for generating a three-dimensional memory carrier provided by an embodiment of the present application, and the method includes:

[0082] Step 101, generate a first style feature vector and a first semantic feature vector based on the first scenario description text input by the target user, and extract first keyword information based on the first text to be memorized input by the target user.

[0083] In an embodiment of the present application, a target user can input a first scene description text and a first text to be memorized into a control center. The first scene description text can be used to define the scene background of a memory carrier, and the first text to be memorized is the content text that the target user wants to memorize. The control center can generate a corresponding first style feature vector and a first semantic feature vector according to the first scene description text; at the same time, it can also extract the first keyword information of the first text to be memorized. Among them, the first style feature vector can be obtained by inputting the first scene description text into a style feature extraction neural network model, such as a Recurrent neural network (RNN), a Long Short-Term Memory (LSTM), and a traditional Bidirectional Encoder Representations from Transformers (BERT), or by inputting the first scene description text into a style feature extraction module in the above network; the first keyword information can be obtained by inputting the first text to be memorized into a text semantic segmentation model based on Transformer, or can be obtained by inputting the first text to be memorized into a specific keyword extraction model, where the keyword library built in the keyword extraction model can be set and updated manually.

[0084] For example, if the scene description text is "an empty palace", its corresponding style feature is "empty" and the semantic feature is "palace"; if the text to be memorized is "After lighting dry firewood, it will catch fire and release light and heat at the same time", the first keyword information can be "firewood, fire, light, heat".

[0085] Step 102: Input the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model; where the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network.

[0086] In an embodiment of the present application, the output ends of a convolutional neural network (CNN) and a recurrent neural network can be connected through a fully connected layer to form a fusion network, and the output end of the fusion network can be connected to the input end of a generative adversarial network (GAN) to form a first combined neural network model. After the first style feature vector and the first keyword information are input into the first combined neural network model, the convolutional neural network in the first combined neural network model can extract the visual information in the style feature vector, and the sequence information in the key content list can be processed through the recurrent neural network. After that, a fully connected layer can be introduced to connect the output ends of the convolutional neural network and the recurrent neural network, so that the first stylized keyword information can be output at the output end.

[0087] Continuing with the above example, if the style feature is "empty" and the first keyword information is "firewood, fire, light, heat", the first stylized keyword information finally obtained can be "firewood-empty, fire-empty, light-empty, heat-empty", where the symbol "-" is a connector.

[0088] Step 103: Generate a first memory carrier model based on the first stylized keyword information, and generate a first three-dimensional geometric scene model based on the first semantic feature vector.

[0089] In an embodiment of the present application, a first memory carrier modeling can be generated by various virtual reality (VR) software or augmented reality (AR) software according to the first stylized keyword information; similarly, a first three-dimensional geometric scene modeling can be generated by VR software or AR software according to the first semantic feature vector. Specifically, VR software and AR software can extract memory carrier rendering features from the first stylized keyword information and match them with the database in the software. When the corresponding data is matched, the data is rendered to form a first memory carrier, and can be displayed in the form of a model on the display interface. Similarly, VR software and AR software can extract three-dimensional geometric scene rendering features from the first semantic feature vector and match them with the database in the software. When the corresponding data is matched, the data is rendered to form a first three-dimensional geometric scene, and can be displayed in the form of a model on the display interface.

[0090] Step 104: Generate a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling and the first three-dimensional geometric scene modeling.

[0091] In an embodiment of the present application, a first memory carrier geometric model can be generated based on a first semantic feature vector, a first memory carrier modeling, and a first three-dimensional geometric scene modeling. Specifically, a word sequence corresponding to the first semantic feature vector can be determined, and based on this word sequence, the corresponding relationship between the first memory carrier modeling and the first semantic feature vector can be further determined. Then, based on this corresponding relationship, the first memory carrier modeling can be rendered in the first three-dimensional geometric scene modeling, thereby obtaining the first memory carrier geometric model. Among them, a texture mapping algorithm can be used to fuse the first memory carrier modeling and the first three-dimensional geometric scene modeling to form a first memory carrier set modeling that finally carries the memory carrier.

[0092] In an embodiment of the present application, by generating a first style feature vector and a first semantic feature vector based on the first scene description text input by the target user, extracting first keyword information based on the first text to be memorized input by the target user, and inputting the first style feature vector and the first keyword information into a first combined neural network model, first stylized keyword information output by the first combined neural network model is obtained; wherein, the first combined neural network model is obtained by combining a convolutional neural network, a recurrent neural network, and a generative adversarial network. Based on the first stylized keyword information, a first memory carrier modeling is generated, based on the first semantic feature vector, a first three-dimensional geometric scene modeling is generated, and based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling, a first memory carrier set modeling is generated, which can generate a corresponding first memory carrier geometric model according to the first scene description text and the first text to be memorized input by the target user to assist the user in memorizing the first text to be memorized, and to a certain extent, improves the matching degree between the memory carrier set modeling and the text to be memorized, and improves the usability of the memory carrier geometric model.

[0093] Referring to Figure 2 , Figure 2 FIG. is a schematic flowchart of another three-dimensional memory carrier generation method provided by an embodiment of the present application. The method may include:

[0094] Step 201, generating a first style feature vector and a first semantic feature vector based on the first scene description text input by the target user, and determining and extracting first keyword information based on the first text to be memorized input by the target user.

[0095] In an embodiment of the present application, the implementation content of this step can refer to the embodiment content of step 101 and will not be elaborated here.

[0096] Optionally, step 201 may include the following sub-steps:

[0097] Sub-step 2011: Input the first scene description text entered by the target user into the second combined neural network model to obtain the first style feature vector output by the second combined neural network model. Among them, the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model.

[0098] In the embodiments of the present application, after receiving the first scene description text entered by the target user, the first scene description text can be input into the second combined neural network model, so that the first style feature vector output by the second combined neural network model can be obtained. Among them, the second combined neural network model can be obtained by combining a traditional bidirectional encoder representation model (BERT) and a style transfer network model. The style transfer network model can be composed of multiple fully connected layers, and each fully connected layer is followed by a ReLU activation function. At the same time, in order to enhance the model's understanding of the text style, a style encoder can also be introduced between the BERT model and the style transfer network. The style encoder can be adjusted and trained based on the Variational Autoencoder (VAE). The first style feature vector can contain all the style features corresponding to the first scene description text.

[0099] Sub-step 2012: Input the first scene description text into the vision transformer model to obtain the first semantic feature vector output by the vision transformer model.

[0100] In the embodiments of the present application, the first scene description text can be input into the Vision Transformer (ViT) model, so that the first semantic feature vector output by the vision transformer model can be obtained. The first semantic feature vector can indicate the specific scene corresponding to the first scene description text, such as "building", "palace", "quadrangle courtyard", "grassland", etc.

[0101] In the embodiments of the present application, by inputting the first scene description text entered by the target user into the second combined neural network model to obtain the first style feature vector output by the second combined neural network model, where the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model, and inputting the first scene description text into the vision transformer model to obtain the first semantic feature vector output by the vision transformer model, the first style feature vector and the first semantic feature vector corresponding to the first scene description text can be obtained respectively through the second combined neural network model and the vision transformer model, which improves the accuracy and usability of the first style feature vector and the first semantic feature vector to a certain extent.

[0102] Sub-step 2013: Input the first text to be memorized entered by the target user into the extractive summarization model, and obtain the first summary memory information output by the extractive summarization model.

[0103] In the embodiments of the present application, the extractive summarization model can be composed of multiple traditional recurrent neural network layers. At the output end of the multiple recurrent neural networks, they can be connected through a fully connected layer, and the activation function of the fully connected layer can be the ReLU function. After inputting the first text to be memorized entered by the target user into this extractive summarization model, the first summary memory information output by this extractive summarization model can be obtained.

[0104] Continuing with the above example, if the first text to be memorized is "After lighting dry firewood, it will catch fire, and at the same time release light and heat", then the first summary memory information can be "Light the firewood, it will catch fire, release light and heat".

[0105] Sub-step 2014: Optimize the first summary information based on the attention weights between the various word segments included in the first summary memory information to obtain the first keyword information.

[0106] In the embodiments of the present application, an attention layer can be embedded at the output end of the extractive summarization model. The attention layer is composed of multiple fully connected layers and can be used to calculate the attention weights. Then, these attention weights can be applied to the initial summary to optimize and select the summary information. The first summary memory information can include multiple word segments. After performing word segmentation on the first summary memory information, these word segments can be obtained. At the same time, since the attention weights in the extractive summarization model are introduced when generating the first summary memory information, the attention weights obtained by the extractive summarization model can be acquired, and the first summary memory information can be optimized through these attention weights, thereby obtaining the first keyword information.

[0107] In the embodiments of the present application, by inputting the first text to be memorized entered by the target user into the extractive summarization model, obtaining the first summary memory information output by the extractive summarization model, and optimizing the first summary information based on the attention weights between the various word segments included in the first summary memory information to obtain the first keyword information, the first keyword information corresponding to the first text to be memorized can be obtained, improving the matching degree between the first keyword information and the text to be memorized, and improving the reliability of the first keyword information.

[0108] Step 202: Input the first style feature vector and the first keyword information into the first combined neural network model, and obtain the first stylized keyword information output by the first combined neural network model; wherein, the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network.

[0109] In the embodiments of the present application, the implementation content of this step can refer to the implementation content of step 102, which will not be elaborated here.

[0110] Step 203: Generate a first memory carrier modeling based on the first stylized keyword information, and generate a first three-dimensional geometric scene modeling based on the first semantic feature vector.

[0111] In the embodiments of the present application, the implementation content of this step can refer to the implementation content of step 103, which will not be elaborated here.

[0112] Optionally, step 203 may include the following sub-steps:

[0113] Sub-step 2031: Generate a first virtual model based on the first stylized keyword information.

[0114] In the embodiments of the present application, the first stylized keyword information can be input into a generator network or a language model based on the generator network, so as to obtain a virtual model output by the generator network or the language model. The generator network or the language model based on the generator network can generate a corresponding first virtual model according to the first stylized keyword information.

[0115] Sub-step 2032: When the similarity between the first virtual model and the first real model corresponding to the first memory text and the first scene description text is greater than or equal to a first threshold, determine the first virtual model as the first memory carrier modeling.

[0116] In the embodiments of the present application, a discriminator can be used to judge the similarity between the first virtual model and the first real model. Among them, the first real model can correspond to the first memory text and the first scene description text, that is, the model expressed by the first memory text and the first scene description text; the discriminator can be composed of multiple convolutional layers, batch normalization layers and Dropout layers, and is used to distinguish the generated virtual model and the real model through similarity. When the similarity is greater than or equal to the first threshold, the first virtual model can be determined as the first memory carrier modeling.

[0117] In an embodiment of the present application, a first virtual model is generated based on first stylized keyword information. When the similarity between the first real model corresponding to the first memory text and the first scene description text and the first virtual model is greater than or equal to a first threshold, the first virtual model is determined as the first memory carrier modeling. The first virtual model can be generated according to the first stylized keyword information, and when the similarity between the first virtual model and the first real model is greater than or equal to the first threshold, the first virtual model is determined as the first memory carrier modeling, which improves the accuracy and reliability of the first memory carrier modeling to a certain extent.

[0118] Sub-step 2033: Input the first semantic feature vector into a conditional generative adversarial network model to obtain a geometric scene feature vector output by the conditional generative adversarial network model.

[0119] In an embodiment of the present application, a conditional generative adversarial network model (Conditional Generative Adversarial Nets, cGAN) can be specifically optimized to adapt to text-driven 3D scene generation tasks. At the same time, to avoid generating incoherent geometric scenes, the conditional generative adversarial network model can be adjusted so that it can output a feature vector. On this basis, the first semantic feature vector can be input into the conditional generative adversarial network model, and thus a geometric scene feature vector output by the conditional generative adversarial network model can be obtained.

[0120] Sub-step 2034: Perform geometric constraints on the geometric scene feature vector to obtain a first 3D geometric scene modeling.

[0121] In an embodiment of the present application, the geometric scene features included in the geometric scene feature vector can jointly form a geometric scene. After obtaining the geometric scene feature vector, geometric constraints can be performed on the geometric scene feature vector, and thus a coherent first 3D geometric scene modeling can be obtained. Among them, the geometric constraint can be a loss function F loss = Loss(L MSE , R MSE ), where L MSE is the mean square error of the edge length, and R MSE is the mean square error of the angle deviation, so as to achieve the geometric connectivity and consistency in the predicted scene. The first 3D geometric scene modeling is a triangular mesh composed of vertices and faces, representing the 3D space scene shape and being a 3D space model.

[0122] Optionally, sub-step 2034 may include the following sub-steps:

[0123] Sub-step A1: Input the first semantic feature vector into the conditional generative adversarial network model to obtain the confidence level output by the conditional generative adversarial network model.

[0124] In the embodiments of the present application, a discriminator network can be incorporated into the conditional generative adversarial network model. After inputting the first semantic feature vector into the conditional generative adversarial network model, the discriminator network can output a confidence level, and the value of this confidence level can indicate the probability that the input data is the corresponding data of the real model.

[0125] Sub-step A2: When the confidence level is greater than or equal to the second threshold, perform geometric constraints on the geometric scene feature vector to obtain the first three-dimensional geometric scene modeling.

[0126] In the embodiments of the present application, if the confidence level output by the conditional generative adversarial network model is greater than or equal to the second threshold, it can be considered that the first semantic feature vector is close to the real model. At this time, geometric constraints can be performed on the geometric scene feature vector, thereby obtaining the first three-dimensional geometric scene modeling. The specific implementation content of the geometric constraints can refer to the embodiment content of step 2034 and will not be elaborated here.

[0127] In the embodiments of the present application, by inputting the first semantic feature vector into the conditional generative adversarial network model to obtain the confidence level output by the conditional generative adversarial network model, and performing geometric constraints on the geometric scene feature vector when the confidence level is greater than or equal to the second threshold to obtain the first three-dimensional geometric scene modeling, geometric constraints can be performed on the geometric scene feature vector when the confidence level is greater than or equal to the second threshold, thereby obtaining the first three-dimensional geometric scene modeling, which improves the similarity between the first three-dimensional geometric scene modeling and the real scene to a certain extent.

[0128] In the embodiments of the present application, by inputting the first semantic feature vector into the conditional generative adversarial network model to obtain the geometric scene feature vector output by the conditional generative adversarial network model, performing geometric constraints on the geometric scene feature vector to obtain the first three-dimensional geometric scene modeling, it avoids the situation of discontinuous modeling in the first three-dimensional geometric scene modeling and improves the reliability of the first three-dimensional geometric scene modeling.

[0129] Step 204: Based on the first bounded surface where the entrance of the first three-dimensional geometric scene modeling is located and the second bounded surfaces corresponding to each of the first memory carriers, determine the first entrance connection surface included in the first three-dimensional geometric scene modeling; wherein, the first entrance connection surface represents a second bounded surface where a curve segment formed by connecting any point not including the boundary in the surface and any point in the first bounded surface has no intersection with any surface of the first three-dimensional geometric scene modeling; for the bounded surface, all points included in the line connecting any two points are on the same side of the bounded surface, and the boundary of the bounded surface is a closed curve.

[0130] In an embodiment of the present application, a bounded surface refers to a plane or a curved surface with a boundary, that is, a surface with a fixed shape and size, and the boundaries of such planes or curved surfaces are all closed curves. In addition, all points included in the line connecting any two points in the bounded surface are on the same side of the bounded surface. Among them, the first bounded surface may be the surface where the entrance of the first three-dimensional geometric scene modeling is located, and the second bounded surfaces may correspond to each of the first memory carriers respectively. Wherein, if the bounded surface intersects with the memory carrier, it can be considered that the bounded surface corresponds to the memory carrier. After determining the first bounded surface and the second bounded surfaces, the geometric relationship between the first bounded surface and the second bounded surfaces can be determined, and then the first entrance connection surface included in the first three-dimensional geometric scene modeling can be determined according to this geometric relationship. Among them, the first entrance connection surface may represent a second bounded surface where a curve segment formed by connecting any point not including the boundary in the surface and any point in the first bounded surface has no intersection with any surface of the first three-dimensional geometric scene modeling.

[0131] For example, if there are three surfaces a, b, and c, as long as there is a curve segment that does not pass through surface b for any two points (the two points come from different surfaces) in surfaces a and c, it can be determined that they are connected, and a can be called a connection surface of c. When the line connecting any two points in surfaces a and c must pass through b, it can be considered that a and c are not connected. For example, if a is enclosed in a closed space by several surfaces, then a and c are not connected, and a is not a connection surface of c.

[0132] In a possible embodiment, the surface-to-surface relationship between each of the second bounded surfaces can also be determined. The surface-to-surface relationship may include two surfaces sharing an edge and two surfaces not sharing an edge. When determining the first entrance connection surface, since the second bounded surfaces sharing an edge can obtain the same first entrance connection surface, any one of the second bounded surfaces sharing an edge can be selected to participate in the determination process of the first entrance connection surface.

[0133] Step 205: Determine the midpoint of the line connecting the two points with the largest distance among each of the first entrance connection surfaces as the memory carrier modeling rendering point.

[0134] In an embodiment of the present application, the midpoint of the line connecting the two points with the largest distance in the first inlet connection plane can be determined as the memory carrier modeling rendering point, and this point can be used for rendering the memory carrier. Wherein, the distance between the two points can refer to the shortest connection path between the two points along the first inlet connection plane.

[0135] Optionally, in step 205, the following sub-steps may be included:

[0136] Sub-step 2051, in the case where the number of the first inlet connection planes is less than the number of the first memory carrier modelings, determine the orthogonal plane corresponding to the first inlet connection plane; wherein, the orthogonal plane contains the line connecting the two points with the largest distance in the first inlet connection plane.

[0137] In an embodiment of the present application, if the number of the first inlet connection planes is less than the number of the first memory carrier modelings, the orthogonal plane corresponding to the first inlet connection plane can be determined. The orthogonal planes corresponding to each of the first inlet connection planes can be determined, or some of the first inlet connection planes can be selected, and then the orthogonal planes corresponding to these selected first inlet connection planes can be determined. Wherein, the orthogonal plane can contain the line connecting the two points with the largest distance in the first inlet connection plane. If the first inlet connection plane is a plane, the corresponding orthogonal plane is perpendicular to the first inlet connection plane.

[0138] Sub-step 2052, based on the orthogonal plane, divide the first inlet connection plane to obtain a plurality of second inlet connection planes.

[0139] In an embodiment of the present application, the first inlet connection plane can be divided according to the orthogonal plane, so as to obtain a plurality of second inlet connection planes. That is, the first inlet connection plane can be cut along the direction of the orthogonal plane, so as to obtain a plurality of second inlet connection planes.

[0140] Sub-step 2053, in the case where the number of the second inlet connection planes is greater than or equal to the number of the first memory carrier modelings, determine the midpoint of the line connecting the two points with the largest distance in each of the second inlet connection planes as the memory carrier modeling rendering point.

[0141] In an embodiment of the present application, if the number of the second inlet connection planes is greater than or equal to the number of the first memory carrier modelings, the midpoint of the line connecting the two points with the largest distance in each of the second inlet connection planes can be determined as the memory carrier modeling rendering point. Wherein, the distance between the two points can refer to the shortest connection path between the two points along the first inlet connection plane. Obviously, the memory carrier modeling rendering points are all in the first three-dimensional geometric scene modeling.

[0142] In an embodiment of the present application, when the number of first inlet connection surfaces is less than the number of first memory carrier modelings, an orthogonal plane corresponding to the first inlet connection surface is determined; wherein, the orthogonal plane includes a connection line between two points with the largest distance in the first inlet connection surface. Based on the orthogonal plane, the first inlet connection surface is segmented to obtain a plurality of second inlet connection surfaces. When the number of second inlet connection surfaces is greater than or equal to the number of first memory carrier modelings, the midpoint of the connection line between two points with the largest distance in each second inlet connection surface is determined as the memory carrier modeling rendering point. When the number of first inlet connection surfaces is less than the number of first memory carrier modelings, the first inlet connection surface can be cut to obtain a plurality of second inlet connection surfaces, and when the number of second inlet connection surfaces is greater than or equal to the number of first memory carrier modelings, the memory carrier modeling rendering point can be determined, ensuring that all first memory carrier modelings can have corresponding memory carrier modeling rendering points, and improving the reliability of the overall generation method of the memory carrier modeling.

[0143] Step 206, based on the first semantic feature vector, render the first memory carrier modeling on the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling to obtain a first three-dimensional memory carrier set modeling.

[0144] In an embodiment of the present application, according to the first semantic feature vector, the first memory carrier modeling can be rendered on the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling, so that a first three-dimensional memory carrier set modeling can be obtained. Specifically, the to-be-remembered features included in the first semantic feature vector can be obtained, and the first memory carrier modeling corresponding to the to-be-remembered features can be rendered on its corresponding memory carrier modeling rendering point, so that a first three-dimensional memory carrier set modeling can be obtained.

[0145] Optionally, step 206 may include the following sub-steps:

[0146] Sub-step 2061, determine the word vector sequence corresponding to the first semantic feature vector.

[0147] In an embodiment of the present application, the first semantic feature vector may include multiple word vectors. Since the first semantic feature vector is obtained through semantic recognition, these word vectors may have a certain arrangement order, thus forming a word vector sequence. On this basis, the word vector sequence corresponding to the first semantic feature vector can be determined.

[0148] Sub-step 2062, based on the order of each word vector in the word vector sequence, render the first memory carrier modelings corresponding to each word vector in the word vector sequence on the memory carrier modeling rendering point to obtain a first three-dimensional memory carrier set modeling.

[0149] In an embodiment of the present application, according to the sequence of each word vector in the word vector sequence, the first memory carriers corresponding to each word vector in the word vector sequence can be modeled and rendered on the memory carrier modeling and rendering points, so that a first three-dimensional memory carrier set modeling can be obtained.

[0150] In an embodiment of the present application, by determining the word vector sequence corresponding to the first semantic feature vector, and based on the sequence of each word vector in the word vector sequence, the first memory carriers corresponding to each word vector in the word vector sequence are modeled and rendered on the memory carrier modeling and rendering points, and a first three-dimensional memory carrier set modeling is obtained. According to the sequence of each word vector included in the word vector sequence corresponding to the first semantic feature vector, the first memory carrier can be modeled and rendered on the memory carrier modeling and rendering points, so that a first three-dimensional memory carrier set modeling can be obtained, improving the matching degree between the first three-dimensional memory carrier set modeling and the first text to be memorized input by the target user, and further improving the usability of the first three-dimensional memory carrier set modeling.

[0151] In an embodiment of the present application, based on the first bounded surface where the entrance of the first three-dimensional geometric scene modeling is located, and the second bounded surfaces corresponding to each of the first memory carriers, the first entrance connection surface included in the first three-dimensional geometric scene modeling is determined; wherein, the first entrance connection surface represents a curve segment formed by connecting any point without a boundary in the second bounded surface and any point in the first bounded surface having no intersection with any surface of the first three-dimensional geometric scene modeling; any line segment connecting two points in the bounded surface contains all points on the same side of the bounded surface, and the boundary of the bounded surface is a closed curve. The midpoint of the line segment connecting the two points with the largest distance among each of the first entrance connection surfaces is determined as the memory carrier modeling and rendering point. Based on the first semantic feature vector, the first memory carrier is modeled and rendered on the memory carrier modeling and rendering point in the first three-dimensional geometric scene modeling, and a first three-dimensional memory carrier set modeling is obtained. The first entrance connection surface can be obtained according to the geometric relationship between the first bounded surface and the second bounded surface, and the memory carrier modeling and rendering point can be determined according to the first entrance connection surface, so that the first memory carrier modeling can be rendered on the memory carrier modeling and rendering point to obtain a first three-dimensional memory carrier set modeling, improving the matching degree between the first three-dimensional memory carrier set modeling and the first text to be memorized input by the target user, and improving the usability of the first three-dimensional memory carrier geometric model.

[0152] Refer to Figure 3 , Figure 3It is a schematic flowchart of a memory carrier generation method provided by an embodiment of the present application. In the figure, the memory carrier generation method can be divided into three steps, namely: text preprocessing, stylized content fusion, and memory carrier image generation. Among them, in the text preprocessing step, a BERT base model and an RNN model can be included. In the BERT base model, a variational autoencoder (VAE) and a style transfer network can be embedded to output a style feature vector when inputting a scene description text. After the RNN model, an attention layer based on Transformer can be added to output key memory information when inputting the text to be memorized. In the stylized content fusion step, the key memory information and the style feature vector can be input into a CNN network and an RNN network respectively. The CNN network is used to extract visual information from the key memory information and the style feature vector, and the RNN network is used to extract sequence information from the key memory information. Then, the visual information and the sequence information can be input into a GAN network through a fully connected layer, so as to obtain the stylized key information output by the GAN network. In the memory carrier image generation step, the stylized key information can be input into a cGAN network, and geometric constraints can be performed through a geometric constraint function, so as to obtain the stylized image output by the cGAN network, that is, the memory carrier.

[0153] Refer to Figure 4 , Figure 4 It is a schematic flowchart of a memory carrier set modeling generation method provided by an embodiment of the present application. In the figure, the memory carrier set modeling generation method can include three steps, namely: text feature extraction, 3D scene generation, and image fusion. Among them, in the text feature extraction step, first, the scene description text can be tokenized to obtain each word, and then position encoding can be performed on each word to obtain the word sequence of the scene description text. Then, the word sequence and the memory carrier can be input into an extractive summary model to obtain the stylized key information output by the extractive summary model. In the 3D scene generation step, a 3D scene can be generated according to VR software or AR software through the stylized key information. In the image fusion scenario, the stylized key information and the 3D scene can be fused through a texture mapping algorithm by a VGG model, so as to obtain the memory carrier set modeling. The memory carrier set modeling includes a 3D scene and each memory carrier.

[0154] Refer to Figure 5 , Figure 5It is a flowchart of a specific implementation manner of a three-dimensional memory carrier generation method provided by an embodiment of the present application. In the figure, the scene description text and the text to be memorized can be combined to jointly generate a memory carrier modeling. The scene description text can generate a three-dimensional scene modeling. After obtaining the three-dimensional scene modeling and the memory carrier modeling, rendering point planning can be performed to obtain the memory carrier rendering points in the three-dimensional scene modeling. Then, rendering the memory carrier modeling on the memory carrier rendering points can obtain the memory carrier set modeling.

[0155] Referring to Figure 6 , Figure 6 is a logic block diagram of a three-dimensional memory carrier generation device provided by an embodiment of the present application. The three-dimensional memory carrier generation device 600 may include:

[0156] A generation and extraction module 601, configured to generate a first style feature vector and a first semantic feature vector based on the first scene description text input by the target user, and extract first keyword information based on the first text to be memorized input by the target user;

[0157] An input module 602, configured to input the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model; wherein, the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network;

[0158] A first generation module 603, configured to generate a first memory carrier modeling based on the first stylized keyword information, and generate a first three-dimensional geometric scene modeling based on the first semantic feature vector;

[0159] A second generation module 604, configured to generate a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling, and the first three-dimensional geometric scene modeling.

[0160] Optionally, the generation and extraction module 601 includes:

[0161] A first input sub-module, configured to input the first scene description text input by the target user into a second combined neural network model to obtain a first style feature vector output by the second combined neural network model; wherein, the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model;

[0162] A second input sub-module, configured to input the first scene description text into a vision transformer model to obtain a first semantic feature vector output by the vision transformer model.

[0163] Optionally, the generation and extraction module 601 includes:

[0164] A third input sub-module, configured to input the first text to be memorized input by the target user into an extractive summarization model, and obtain first summary memory information output by the extractive summarization model;

[0165] A fourth input sub-module, configured to optimize the first summary information based on the attention weights between the various word segments included in the first summary memory information, and obtain first keyword information.

[0166] Optionally, the first generation module 603 includes:

[0167] A first generation sub-module, configured to generate a first virtual model based on the first stylized keyword information;

[0168] A first determination sub-module, configured to, when the similarity between the first virtual model and the first real model corresponding to the first memory text and the first scene description text is greater than or equal to a first threshold, determine the first virtual model as the first memory carrier modeling.

[0169] Optionally, the first generation module 603 includes:

[0170] A fifth input sub-module, configured to input the first semantic feature vector into a conditional generative adversarial network model, and obtain a geometric scene feature vector output by the conditional generative adversarial network model;

[0171] A geometric constraint sub-module, configured to perform geometric constraint on the geometric scene feature vector to obtain a first three-dimensional geometric scene modeling.

[0172] Optionally, the geometric constraint sub-module includes:

[0173] A first input unit, configured to input the first semantic feature vector into a conditional generative adversarial network model, and obtain the confidence output by the conditional generative adversarial network model;

[0174] A geometric constraint unit, configured to, when the confidence is greater than or equal to a second threshold, perform geometric constraint on the geometric scene feature vector to obtain a first three-dimensional geometric scene modeling.

[0175] Optionally, the second generation module 604 includes:

[0176] A second determination sub-module, configured to determine a first entrance connection surface included in the first three-dimensional geometric scene modeling based on a first bounded surface where an entrance of the first three-dimensional geometric scene modeling is located, and second bounded surfaces respectively corresponding to the first memory carriers; wherein, the first entrance connection surface represents a curve segment formed by connecting any point within the second bounded surface excluding the boundary and any point within the first bounded surface having no intersection with any surface of the first three-dimensional geometric scene modeling; a connection line between any two points within the bounded surface contains all points on the same side of the bounded surface, and the boundary of the bounded surface is a closed curve;

[0177] A third determination sub-module, configured to determine a midpoint of a connection line between two points with the largest distance among the first entrance connection surfaces as a memory carrier modeling rendering point;

[0178] A rendering sub-module, configured to render the first memory carrier modeling on the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling based on the first semantic feature vector, to obtain a first three-dimensional memory carrier set modeling.

[0179] Optionally, the third determination sub-module includes:

[0180] A first determination unit, configured to determine an orthogonal plane corresponding to the first entrance connection surface when the number of the first entrance connection surfaces is less than the number of the first memory carrier modelings; wherein, the orthogonal plane contains a connection line between two points with the largest distance among the first entrance connection surfaces;

[0181] A segmentation unit, configured to segment the first entrance connection surface based on the orthogonal plane to obtain a plurality of second entrance connection surfaces;

[0182] A second determination unit, configured to determine a midpoint of a connection line between two points with the largest distance among the second entrance connection surfaces as a memory carrier modeling rendering point when the number of the second entrance connection surfaces is greater than or equal to the number of the first memory carrier modelings.

[0183] Optionally, the rendering sub-module includes:

[0184] A third determination unit, configured to determine a word vector sequence corresponding to the first semantic feature vector;

[0185] A rendering unit, configured to render the first memory carrier modelings respectively corresponding to the word vectors in the word vector sequence on the memory carrier modeling rendering point based on the sequence order of the word vectors in the word vector sequence, to obtain a first three-dimensional memory carrier set modeling.

[0186] The three-dimensional memory carrier generation device in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than terminals. Exemplarily, the electronic device may be a GPU BOX, a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0187] The three-dimensional memory carrier generation device in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, a Linux, Windows operating system, etc., or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0188] The three-dimensional memory carrier generation device provided in the embodiments of the present application can implement Figures 1 to 5 each process implemented by the method embodiments. To avoid repetition, it will not be elaborated here.

[0189] The embodiments of the present application provide an electronic device. Refer to Figure 7 , the electronic device 70 includes: a processor 701, a memory 702, and a computer program 7021 stored on the memory 702 and executable on the processor 701. When the processor 701 executes the program, it implements the three-dimensional memory carrier generation method of the foregoing embodiments.

[0190] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, the steps in the three-dimensional memory carrier generation method disclosed in the embodiments of the present application are implemented.

[0191] The embodiments of the present application also provide a computer program product. When the computer program product runs on an electronic device, it causes the processor to execute the steps in the three-dimensional memory carrier generation method disclosed in the embodiments of the present application when executed.

[0192] The embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0193] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0194] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0195] These computer program instructions can also be loaded onto the computer or other programmable data processing terminal devices, so that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal devices provide steps for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0196] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0197] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0198] The above has introduced in detail a resource decoupling system, execution method and device for deep learning applications provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for generating a three-dimensional memory carrier, characterized in that: The method comprises: Based on the first scene description text input by the target user, a first style feature vector and a first semantic feature vector are generated, and based on the first text to be memorized input by the target user, first keyword information is extracted; Inputting the first style feature vector and the first keyword information into a first combined neural network model to obtain first stylized keyword information output by the first combined neural network model; wherein the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network; Generate a first memory carrier model based on the first stylized keyword information, and generate a first three-dimensional geometric scene model based on the first semantic feature vector; Generate a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling and the first three-dimensional geometric scene modeling; The generating a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling and the first three-dimensional geometric scene modeling includes: Based on the first interface where the entrance of the first three-dimensional geometric scene modeling is located, and the second interfaces corresponding to each of the first memory carriers, determine the first entrance connection surface contained in the first three-dimensional geometric scene modeling; wherein the first entrance connection surface means that the curve segment formed by connecting any point in the second interface excluding the boundary and any point in the first interface has no intersection with any surface of the first three-dimensional geometric scene modeling; all points contained in the line between any two points in the interface are on the same side of the interface, and the boundary of the interface is a closed curve; Determine the midpoint of the line between the two points with the largest distance in each of the first inlet connecting surfaces as the memory carrier modeling rendering point; Based on the first semantic feature vector, the first memory carrier modeling is rendered at the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling to obtain a first three-dimensional memory carrier set modeling.

2. The method according to claim 1, characterized in that: The step of generating a first style feature vector and a first semantic feature vector based on a first scene description text input by a target user includes: Inputting a first scene description text input by a target user into a second combined neural network model to obtain a first style feature vector output by the second combined neural network model; wherein the second combined neural network model is obtained by combining a traditional bidirectional encoder representation model and a style transfer network model; The first scene description text is input into a visual converter model to obtain a first semantic feature vector output by the visual converter model.

3. The method according to claim 1, characterized in that The extracting first keyword information based on the first text to be memorized input by the target user includes: Inputting the first to-be-remembered text input by the target user into the extractive summarization model to obtain first summary memory information output by the extractive summarization model; Based on the attention weights between the word segments contained in the first summary memory information, the first summary memory information is optimized to obtain first keyword information.

4. The method according to claim 1, characterized in that: The step of generating a first memory carrier model based on the first stylized keyword information includes: generating a first virtual model based on the first stylized keyword information; When the similarity between the first real model corresponding to the first text to be memorized and the first scene description text and the first virtual model is greater than or equal to a first threshold, the first virtual model is determined to be modeled as the first memory carrier.

5. The method according to claim 1, characterized in that: The step of generating a first three-dimensional geometric scene model based on the first semantic feature vector includes: Inputting the first semantic feature vector into a conditional generative adversarial network model to obtain a geometric scene feature vector output by the conditional generative adversarial network model; Geometric constraints are applied to the geometric scene feature vector to obtain a first three-dimensional geometric scene model.

6. The method according to claim 5, characterized in that The step of geometrically constraining the geometric scene feature vector to obtain first three-dimensional geometric scene modeling includes: Inputting the first semantic feature vector into a conditional generative adversarial network model to obtain a confidence level output by the conditional generative adversarial network model; When the confidence level is greater than or equal to the second threshold, geometric constraints are performed on the geometric scene feature vector to obtain a first three-dimensional geometric scene modeling.

7. The method according to claim 1, characterized in that The step of determining the midpoint of the line between two points with the largest distance in each of the first inlet connecting surfaces as the memory carrier modeling rendering point includes: In a case where the number of the first inlet communication surfaces is less than the number of the first memory carrier modeling, determining an orthogonal plane corresponding to the first inlet communication surface; wherein the orthogonal plane includes a line between two points with the largest distance in the first inlet communication surface; Based on the orthogonal plane, the first inlet communication surface is divided to obtain a plurality of second inlet communication surfaces; When the number of the second inlet connecting surfaces is greater than or equal to the number of the first memory carrier modeling, the midpoint of the line between the two points with the largest distance in each of the second inlet connecting surfaces is determined as the memory carrier modeling rendering point.

8. The method according to claim 1, characterized in that The step of rendering the first memory carrier modeling at the memory carrier modeling rendering point based on the first semantic feature vector to obtain a first three-dimensional memory carrier set modeling includes: Determine a word vector sequence corresponding to the first semantic feature vector; Based on the sequence of each word vector in the word vector sequence, the first memory carrier modeling corresponding to each word vector in the word vector sequence is rendered on the memory carrier modeling rendering point to obtain a first three-dimensional memory carrier set modeling.

9. A three-dimensional memory carrier generating device, characterized in that: The device comprises: A generating and extracting module is used to generate a first style feature vector and a first semantic feature vector based on a first scene description text input by a target user, and to extract first keyword information based on a first text to be memorized input by the target user; An input module, used for inputting the first style feature vector and the first keyword information into a first combined neural network model to obtain the first stylized keyword information output by the first combined neural network model; wherein the first combined neural network model is obtained by combining a convolutional neural network and a recurrent neural network; A first generating module, configured to generate a first memory carrier modeling based on the first stylized keyword information, and generate a first three-dimensional geometric scene modeling based on the first semantic feature vector; A second generating module, configured to generate a first memory carrier set modeling based on the first semantic feature vector, the first memory carrier modeling and the first three-dimensional geometric scene modeling; The second generation module is used to determine the first entrance connecting surface contained in the first three-dimensional geometric scene modeling based on the first interface where the entrance of the first three-dimensional geometric scene modeling is located, and the second interfaces corresponding to each of the first memory carriers; wherein the first entrance connecting surface represents that the curve segment formed by connecting any point in the second interface excluding the boundary and any point in the first interface has no intersection with any surface of the first three-dimensional geometric scene modeling; all points contained in the line between any two points in the interface are on the same side of the interface, and the boundary of the interface is a closed curve; the midpoint of the line between the two points with the largest distance in each of the first entrance connecting surfaces is determined as the memory carrier modeling rendering point; based on the first semantic feature vector, the first memory carrier modeling is rendered on the memory carrier modeling rendering point in the first three-dimensional geometric scene modeling to obtain the first three-dimensional memory carrier set modeling.

Images