Dataset generation method, three-dimensional content generation method, device, equipment and medium

Abstract

The application discloses a kind of dataset generation method, three-dimensional content generation method, device, equipment and medium, applied to artificial intelligence technical field.Therein, method includes using pre-tuning text question and answer model based on pre-training language model to obtain the component name information contained in three-dimensional content dataset, obtains the two-dimensional content image of each three-dimensional content in multiple perspectives in the data set.According to component name information and each two-dimensional content image, based on pixel, component name, the corresponding relationship between each three-dimensional point, determine all three-dimensional points corresponding to each component name;According to the component name of each three-dimensional content in three-dimensional content dataset and corresponding three-dimensional point, obtain text and three-dimensional content component corresponding dataset.The application can solve the problem that the precision and scale of text and three-dimensional content component corresponding dataset generated by related technology do not meet user demand, can generate large-scale high-quality text and three-dimensional content component corresponding dataset.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a dataset generation method, a three-dimensional content generation method, apparatus, device, and medium. Background Technology

[0002] With the rapid development and increasing popularity of artificial intelligence technology, AIGC (Artificial Intelligence Generated Content) technology has become a trend. AIGC uses artificial intelligence technology to automatically generate digital content in modalities, including but not limited to text, audio, images, and 3D content. It can greatly improve the work efficiency and quality of content creators in the film, entertainment, and media industries, and help enterprises accelerate their digitalization and intelligentization process.

[0003] Among these methods, text-based conditions can generate better 3D content. This approach uses given text conditions as input and controls the deep neural network to generate 3D content that matches the condition requirements through appropriate condition introduction. During the 3D content generation process, a dataset corresponding to the text content and 3D components needs to be generated for subsequent training and inference of the deep neural network. However, the scale and accuracy of the dataset generated by related technologies cannot meet the real-world needs of users.

[0004] Therefore, generating a large-scale, high-quality dataset of text content and corresponding 3D parts is a technical problem that needs to be solved by professionals in this field. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and readable storage medium for generating 3D content and its dataset based on a pre-trained language model, which can generate large-scale, high-quality datasets of text content and corresponding 3D parts.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for generating datasets based on pre-trained language models, comprising: Using the text question-answering model, obtain the component name information described in text form from the three-dimensional content dataset; Obtain two-dimensional content images from different perspectives corresponding to at least one three-dimensional content in the three-dimensional content dataset; Based on the component name information and each two-dimensional content image, and based on the correspondence between pixels, component names, and each three-dimensional point, determine the component name of at least one three-dimensional content and its corresponding three-dimensional point, and determine all three-dimensional points corresponding to each component name. Based on the corresponding component name and 3D point in each 3D content in the 3D content dataset, a dataset corresponding to text and 3D content components is obtained.

[0007] In a first exemplary implementation, the step of determining all three-dimensional points corresponding to each component name based on the component name information and each two-dimensional content image, and based on the correspondence between pixels, component names, and each three-dimensional point, at least one three-dimensional content component name and its corresponding three-dimensional point, includes: Based on the component name information and each two-dimensional content image, determine the component name corresponding to at least one target pixel in at least one two-dimensional content image; Based on the aforementioned 3D content dataset, the component names corresponding to each target pixel are fused, and all 3D points corresponding to each component name are determined according to the fusion result.

[0008] In a second exemplary implementation, determining the component name corresponding to at least one target pixel in at least one two-dimensional content image based on the component name information and each two-dimensional content image includes: The component names in the component name information are combined with at least one two-dimensional content image to obtain multiple image and text combination results; Based on the component names in each image-text combination result, the component names corresponding to each target pixel in each two-dimensional content image are obtained by performing image segmentation processing on the two-dimensional content images in the corresponding image-text combination results.

[0009] In a third exemplary implementation, the step of obtaining the component name corresponding to each target pixel in each two-dimensional content image by performing image segmentation processing on the two-dimensional content image in each image-text combination result based on the component name in each image-text combination result includes: Pre-train a model to segment everything; Each image and text combination result is input into the segmentation model in turn to obtain all pixels in the current two-dimensional content image that belong to the corresponding component name; Based on all pixels in each two-dimensional content image that belong to the corresponding component name, determine the component name corresponding to each target pixel in each two-dimensional content image.

[0010] In the fourth exemplary implementation, the step of fusing the component names corresponding to each target pixel based on the three-dimensional content dataset and determining all three-dimensional points corresponding to each component name based on the fusion result includes: obtaining the probability information of each three-dimensional point belonging to each type of component based on the three-dimensional content dataset. Based on the component name corresponding to each 3D point and the probability information, determine all 3D points corresponding to each component name.

[0011] In the fifth exemplary implementation, before determining all three-dimensional points corresponding to each component name based on the component name corresponding to each three-dimensional point and the probability information, the method further includes: Pre-build a data fusion network; Obtain the component category corresponding to at least one target pixel in at least one two-dimensional content image; The data fusion network is trained based on the probability information of each 3D point belonging to each type of component, using the component name corresponding to each target pixel as the prediction result and the component category corresponding to each target pixel as the ground value.

[0012] In the sixth exemplary implementation, determining all three-dimensional points corresponding to each component name based on the component name corresponding to each three-dimensional point and the probability information includes: The probability information and the component name corresponding to each target pixel are used as input data and fed into the trained data fusion network to obtain all three-dimensional points corresponding to each component name.

[0013] In the seventh exemplary implementation, training the data fusion network includes: The data fusion network is trained by invoking the fusion loss function formula, which is: ; In the formula, L ( Ppred , Pseg ) represents the fusion loss function relationship. Ppred The prediction result, Pseg For the stated truth value, c This represents the total number of component categories. i Indicates the first i Component categories.

[0014] In the eighth exemplary implementation, obtaining the component category corresponding to at least one target pixel in at least one two-dimensional content image includes: Each two-dimensional content image is sequentially input into the trained segmentation model to obtain the predicted pixel region for each component category in the current two-dimensional content image.

[0015] In the ninth exemplary implementation, obtaining the probability information of each three-dimensional point belonging to each type of component includes: A component category recognition network is pre-constructed; the component category recognition network includes a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron; Multidimensional data information is determined based on the component categories contained in the three-dimensional content dataset, the total number of three-dimensional points in each three-dimensional content, and the maximum number of component categories to which each three-dimensional point belongs. The multidimensional data information is input into the component category recognition network. The multilayer perceptrons of the component category recognition network perform dimensionality reduction processing on the multidimensional data information to obtain the probability information of each three-dimensional point belonging to each type of component.

[0016] In the tenth exemplary implementation, determining the multidimensional data information based on the component categories included in the three-dimensional content dataset, the total number of three-dimensional points for each three-dimensional content, and the maximum number of component categories to which each three-dimensional point belongs includes: Obtain the component categories contained in the 3D content dataset; Each component category is converted into a multidimensional one-hot encoded vector.

[0017] In the eleventh exemplary implementation, the step of fusing the component names corresponding to each target pixel based on the three-dimensional content dataset, and determining all three-dimensional points corresponding to each component name according to the fusion result, includes: Pre-construct a network structure that fuses the correspondence between multi-view pixels and component names; The multidimensional data information determined based on the three-dimensional content dataset and the component name corresponding to each target pixel are input into the multi-view pixel and component name correspondence fusion network structure to obtain all three-dimensional points corresponding to each component name; The multi-view pixel-to-part name correspondence fusion network structure includes a first input terminal, a second input terminal, a third input terminal, a part category recognition network, a data fusion network, and an output terminal. The first input terminal is connected to the part category recognition network, and both the part category recognition network and the second input terminal are connected to the data fusion network. The data fusion network is connected to the output terminal. The first input terminal is used to input the multi-dimensional data information, and the second input terminal is used to input the part name corresponding to each target pixel. During the training process, the multi-view pixel-to-part name correspondence fusion network structure also includes a third input terminal, which is connected to the data fusion network and is used to input the part category corresponding to each pixel in each two-dimensional content image.

[0018] In the twelfth exemplary implementation, the step of using the text question-answering model to obtain the component name information described in text form within the three-dimensional content dataset includes: Retrieve the name of at least one 3D content item from the 3D content dataset; Based on the names of each 3D content, the component name information described in text form is obtained using the text question-and-answer model.

[0019] In the thirteenth exemplary implementation, obtaining the component name information in text form based on the names of each three-dimensional content using the text question-answering model includes: Based on the user's input of questions and auxiliary questions related to the 3D content components, generate a set of questions and a set of auxiliary questions; The set of auxiliary questions is input into the text question-answering model, and the text question-answering model is used to obtain the answers to the auxiliary questions corresponding to the set of auxiliary questions. The set of questions and the answers to the auxiliary questions are input into the text question-answering model to obtain the component name information.

[0020] In the fourteenth exemplary implementation, acquiring two-dimensional content images from different perspectives corresponding to at least one three-dimensional content in the three-dimensional content dataset includes: The pose information of the virtual camera is calculated based on its position and orientation; the virtual camera is fixedly pointed at each 3D content in the 3D content dataset, and the virtual camera is deployed in multiple locations to cover the acquisition of data from multiple perspectives of each 3D content; Based on the pose information of each virtual camera, each 3D content is rendered to obtain 2D content images of the same 3D content from different viewpoints.

[0021] In the fifteenth exemplary implementation, before rendering each 3D content according to each pose information of the virtual camera, the method further includes: All 3D component points of each 3D content in the 3D content dataset are aligned with the target coordinate system, and the dimensions of each 3D content are normalized to a preset standard size.

[0022] In the sixteenth exemplary implementation, calculating the pose information of the virtual camera based on its position and orientation includes: Set the virtual camera in a spherical coordinate system; The positions of multiple virtual cameras can be obtained by changing the radius and / or polar angle and / or azimuth angle; Based on the orientation of the virtual camera, multiple pose information is calculated according to different positions of the virtual camera.

[0023] Another aspect of the present invention provides a dataset generation device based on a pre-trained language model, comprising: The question-answering model building module is used to fine-tune the pre-trained language model based on the execution of the question-answering task using a three-dimensional content question-answering sample set to obtain a text question-answering model. The image and text data acquisition module is used to acquire the component name information described in text form in the three-dimensional content dataset using the text question-and-answer model; and to acquire two-dimensional content images of at least one three-dimensional content in the three-dimensional content dataset from different perspectives. The image and text data correspondence module is used to determine all three-dimensional points corresponding to each component name based on the component name information and each two-dimensional content image, based on the correspondence between pixels, component names, and each three-dimensional point, at least one three-dimensional content component name and its corresponding three-dimensional point. The dataset generation module is used to obtain a dataset corresponding to text and three-dimensional content components based on the component names and three-dimensional points of each three-dimensional content in the three-dimensional content dataset.

[0024] Another aspect of the present invention provides a three-dimensional content method based on a pre-trained language model, comprising: Based on the steps of the dataset generation method based on the pre-trained language model as described in the previous item, a dataset corresponding to text content and 3D parts is obtained; Based on a dataset of text content and 3D component correspondences, a 3D content generation network model is trained to automatically generate 3D content based on user input data.

[0025] In another aspect, the present invention provides a three-dimensional content generation device based on a pre-trained language model, comprising: The dataset acquisition module is used to obtain a dataset corresponding to text content and 3D parts based on the steps of the dataset generation method based on the pre-trained language model as described in the previous item. The model training module is used to train a 3D content generation network model based on a dataset of text content and 3D parts correspondence. The content generation module is used to automatically generate 3D content based on user input data using a trained 3D content generation network model.

[0026] The present invention also provides an electronic device including a processor for executing a computer program stored in a memory to implement the steps of the dataset generation method based on a pre-trained language model as described in any of the preceding claims.

[0027] Finally, the present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the dataset generation method based on a pre-trained language model as described in any of the preceding claims.

[0028] The advantage of the technical solution provided by this invention lies in that it utilizes the powerful cognitive ability of a large-scale pre-trained language model to generate a set of component names contained in 3D content, and obtains a large number of two-dimensional images by rendering the three-dimensional content data of the three-dimensional content dataset from different perspectives. By using all the pixels belonging to the component names in each two-dimensional content image, the correspondence between each point of the three-dimensional content and the component name can be determined. In this way, a large-scale, high-quality dataset of text content and corresponding 3D components can be generated, which meets the user's realistic needs for the scale and accuracy of the dataset required in the process of generating three-dimensional content and is conducive to improving the accuracy of three-dimensional content generation.

[0029] Furthermore, the present invention also provides a corresponding implementation device, electronic device, and readable storage medium for the dataset generation method based on pre-trained language models, further making the method more practical. The three-dimensional content generation method, device, electronic device, and readable storage medium have corresponding advantages.

[0030] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a dataset generation method based on a pre-trained language model provided by this invention; Figure 2 A schematic diagram of an exemplary structural framework for the data fusion network provided by the present invention; Figure 3 A schematic diagram of an exemplary structural framework for the component category identification network provided by the present invention; Figure 4 This is a schematic diagram of an exemplary structural framework for the multi-view pixel-to-component name correspondence fusion network structure provided by the present invention. Figure 5 A schematic diagram of the hardware framework for an exemplary application scenario provided by the present invention; Figure 6 A flowchart illustrating another dataset generation method based on a pre-trained language model provided by the present invention; Figure 7 A flowchart illustrating a three-dimensional content generation method based on a pre-trained language model provided by the present invention; Figure 8 A structural diagram of a specific embodiment of the dataset generation device based on a pre-trained language model provided by the present invention; Figure 9 A structural diagram of a specific embodiment of the 3D content generation device based on a pre-trained language model provided by the present invention; Figure 10 A structural diagram of one specific embodiment of the electronic device provided in this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments.

[0034] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. The terms "first," "second," "third," "fourth," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0035] Various non-limiting embodiments of the present invention are described in detail below. To better illustrate the invention, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that the invention can be practiced without these specific details. In other instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0036] Please see first. Figure 1 , Figure 1 This is a flowchart illustrating a dataset generation method based on a pre-trained language model provided by the present invention. The present invention may include the following: S101: Using a three-dimensional content question-answering sample set, the pre-trained language model is fine-tuned based on the question-answering task to obtain a text question-answering model.

[0037] In this embodiment, the pre-trained language model can be any model trained on a large-scale dataset and possessing strong generalization ability. The neural network algorithm structure used for training the pre-trained language model includes, but is not limited to, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory). It can also be a model built with attention networks, such as LLM (Large Language Model), Transformer, BERT (Bidirectional Encoder Representation from Transformers), GPT (Generative Pre-trained Transformer), and Clip (Contrastive Language-Image Pre-training). This invention does not impose any limitations on these models. Here, an attention network refers to a network model trained using an attention mechanism. This model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, resulting in a more accurate output.

[0038] The training of a pre-trained language model comprises two stages: pre-training and fine-tuning. In the pre-training stage, a large-scale neural network algorithm structure is trained based on a large corpus for a specific language model training task. The resulting large-scale neural network algorithm structure and parameters constitute the pre-trained language model. In the fine-tuning stage, small-scale training is performed for a specific task objective (downstream task) and task data (downstream data) to make minor adjustments to the parameters of the pre-trained model, ultimately obtaining a model adapted to the specific task and data. In this embodiment, the task objective is a question-and-answer task, and the task data is a 3D content question-and-answer sample set. This set includes multiple training samples, each consisting of a 3D content name and its corresponding component names. By using this dataset—the current downstream application task—which is the 3D content question-and-answer sample set, any of the pre-trained language models can be fine-tuned to make it suitable for the downstream application task, i.e., the text question-and-answer model in this step.

[0039] S102: Use a text question-answering model to obtain the component name information described in text form from the 3D content dataset.

[0040] The text-based question-answering model built in the previous step, leveraging the powerful cognitive capabilities of its large-scale pre-trained language model, can generate a set of component names within the 3D content based on the input text-based question, i.e., the 3D content name. This is the component name information in this step. Here, the 3D content name is the name of the corresponding 3D entity, and the component names are the names of the parts, components, or unit structures that compose that 3D entity. For example, if the 3D content is a server, then the name of the 3D content is "server," and the component names would be indicator lights, motherboard, casing, monitor, host, etc.

[0041] S103: Obtain two-dimensional content images from different perspectives corresponding to at least one three-dimensional content in the three-dimensional content dataset.

[0042] The 3D content dataset includes multiple 3D content items. In this embodiment, the same operation can be performed on each 3D content item, that is, on multiple specified 3D content items, to obtain the correspondence between the component names and 3D component points in each 3D content item.

[0043] It is understandable that each 3D content in a 3D dataset is composed of multiple points, i.e., point cloud data. To obtain a large-scale and high-precision dataset, it is necessary to obtain as many high-precision component names and their corresponding 3D component points as possible. This embodiment performs multi-view rendering on each 3D content in the 3D dataset to obtain multiple 2D images, i.e., the 2D content images defined in this step, which can obtain the correspondence between 3D points and pixels in the 2D images. The selection and number of viewpoints can be flexibly determined according to the actual situation, and this invention does not impose any limitations on them.

[0044] S104: Based on the component name information and each two-dimensional content image, and based on the correspondence between pixels, component names, and each three-dimensional point, determine all three-dimensional points corresponding to each component name, including the component name and corresponding three-dimensional points of at least one three-dimensional content.

[0045] In this step, the two-dimensional content image is rendered from the three-dimensional content, so the correspondence between 3D component points and pixels in the 2D image can be established. The component name information is obtained from the three-dimensional content through a pre-trained language model, meaning that the component name information and the three-dimensional content have a correspondence. Therefore, there must be a correspondence between pixels, component names, and three-dimensional points. In S102 and S103, the component name information and each two-dimensional content image are obtained. By finding the correspondence between each component name in the component name information and the two-dimensional content image, all pixels belonging to the corresponding component name information in the rendering result of each two-dimensional content image can be obtained, thus obtaining all the three-dimensional points corresponding to each component name.

[0046] S105: Based on the corresponding component name and 3D point in each 3D content in the 3D content dataset, obtain the corresponding dataset of text and 3D content components.

[0047] Based on the previous step of determining the correspondence between component names and 3D points in each 3D content, for each 3D content, the correspondence between each point of that 3D content and the component name can be determined. Thus, the point belonging to each component name is the 3D component point of the corresponding 3D component. By statistically analyzing the correspondence between component names and 3D points obtained from all 3D content, a dataset of text and 3D content component correspondence can be obtained.

[0048] In the technical solution provided by this invention, the powerful cognitive ability of a large-scale pre-trained language model is used to generate a set of component names contained in the three-dimensional content. The three-dimensional content data of the three-dimensional content dataset is rendered from different perspectives to obtain a large number of two-dimensional images. By using all the pixels belonging to the component names in each two-dimensional content image, the correspondence between each point of the three-dimensional content and the component name can be determined. In this way, a large-scale, high-quality dataset of text content and corresponding three-dimensional components can be generated, which meets the user's practical needs for the scale and accuracy of the dataset required in the three-dimensional content generation process and is conducive to improving the accuracy of three-dimensional content generation.

[0049] In the above embodiments, there is no limitation on how to determine all the three-dimensional points corresponding to each component name. This embodiment provides a method for determining the correspondence between each component name and three-dimensional points, which may include the following steps: Based on the component name information and each two-dimensional content image, determine the component name corresponding to at least one target pixel in at least one two-dimensional content image; based on the three-dimensional content dataset, perform fusion processing on the component names corresponding to each target pixel, and determine all three-dimensional points corresponding to each component name based on the fusion result.

[0050] It is understandable that not all pixels in a 2D content image correspond to a component name. For example, the background does not belong to any component, and therefore, background pixels do not have component names. This embodiment aims to determine the pixels corresponding to each component name of the object. For ease of description, this embodiment defines pixels with component names as target pixels. Furthermore, depending on different application scenarios, target pixels can be further limited to pixels with component names required by the user. For each component name contained in the component name information, all pixels belonging to that component name in each image rendering result can be determined through the component name and each 2D content image. It is understandable that overlapping phenomena inevitably occur in the content rendered from multiple perspectives. Each point of the 3D content is very likely to be rendered in different 2D content images, thus corresponding to different pixels in different 2D content images, and these pixels may correspond to different component categories. In order to obtain the correspondence between each point of the 3D content and the component name, it is necessary to fuse the correspondence results between pixels from multiple perspectives and the component names. Based on the fused result, the correspondence between each point of the 3D content and the component name can be accurately determined, thereby improving the generation accuracy of the 3D content.

[0051] The above embodiments do not limit how to determine the correspondence between each pixel and component name in a multi-view two-dimensional content image. The present invention also provides an exemplary implementation method, which may include the following: Each component name in the component name information is combined with at least one two-dimensional content image to obtain multiple image-text combination results; based on the component name in each image-text combination result, the component name corresponding to each target pixel in each two-dimensional content image is obtained by performing image segmentation processing on the two-dimensional content image in the corresponding image-text combination result.

[0052] In this embodiment, the image and text combination result is a set obtained by arbitrarily combining the names of each component and each two-dimensional content image. For example, if there are q 3D component names in the component name information, p rendered two-dimensional content images can be obtained through S103. Then, they are combined to obtain p*q combination results. That is, the image and text combination result includes p*q combination results. Each result in the combination result includes the component name and the rendered two-dimensional content image. After obtaining the image combination results, for each combination result, the matching image region is determined in the corresponding two-dimensional content image based on the component name in the current result. For example, the three-dimensional content is a server, and the component name information may include indicator lights, motherboard, shell, etc. The three-dimensional point cloud data of the server is rendered to obtain multiple two-dimensional server images. For the combination result of the indicator light and the first two-dimensional server image, based on the indicator light, the image block belonging to the indicator light can be determined in the first two-dimensional server image. The so-called image segmentation processing is to locate the image block, i.e. the image region, belonging to the corresponding component name in the two-dimensional content image. Based on the located image region, all pixels belonging to the component name in each image rendering result can be obtained. By performing statistical processing, the component name corresponding to each target pixel in each two-dimensional content image can be obtained.

[0053] Furthermore, in order to improve the accuracy and efficiency of determining the correspondence between pixels and component names in multi-view two-dimensional content images, based on the above embodiments, the present invention also provides another efficient implementation method, which may include the following: The "Segment All" model is pre-trained; each image-text combination result is sequentially input into the "Segment All" model to obtain all pixels in the current 2D content image that belong to the corresponding component name; based on all pixels in each 2D content image that belong to the corresponding component name, the component name corresponding to each target pixel in each 2D content image is determined.

[0054] This embodiment utilizes a pre-trained model based on image-text segmentation to map each pixel of a two-dimensional content image to a component name. Those skilled in the art can choose any pre-trained model capable of image-text segmentation and generate a model suitable for executing this embodiment through pre-training and fine-tuning stages, without affecting the implementation of the invention. For example, this embodiment uses the Segment Anything Model (SAM) to map each pixel of a two-dimensional content image to a component name.

[0055] As is known, this embodiment, by leveraging the powerful segmentation capabilities of a large-scale pre-trained model for image-text segmentation, can locate the pixel region in the image corresponding to the component name based on the user's textual prompts, i.e., the component name, and obtain the correspondence between the text and the 3D component.

[0056] The above embodiments do not impose any limitations on how to fuse the multi-view pixel and component name correspondence results. In order to improve the fusion accuracy and obtain more accurate correspondence results, the present invention also provides an exemplary implementation method of "fusing the component names corresponding to each target pixel based on the three-dimensional content dataset, and determining all three-dimensional points corresponding to each component name based on the fusion result", which may include the following: obtaining the probability information of each three-dimensional point belonging to each type of component based on the three-dimensional content dataset; determining all three-dimensional points corresponding to each component name based on the component name and probability information corresponding to each three-dimensional point.

[0057] In this embodiment, to avoid the same 3D point being mapped to different pixels multiple times, which could lead to inaccurate results, this step pre-calculates the probability that each 3D point belongs to each type of component. Each type of component refers to the types of components contained in the corresponding 3D content dataset. By considering this probability information during fusion, the probability of the same 3D point being mapped to different pixels multiple times can be minimized, thereby ensuring the accuracy of the final mapping.

[0058] As an efficient implementation method of this embodiment, the present invention can perform data fusion by leveraging the advantages of pre-trained models, and may include the following: A data fusion network is pre-constructed; the component category corresponding to at least one target pixel in at least one two-dimensional content image is obtained; the data fusion network is trained based on the probability information of each three-dimensional point belonging to each component category, using the component name corresponding to each target pixel as the prediction result and the component category corresponding to each target pixel as the ground value.

[0059] In this embodiment, the input to the data fusion network is the component category corresponding to each pixel in each two-dimensional content image, and the output is the prediction result corresponding to the pixel and the component name, thereby obtaining all three-dimensional points corresponding to each component name. During the training of the data fusion network, the model training needs to be continuously guided by the difference between the predicted value and the ground truth. For each two-dimensional content image, the component category corresponding to each pixel in each two-dimensional content image determined by the pre-trained model based on image segmentation can be used as the ground truth, such as... Figure 2 As shown. For example, the segmentation model trained in the aforementioned steps can be used directly. That is, each two-dimensional content image can be input into the trained segmentation model to obtain the predicted pixel region of each component category in the current two-dimensional content image. After the data fusion network is trained, that is, in the inference stage, the probability information and the component name corresponding to each target pixel can be used as input data and directly input into the trained data fusion network to obtain all three-dimensional points corresponding to each component name.

[0060] For example, in order to improve the training accuracy of the data fusion network, this embodiment can use the following fusion loss function to train the data fusion network: ; In the formula, L ( Ppred , Pseg ) represents the fusion loss function relationship. Ppred For the predicted results, Pseg If it is true, c This represents the total number of component categories. i Indicates the first i Component categories.

[0061] The above embodiments do not limit how to determine the probability information of each three-dimensional point belonging to each type of component. Based on the above embodiments, the present invention also provides an exemplary method for determining probability information, which may include the following: A component category recognition network is pre-constructed, comprising a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron. Multidimensional data information is determined based on the component categories contained in the 3D content dataset, the total number of 3D points in each 3D content, and the maximum number of component categories to which each 3D point belongs. This multidimensional data information is then input into the component category recognition network, and the multilayer perceptrons within the network perform dimensionality reduction processing to obtain the probability information of each 3D point belonging to each component category.

[0062] The component category recognition network in this embodiment is used to predict the probability that each 3D point in the 3D content belongs to each component category. It is composed of a multi-layer MLP (Multi-Layer Perception) network structure, such as... Figure 3As shown, the component category recognition network performs dimensionality reduction on the input 3D content through each MLP until it obtains 2D data. This 2D data represents the probability that each 3D point belongs to that component category. Considering that a 3D point belongs to one component type, to ensure a fixed-length input for the network model, the component categories contained in the 3D content dataset are obtained; each component category is converted into a multi-dimensional one-hot encoded vector. One-hot encoding, also known as one-bit encoding, uses an N-bit state register to encode N states. Each state has its own independent register bit, and only one bit is valid at any given time. For example, if all 3D content contains c categories of part names, then each category can be converted into a c-dimensional one-hot encoding; each 3D content has n 3D points, and each 3D point can be assigned at most k part categories, resulting in [n, c, k]-dimensional input data; using various MLPs, the input can be converted into [n, c, 16]-dimensional, [n, c, 8]-dimensional, and [n, c]-dimensional vectors respectively, and the final [n, c] is the output predicted category result, that is, the probability that each point belongs to any of the c-category parts.

[0063] Furthermore, in order to improve the efficiency of determining the correspondence between component names and three-dimensional points, based on the above embodiments, the present invention also provides another exemplary embodiment, which may include the following: A multi-view pixel-to-part name correspondence fusion network structure is pre-constructed; the multi-dimensional data information determined based on the 3D content dataset and the part name corresponding to each target pixel are input into the multi-view pixel-to-part name correspondence fusion network structure to obtain all 3D points corresponding to each part name.

[0064] In this embodiment, the multi-view pixel-to-part name correspondence fusion network structure is an end-to-end network model, which includes two parts: a data fusion network and a part category recognition network, such as... Figure 4 As shown, the structure of the data fusion network differs during inference and training processes, as described in the above embodiments. The entire multi-view pixel-to-part name correspondence fusion network structure includes a first input terminal, a second input terminal, a third input terminal, a part category recognition network, a data fusion network, and an output terminal. The first input terminal is connected to the part category recognition network, and both the part category recognition network and the second input terminal are connected to the data fusion network. The data fusion network is connected to the output terminal. The first input terminal is used to input multi-dimensional data information, and the second input terminal is used to input the part name corresponding to each target pixel. During training, the multi-view pixel-to-part name correspondence fusion network structure also includes a third input terminal, which is connected to the data fusion network and used to input the part category corresponding to each pixel in each two-dimensional content image.

[0065] During training, the 3D content used for training sample data is rendered from random perspectives to obtain a set of rendered images. The predicted part category of each 3D point is mapped to an image pixel, resulting in a part category prediction for each pixel in the image. Based on a large-scale pre-trained model for image segmentation, the pixel region predicted for each part category in the image is obtained, i.e., the category corresponding to each pixel. The cross-entropy loss function is used to train the entire multi-view pixel-to-part-name mapping fusion network structure. By using the 2D image segmentation results obtained from the large-scale pre-trained model SAM based on image segmentation as ground truth to train the multi-view pixel-to-part-name mapping fusion network structure, the network model can be guided towards achieving highly accurate segmentation results for both the fused 3D content and the rendered 2D image from any perspective. This ensures high accuracy for the multi-view pixel-to-part-name mapping fusion network structure while avoiding tedious and costly manual data annotation, thus improving its practicality. Finally, based on the trained multi-view pixel-part name correspondence fusion network structure, the correspondence results of multi-view pixels and part names can be fused to obtain the correspondence result of each point of the 3D content with a unique part name, that is, to obtain all the 3D points corresponding to each part name contained in the 3D content.

[0066] As can be seen from the above, this embodiment combines a network structure that integrates multi-view pixel and component name correspondence obtained by training a large-scale pre-trained model. This network structure can accurately and efficiently obtain all 3D points corresponding to each component name in the 3D content.

[0067] The above embodiments do not limit the process of obtaining component name information from a three-dimensional dataset. Based on this, the present invention also provides an illustrative implementation, which may include the following: Obtain the name of at least one 3D content from the 3D content dataset; based on the name of each 3D content, use the text question-answering model to obtain the component name information in text form.

[0068] The pre-trained language model possesses powerful cognitive capabilities, enabling it to produce high-quality responses based on the input text format. Therefore, this embodiment uses the name of each 3D content as the input text format, leveraging the powerful cognitive capabilities of the pre-trained language model to obtain the set of component names contained within that 3D content name.

[0069] Considering that directly asking questions to a large-scale pre-trained language model may result in low-quality answers, i.e., inaccurate component name information, this invention proposes a component name information generation method based on a pre-trained language model, which includes the following steps: generating a set of questions and a set of auxiliary questions based on user-inputted 3D content component questions; inputting the set of auxiliary questions into a text question-answering model to obtain the answers to the auxiliary questions corresponding to the set of auxiliary questions; and inputting the set of questions and the answers to the auxiliary questions into the text question-answering model to obtain the component name information.

[0070] In this embodiment, the large-scale pre-trained language model is first asked questions about the principles related to 3D content, followed by the target question. The questions about the principles related to 3D content are also auxiliary questions, while the target question is related to the names of 3D components contained in the 3D content, which is the question about 3D content components in this embodiment. For example, the name of each 3D content item in the 3D content dataset can be obtained first, such as "airplane," "seat," and "bicycle." A set of questions is constructed based on the user-inputted questions. Since 3D content has at least one or more component divisions, to obtain the component names for any 3D content, questions can be asked of the large-scale pre-trained language model. Multiple questions are set, such as "What component names does 3D content contain?", "What parts can 3D content be divided into?", and "Which 3D components can constitute 3D content?", resulting in a set of questions about 3D content components. Then, an auxiliary question set is constructed based on the user-inputted auxiliary questions. To assist the large-scale pre-trained language model in answering questions better, auxiliary questions can be set, including questions about the basic principles related to 3D content, thus giving the large-scale pre-trained model steps to think. The auxiliary questions are answered first, followed by the target question. A set of auxiliary questions is set, such as "What is the function of 3D content?", "What are the design principles of 3D content?", and "What is 3D content essentially?", to obtain a set of auxiliary questions for 3D content components. Then, based on this set of auxiliary questions, each auxiliary question is posed to a large-scale pre-trained language model, yielding a series of answers to these auxiliary questions. Finally, the original questions, along with the answers to the auxiliary questions, are input into the large-scale pre-trained language model to obtain the final answer to the set of 3D content component names.

[0071] As can be seen from the above, this embodiment, through a thought chain approach based on auxiliary question answers, can significantly improve the answer accuracy of the pre-trained language model, obtaining a more reasonable 3D component segmentation result that aligns with public perception, and effectively improving the accuracy of the dataset corresponding to text and 3D content components. Furthermore, the use of an interactive user input strategy, i.e., obtaining the set of components contained in the 3D content by asking the user, enhances practicality and flexibility.

[0072] The above embodiments do not limit the rendering of two-dimensional content images based on three-dimensional point cloud data. The present invention also provides an illustrative implementation method, which may include the following: The pose information of the virtual camera is calculated based on its position and orientation. The virtual camera is fixedly pointed at each 3D content in the 3D content dataset. The virtual camera is deployed in multiple locations to cover and collect data from multiple perspectives of each 3D content. Based on the pose information of each virtual camera, each 3D content is rendered to obtain 2D content images of the same 3D content from different perspectives.

[0073] It is understandable that the pose of a virtual camera consists of two parts: position and orientation. To ensure comprehensive acquisition of information from all perspectives of the 3D content, this embodiment sets up a large number of virtual cameras surrounding the upper half of the 3D content. During the calculation process, the position of the virtual cameras varies, but the orientation of the virtual cameras remains fixed, pointing from the camera position to the center of the object corresponding to the 3D content. As a simple method for deploying virtual camera positions, this embodiment sets the virtual cameras in a spherical coordinate system, and the coordinate position of the virtual cameras can be represented as follows: According to the calculation formula in spherical coordinates: , , Where r is the radius of the sphere, θ is the polar angle in the vertical direction, and φ is the azimuth angle in the horizontal direction. By changing the radius and / or the polar angle and / or the azimuth angle, the positions of multiple virtual cameras can be obtained. Based on the orientation of the virtual cameras, multiple pose information is calculated according to the different positions of the virtual cameras. For example, if the radius r exists... r In this case, θ exists n θ In this case, φ exists n φ In this case, the final rendering result can be n=n r ×n θ ×n φ Images from multiple perspectives. For example, we can fix the radius r to four cases: 3, 2.5, 2, and 1.5; let the polar angle θ have two cases: 60 degrees and 90 degrees; and let the azimuth angle φ have eight cases: 0 degrees, 45 degrees, up to 360 degrees. Then there are a total of 64 virtual camera positions (4×2×8), resulting in 64 rendered 2D content images. Based on the calculated virtual camera positions, the 3D model is rendered into multiple 2D content images, allowing us to determine which 3D points are rendered to which pixels in each 2D content image, thus obtaining the 3D point corresponding to each pixel in each 2D content image.

[0074] To further improve the accuracy of the final 2D content image generation, the 3D point cloud data needs to be preprocessed before generating the 2D content image. This preprocessing may include the following: All 3D component points of each 3D content in the 3D content dataset are aligned with the target coordinate system, and the dimensions of each 3D content are normalized to the preset standard dimensions.

[0075] Since the position of the 3D point cloud of each 3D content in space is not fixed, this embodiment performs two preprocessing steps—coordinate system alignment and scale scaling—to ensure controllability in the rendering process. Coordinate system alignment involves unifying all points of the 3D content to align with the world coordinate system. First, the center of the current 3D content is calculated. Then, based on this center and the original coordinates of each point, the new coordinates of each point are calculated. For example, the original coordinates of each point minus the bus coordinates equal the new coordinates, thus completing the coordinate system alignment preprocessing. Scale scaling involves normalizing the 3D content to a standard scale. For example, the object corresponding to the 3D content can be scaled to a cube with a side length of 1. First, the difference between the maximum and minimum values ​​of the 3D content on the x, y, and z axes is calculated. The maximum value is taken, and the reciprocal is taken as the scaling factor. The coordinates of each point of the 3D content are multiplied by a scaling factor to complete the scale scaling of each 3D content point.

[0076] To enable those skilled in the art to more clearly understand the technical solution of this invention, this invention provides an exemplary implementation method, illustrating some possible application scenarios involved in the technical solution of this invention, such as... Figure 5 As shown, Figure 5 A schematic diagram of the hardware framework applicable to the dataset generation method based on a pre-trained language model provided by the present invention may include the following: The hardware framework may include a server 51 and a user terminal 52, which are connected via a network 53. The server 51 is equipped with a processor for executing the dataset generation method based on a pre-trained language model as described in any of the above embodiments. The user terminal 52 is a client with a human-computer interaction interface, used to input various control commands and text descriptions, such as questions from a set of questions and a set of auxiliary questions, to the server 51.

[0077] The server 51 in this embodiment is based on Figure 6 The illustrated process generates a large-scale dataset mapping text to 3D parts. This is performed on each 3D element in the 3D content dataset. Figure 6The operation described above obtains the component name and its corresponding 3D component point in text form for each 3D content, completing the construction of a text-to-3D component mapping dataset based on a large-scale pre-trained model. First, the name of each 3D content in the dataset is input into a large-scale pre-trained language model to obtain the set of component names for that 3D content. Second, a virtual camera setting method for 3D content rendering is used to render a large number of 2D content images from different viewpoints. Each 2D content image can be defined as viewpoint 1 image, ..., viewpoint n image. Third, a large-scale pre-trained language model based on image-text segmentation is used to perform image-text segmentation processing on the component name set and each 2D content image, obtaining the correspondence between each pixel in each 2D content image and the component name, i.e., obtaining the component name corresponding to each pixel in the rendered image, i.e., the correspondence between viewpoint 1 pixel and component name, ..., viewpoint n pixel and component name. Then, the correspondence between pixels and component names in each 2D content image is fused to obtain the correspondence between each point of the 3D content and the component name; finally, a large-scale text-to-3D component mapping dataset is obtained based on each 3D content in the dataset.

[0078] It should be noted that the above application scenarios are shown only to facilitate understanding of the ideas and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0079] As can be seen from the above, this embodiment can generate a large-scale, high-quality dataset of text content and corresponding 3D parts, which is beneficial for generating high-quality 3D content.

[0080] Finally, based on the text content and 3D part correspondence dataset generated by any of the above embodiments, this embodiment also provides an application embodiment, such as... Figure 7 As shown, it may include the following: S701: Obtain the dataset corresponding to the text and the 3D content components.

[0081] The dataset corresponding to the text and 3D content components in this step is the dataset obtained by using the dataset generation method based on the pre-trained language model described in any of the above embodiments, and will not be repeated here.

[0082] S702: Based on the text and 3D content component correspondence dataset, a 3D content generation network model is trained to automatically generate 3D content based on user input data using the trained 3D content generation network model.

[0083] After obtaining the dataset corresponding to the text and 3D content components in the previous step, this dataset is used as the training and testing sample sets to train the 3D content generation network model, thereby enabling the model to learn and generate 3D content. The 3D content generation network can utilize any machine learning algorithm to build its network model structure, and can also be built by combining it with large-scale pre-trained models; none of this affects the implementation of this invention.

[0084] For example, a 3D content generation network model may include an input terminal, a text question-answering model, a dual-channel conditional control component generation network, a coarse-grained 3D component fusion network, a sampling structure, a fine-grained 3D component fusion network, and an output terminal. The input terminal is used to input any form of 3D content generation request, enabling the 3D generation network model to clearly define the final 3D content to be generated. The output terminal is the final generated 3D content. The text question-answering model generates component name text data based on 3D content description data. The dual-channel conditional control component generation network generates 3D component point cloud data based on the 3D content description data carried in the 3D content generation request. The coarse-grained 3D component fusion network determines the spatial position of each 3D component based on the 3D content description data, component name text data, and 3D component point cloud data. The sampling structure samples a fixed number of points from the initial fused point cloud data obtained by moving each 3D component to its corresponding spatial position to obtain fused point cloud data. The fine-grained 3D component fusion network calculates the offset information of each point of each 3D component based on the 3D content description data and the fused point cloud data. The output terminal adjusts the spatial position of the corresponding 3D components based on the offset information and generates 3D content based on the adjusted 3D components.

[0085] The dual-channel conditional control component generation network includes an overall text descriptor extraction network, a component text descriptor extraction network, a diffusion time processing network, a point cloud data processing network, a sequence data processing neural network, and a predicted point cloud noise output network. The overall text descriptor extraction network generates overall control descriptor data for 3D content description data; the component text descriptor extraction network generates component control descriptor data for component name text data; the diffusion time processing network generates diffusion time descriptor data; the point cloud data processing network generates point cloud data descriptor data; the sequence data processing neural network learns feature representations of the overall control descriptor data, the component control descriptor data, the diffusion time descriptor data, and the point cloud data descriptor data; and the predicted point cloud noise output network predicts point cloud noise data at the current time based on the feature representations. The overall text descriptor extraction network includes an overall text input terminal, an overall text feature extraction model, and an overall control sub-generation model. The overall text input terminal is used to input 3D content description data, and the overall text feature extraction model is used to extract text features from the 3D content description data as initial overall descriptor data. The overall control sub-generation model is used to subdivide the features in the initial overall descriptor data to obtain fine-grained overall text features, which are then used as overall control descriptor data. Similarly, the component text descriptor extraction network includes a component text input terminal, a component text feature extraction model, and a component control sub-generation model. The component text input terminal is used to input component name text data, and the component text feature extraction model is used to extract text features from the component name text data as initial text descriptor data. The component control sub-generation model is used to subdivide the features in the initial component descriptor data to obtain fine-grained component text features, which are then used as component control descriptor data. The predicted point cloud noise output network includes a noise point cloud data generation model, which is used to obtain the point cloud noise data added for each diffusion by performing regression analysis on the diffuser output descriptor data output by the sequence data processing neural network.The dual-channel training process for adapting the dual-channel conditional control component generation network is as follows: Noise point cloud data is calculated from the noise-free 3D point cloud data in the 3D content sample dataset; training sample data corresponding to the 3D content description data and the component name text data are obtained from the 3D content sample dataset; the overall text descriptor extraction network is set to an inactive state, and the component text descriptor extraction network is set to an active state; the component text descriptor extraction network is trained using the noise point cloud data, diffusion time, and the training sample data corresponding to the component name text data; the trained component text descriptor extraction network is frozen, and the overall text descriptor extraction network is set to an active state; the overall text descriptor extraction network is trained using the noise point cloud data, diffusion time, and the training sample data corresponding to the 3D content description data; the trained component text descriptor extraction network is unfrozen, and the component text descriptor extraction network and the overall text descriptor extraction network are trained again using the noise point cloud data, diffusion time, the training sample data corresponding to the 3D content description data, and the training sample data corresponding to the component name text data.

[0086] For example, a coarse-grained fusion network for 3D components may include a text feature extraction subnetwork, a joint feature layer, and a joint feature processing subnetwork. The text feature extraction subnetwork includes a global text feature extraction model, a component text feature extraction model, and a global feature extraction model. The global text feature extraction model is used to extract high-dimensional global text features from the 3D content description data using a pre-trained language model. The component text feature extraction model is used to extract high-dimensional component text features from the component name text data using a pre-trained language model. The global feature extraction model includes a first multi-layer perception layer, a second multi-layer perception layer, a third multi-layer perception layer, and a fourth multi-layer perception layer connected in sequence. The output features of the fourth multi-layer perception layer are pooled to obtain the 3D global features of the 3D component point cloud data. The joint feature processing subnetwork includes a fifth multi-layer perception layer, a sixth multi-layer perception layer, a seventh multi-layer perception layer, and an eighth multi-layer perception layer. The joint feature processing subnetwork is used to continuously reduce the dimensionality of the joint features through the fifth, sixth, seventh, and eighth multi-layer perception layers until it reaches 1×6 dimensions, to obtain the 3D displacement and 3D scale of the 3D component.

[0087] The sampling structure can be used to: calculate the current number of points based on the total number of 3D parts and the total number of points in the 3D part point cloud; calculate the number of sampled points based on a preset sampling factor and the total number of 3D parts; and determine whether the sampling mode used for each 3D part is upsampling, downsampling, or no sampling by comparing the current number of points and the number of sampled points. To ensure high accuracy, the preset sampling factor can be 2 or 4. The complete 3D content contains the following number of 3D parts: k The number of point clouds for each 3D component is n part The total number of point clouds of the three-dimensional components obtained in step S103 is... n merge for n merge = k × n part Number of points in the sampled point cloud n sample for n sample = m × n part If there are one, then it is necessary to put n merge Sampled to n sample In other words, based on the current number of point clouds and the sampled point cloud data, the number of point clouds for each 3D component is determined from the actual point cloud data sampled to... n part × m / k , m For the preset sampling factor, k This represents the total number of 3D components. For each 3D component, according to... k and m There are three sampling scenarios depending on the differences. k = m If the corresponding 3D component is not sampled, then no sampling will be performed; if k > m Then downsampling is required. To improve the robustness of the algorithm and the sampling speed, a random downsampling method is adopted, that is, random downsampling is performed on the corresponding 3D parts, and one point can be randomly selected for removal until the number of remaining point clouds is [missing information]. n part × m / k。 like k < m Then upsampling is required. To improve the robustness of the algorithm and the sampling speed, random upsampling can be performed on the corresponding 3D parts. One point and two nearest neighbor points can be randomly selected, and their centroid coordinates can be interpolated as new points. This process is repeated until the current number of point clouds reaches a certain value.n part × m / k .

[0088] For example, a fine-grained fusion network for 3D parts may include a point cloud global feature extraction subnetwork, a global text feature extraction subnetwork, and an offset prediction subnetwork. The global text feature extraction subnetwork is used to extract text features from the 3D content description data and can use any network structure capable of extracting high-dimensional text features. The point cloud global feature extraction subnetwork is used to extract global features from the initially fused 3D part point cloud data. It may include sequentially connected tenth-layer, eleventh-layer, twelfth-layer, and thirteenth-layer perceptrons. These layers perform dimensionality upscaling on the input 3D part point cloud data, and the output features of the thirteenth-layer perceptron are then pooled to obtain the global features of the fused point cloud data. To supplement detailed features and improve overall model performance, the output of any branch can be used as local descriptor data; for example, the intermediate n×2048 data points can be used as local descriptor data. The offset prediction subnetwork can be used to fuse features extracted by the point cloud global feature extraction subnetwork, the overall text feature extraction subnetwork, and local detail features, and continuously reduce the dimensionality of the fused features until a six-dimensional output feature is obtained. The offset prediction subnetwork may include a feature mixing layer, a fourteenth multi-layer perceptron layer, a fifteenth multi-layer perceptron layer, a sixteenth multi-layer perceptron layer, and a sixteenth multi-layer perceptron layer; the feature mixing layer is used to mix the output features of the twelfth multi-layer perceptron layer as a copy of local features, multiple copies of global features of 3D parts, and multiple copies of overall text features output by the overall text feature extraction subnetwork; the mixed features are then subjected to dimensionality reduction processing by the fourteenth multi-layer perceptron layer, the fifteenth multi-layer perceptron layer, the sixteenth multi-layer perceptron layer, and the sixteenth multi-layer perceptron layer to obtain the offset information of each point of each 3D part. As can be seen from the above, this embodiment, by generating a large-scale, high-quality dataset of text and 3D content parts, is beneficial to improving the quality of the final 3D content generation.

[0089] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. The above figures are only schematic representations and do not represent that the execution order can only be like this.

[0090] This invention also provides a corresponding apparatus for generating datasets and 3D content based on pre-trained language models, further enhancing the practicality of the method. The apparatus can be described from both functional module and hardware perspectives. The following describes the dataset generation apparatus based on pre-trained language models provided by this invention. This apparatus is used to implement the dataset generation method based on pre-trained language models provided by this invention. In this embodiment, the dataset generation apparatus based on pre-trained language models may include or be divided into one or more program modules. These program modules are stored in a storage medium and executed by one or more processors to complete the dataset generation method based on pre-trained language models disclosed in Embodiment 1. The program module referred to in this invention is a series of computer program instruction segments capable of performing specific functions, which is more suitable than the program itself for describing the execution process of the dataset generation apparatus based on pre-trained language models in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The dataset generation apparatus based on pre-trained language models described below can be referred to in correspondence with the dataset generation method based on pre-trained language models described above.

[0091] From the perspective of functional modules, please refer to Figure 8 , Figure 8 This is a structural diagram of a dataset generation device based on a pre-trained language model provided by the present invention in one specific embodiment. The device may include: The question-answering model building module 801 is used to fine-tune the pre-trained language model based on the execution of the question-answering task using a three-dimensional content question-answering sample set to obtain a text question-answering model. The image and text data acquisition module 802 is used to acquire the component name information described in text form contained in the three-dimensional content dataset using the text question-and-answer model; and to acquire two-dimensional content images of at least one three-dimensional content in the three-dimensional content dataset from different perspectives. The image and text data correspondence module 803 is used to determine all three-dimensional points corresponding to each component name based on the component name information and each two-dimensional content image, based on the correspondence between pixels, component names, and each three-dimensional point, at least one three-dimensional content component name and its corresponding three-dimensional points. The dataset generation module 804 is used to obtain a dataset corresponding to text and three-dimensional content components based on the component names and three-dimensional points in each three-dimensional content in the three-dimensional content dataset.

[0092] For example, in some embodiments of this example, the above-mentioned image and text data correspondence module 803 can also be used for: Based on the component name information and each two-dimensional content image, determine the component name corresponding to at least one target pixel in at least one two-dimensional content image; based on the three-dimensional content dataset, perform fusion processing on the component names corresponding to each target pixel, and determine all three-dimensional points corresponding to each component name based on the fusion result.

[0093] As an illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: Each component name in the component name information is combined with at least one two-dimensional content image to obtain multiple image-text combination results; based on the component name in each image-text combination result, the component name corresponding to each target pixel in each two-dimensional content image is obtained by performing image segmentation processing on the two-dimensional content image in the corresponding image-text combination result.

[0094] As an illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: The "Segment All" model is pre-trained; each image-text combination result is sequentially input into the "Segment All" model to obtain all pixels in the current 2D content image that belong to the corresponding component name; based on all pixels in each 2D content image that belong to the corresponding component name, the component name corresponding to each target pixel in each 2D content image is determined.

[0095] As another illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 may be further used for: Based on the 3D content dataset, obtain the probability information of each 3D point belonging to each type of component; based on the component name and probability information corresponding to each 3D point, determine all 3D points corresponding to each component name.

[0096] As an illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: A data fusion network is pre-constructed; the component category corresponding to at least one target pixel in at least one two-dimensional content image is obtained; the data fusion network is trained based on the probability information of each three-dimensional point belonging to each component category, using the component name corresponding to each target pixel as the prediction result and the component category corresponding to each target pixel as the ground value.

[0097] As an illustrative implementation of the above embodiment, the above-described image and text data corresponding module 803 can be further used for: The probability information and the component name corresponding to each target pixel are used as input data and fed into a trained data fusion network to obtain all 3D points corresponding to each component name.

[0098] As another illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: The data fusion network is trained using the fusion loss function, which is as follows: ; In the formula, L ( Ppred , Pseg ) represents the fusion loss function relationship. Ppred For the predicted results, Pseg If it is true, c This represents the total number of component categories. i Indicates the first i Component categories.

[0099] As another illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can be further used for: Each two-dimensional content image is sequentially input into the trained segmentation model to obtain the predicted pixel region for each component category in the current two-dimensional content image.

[0100] As another illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: A component category recognition network is pre-constructed, comprising a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron. Multidimensional data information is determined based on the component categories contained in the 3D content dataset, the total number of 3D points in each 3D content, and the maximum number of component categories to which each 3D point belongs. This multidimensional data information is then input into the component category recognition network, and the multilayer perceptrons within the network perform dimensionality reduction processing to obtain the probability information of each 3D point belonging to each component category.

[0101] As an illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can also be used for: Obtain the component categories contained in the 3D content dataset; convert each component category into a multidimensional one-hot encoded vector.

[0102] As another illustrative implementation of the above embodiments, the above-described image and text data corresponding module 803 can be further used for: A multi-view pixel-to-part name correspondence fusion network structure is pre-constructed. Multi-dimensional data information determined based on the 3D content dataset and the part names corresponding to each target pixel are input into the multi-view pixel-to-part name correspondence fusion network structure to obtain all 3D points corresponding to each part name. The multi-view pixel-to-part name correspondence fusion network structure includes a first input terminal, a second input terminal, a third input terminal, a part category recognition network, a data fusion network, and an output terminal. The first input terminal is connected to the part category recognition network, and both the part category recognition network and the second input terminal are connected to the data fusion network. The data fusion network is connected to the output terminal. The first input terminal is used to input multi-dimensional data information, and the second input terminal is used to input the part names corresponding to each target pixel. During training, the multi-view pixel-to-part name correspondence fusion network structure also includes a third input terminal, which is connected to the data fusion network and used to input the part category corresponding to each pixel in each 2D content image.

[0103] For example, in some other embodiments of this example, the above-described image and text data acquisition module 802 can also be used for: Obtain the name of at least one 3D content from the 3D content dataset; based on the name of each 3D content, use the text question-answering model to obtain the component name information in text form.

[0104] As an illustrative implementation of the above embodiments, the image and text data acquisition module 802 can also be used for: Based on the user-input questions and auxiliary questions for the 3D content components, a set of questions and a set of auxiliary questions are generated. The set of auxiliary questions is then input into a text-based question-and-answer model to obtain the answers to the auxiliary questions. Finally, the set of questions and the answers to the auxiliary questions are input into the text-based question-and-answer model to obtain the component name information.

[0105] For example, in some further embodiments of this example, the above-described image and text data acquisition module 802 can also be used for: The pose information of the virtual camera is calculated based on its position and orientation. The virtual camera is fixedly pointed at each 3D content in the 3D content dataset. The virtual camera is deployed in multiple locations to cover and collect data from multiple perspectives of each 3D content. Based on the pose information of each virtual camera, each 3D content is rendered to obtain 2D content images of the same 3D content from different perspectives.

[0106] As an exemplary implementation of the above embodiments, the image and text data acquisition module 802 may further be used for: All 3D component points of each 3D content in the 3D content dataset are aligned with the target coordinate system, and the dimensions of each 3D content are normalized to the preset standard dimensions.

[0107] As another exemplary implementation of the above embodiments, the image and text data acquisition module 802 may further be used for: The virtual camera is set in a spherical coordinate system; the positions of multiple virtual cameras are obtained by changing the radius and / or polar angle and / or azimuth angle; based on the orientation of the virtual camera, multiple pose information is calculated according to the different positions of the virtual camera.

[0108] The functions of each module of the dataset generation device based on the pre-trained language model of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0109] As can be seen from the above, this embodiment can generate a large-scale, high-quality dataset corresponding to text and 3D content components.

[0110] Furthermore, the 3D content generation device based on a pre-trained language model provided by this invention is described below. This 3D content generation device based on a pre-trained language model can be referred to in conjunction with the dataset generation method based on a pre-trained language model described above. Please see [link to documentation]. Figure 9 , Figure 9 This is a structural diagram of a 3D content generation device based on a pre-trained language model provided by the present invention, in one specific embodiment. The device may include: The dataset acquisition module 901 is used to obtain a dataset corresponding to text and three-dimensional content components based on the steps of the dataset generation method based on a pre-trained language model as described in any previous embodiment. Model training module 902 is used to train a 3D content generation network model based on a dataset of text and 3D content components; The content generation module 903 is used to automatically generate 3D content based on user input data using a trained 3D content generation network model.

[0111] The dataset generation device based on the pre-trained language model mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 10 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. For example... Figure 10 As shown, the electronic device includes a memory 100 for storing a computer program; and a processor 101 for executing the computer program to implement the steps of the dataset generation method based on a pre-trained language model as described in any of the above embodiments.

[0112] The processor 101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 101 may also be a controller, microcontroller, microprocessor, or other data processing chip. The processor 101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0113] The memory 100 may include one or more computer-readable storage media, which may be non-transitory. The memory 100 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the memory 100 may be an internal storage unit of an electronic device, such as a server hard drive. In other embodiments, the memory 100 may be an external storage device of an electronic device, such as a plug-in hard drive on a server, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 100 may include both internal and external storage units of the electronic device. The memory 100 can be used not only to store application software and various types of data installed on the electronic device, such as code in the process of executing a dataset generation method based on a pre-trained language model, but also to temporarily store data that has been output or will be output. In this embodiment, the memory 100 is used to store at least the following computer program 1001, which, after being loaded and executed by the processor 101, is capable of implementing the relevant steps of the dataset generation method based on a pre-trained language model disclosed in any of the foregoing embodiments. Additionally, the resources stored in the memory 100 may also include an operating system 1002 and data 1003, and the storage method may be temporary or permanent storage. The operating system 1002 may include Windows, Unix, Linux, etc. The data 1003 may include, but is not limited to, data corresponding to the dataset generation results based on the pre-trained language model.

[0114] In some embodiments, the aforementioned electronic device may further include a display screen 102, an input / output interface 103, a communication interface 104 (or network interface), a power supply 105, and a communication bus 106. The display screen 102 and input / output interface 103, such as a keyboard, are user interfaces; optional user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a display screen or display unit, used to display information processed in the electronic device and to display a visual user interface. The communication interface 104 may optionally include a wired interface and / or a wireless interface, such as a Wi-Fi interface, a Bluetooth interface, etc., typically used to establish communication connections between the electronic device and other electronic devices. The communication bus 106 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0115] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, such as sensors 107 that perform various functions.

[0116] The functions of each functional module of the electronic device described in this invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, and will not be repeated here.

[0117] As can be seen from the above, this embodiment can generate a large-scale, high-quality dataset corresponding to text and 3D content components.

[0118] It is understood that if the dataset generation method based on the pre-trained language model in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, and other media capable of storing program code.

[0119] Based on this, the present invention also provides a readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the dataset generation method based on a pre-trained language model described in any of the above embodiments are as follows.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the hardware disclosed in the embodiments, including devices and electronic equipment, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0121] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0122] The foregoing has provided a detailed description of the method, apparatus, electronic device, and readable storage medium for generating 3D content and its dataset based on a pre-trained language model, as provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A method for generating datasets based on pre-trained language models, characterized in that, include: By pre-using a 3D content question-answering sample set, the pre-trained language model is fine-tuned based on the question-answering task to obtain a text question-answering model; Using the text question-answering model, obtain the component name information described in text form from the three-dimensional content dataset; Obtain two-dimensional content images from different perspectives corresponding to at least one three-dimensional content in the three-dimensional content dataset; Based on the component name information and each two-dimensional content image, and according to the correspondence between pixels, component names, and each three-dimensional point, all three-dimensional points corresponding to each component name are determined, including: Based on the component name information and each two-dimensional content image, determine the component name corresponding to at least one target pixel in at least one two-dimensional content image; Based on the aforementioned 3D content dataset, the component names corresponding to each target pixel are fused. Based on the fusion result, all 3D points corresponding to each component name are determined, including: Obtain the component category corresponding to at least one target pixel in at least one two-dimensional content image; based on the probability information of each three-dimensional point belonging to each component category, use the component name corresponding to each target pixel as the prediction result and the component category corresponding to each target pixel as the ground value to train a pre-constructed data fusion network; Based on the three-dimensional content dataset, the probability information of each three-dimensional point belonging to each type of component is obtained; according to the component name corresponding to each three-dimensional point and the probability information, all three-dimensional points corresponding to each component name are determined; the component name corresponding to each three-dimensional point is determined by the component name corresponding to each target pixel based on the correspondence between pixels and three-dimensional points, and the component category and component name are in one-to-one correspondence. Based on the component name and corresponding 3D points of at least one 3D content in the 3D content dataset, a dataset corresponding to text and 3D content components is obtained.

2. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The step of determining the component name corresponding to at least one target pixel in at least one two-dimensional content image based on the component name information and each two-dimensional content image includes: The component names in the component name information are combined with at least one two-dimensional content image to obtain multiple image and text combination results; Based on the component names in each image-text combination result, the component names corresponding to each target pixel in each two-dimensional content image are obtained by performing image segmentation processing on the two-dimensional content images in the corresponding image-text combination results.

3. The dataset generation method based on a pre-trained language model according to claim 2, characterized in that, The step of obtaining the component name corresponding to each target pixel in each two-dimensional content image by performing image segmentation processing on the two-dimensional content image in each image-text combination result based on the component name in each image-text combination result includes: Pre-train a model to segment everything; Each image and text combination result is input into the segmentation model in turn to obtain all pixels in the current two-dimensional content image that belong to the corresponding component name; Based on all pixels in each two-dimensional content image that belong to the corresponding component name, determine the component name corresponding to each target pixel in each two-dimensional content image.

4. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The step of determining all three-dimensional points corresponding to each component name based on the component name corresponding to each three-dimensional point and the probability information includes: The probability information and the component names corresponding to each 3D point are used as input data and fed into a trained data fusion network to obtain all 3D points corresponding to each component name.

5. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, Training the data fusion network includes: The data fusion network is trained by invoking the fusion loss function formula, which is: ； In the formula, L(Ppred, Pseg) represents the fusion loss function relationship, Ppred is the prediction result, Pseg is the true value, c is the total number of component categories, and i represents the i-th component category.

6. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The step of obtaining the component category corresponding to at least one target pixel in at least one two-dimensional content image includes: Each two-dimensional content image is sequentially input into the trained segmentation model to obtain the predicted pixel region for each component category in the current two-dimensional content image.

7. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The acquisition of probability information for each 3D point belonging to each component category includes: A component category recognition network is pre-constructed; the component category recognition network includes a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron; Multidimensional data information is determined based on the component categories contained in the three-dimensional content dataset, the total number of three-dimensional points in each three-dimensional content, and the maximum number of component categories to which each three-dimensional point belongs. The multidimensional data information is input into the component category recognition network. The multilayer perceptrons of the component category recognition network perform dimensionality reduction processing on the multidimensional data information to obtain the probability information of each three-dimensional point belonging to each type of component.

8. The dataset generation method based on a pre-trained language model according to claim 7, characterized in that, The determination of multidimensional data information based on the component categories included in the three-dimensional content dataset, the total number of three-dimensional points in each three-dimensional content, and the maximum number of component categories to which each three-dimensional point belongs, includes: Obtain the component categories contained in the 3D content dataset; Each component category is converted into a multidimensional one-hot encoded vector.

9. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The process of fusing the component names corresponding to each target pixel based on the 3D content dataset, and determining all 3D points corresponding to each component name based on the fusion result, includes: Pre-construct a network structure that fuses the correspondence between multi-view pixels and component names; The multidimensional data information determined based on the three-dimensional content dataset and the component name corresponding to each target pixel are input into the multi-view pixel and component name correspondence fusion network structure to obtain all three-dimensional points corresponding to each component name; The multi-view pixel-to-part name correspondence fusion network structure includes a first input terminal, a second input terminal, a third input terminal, a part category recognition network, a data fusion network, and an output terminal. The first input terminal is connected to the part category recognition network, and both the part category recognition network and the second input terminal are connected to the data fusion network. The data fusion network is connected to the output terminal. The first input terminal is used to input multi-dimensional data information, and the second input terminal is used to input the part name corresponding to each target pixel. During training, the multi-view pixel-to-part name correspondence fusion network structure also includes a third input terminal, which is connected to the data fusion network and is used to input the ground truth value of the part category corresponding to each pixel in each two-dimensional content image. The multi-dimensional data information is determined based on the part categories contained in the three-dimensional content dataset, the total number of three-dimensional points in each three-dimensional content, and the maximum number of part categories to which each three-dimensional point belongs. The part categories and part names are in a one-to-one correspondence.

10. The dataset generation method based on a pre-trained language model according to claim 1, characterized in that, The step of using the text question-answering model to obtain the component name information in text form described in the three-dimensional content dataset includes: Retrieve the name of at least one 3D content item from the 3D content dataset; Based on the names of each 3D content, the component name information described in text form is obtained using the text question-and-answer model.

11. The dataset generation method based on a pre-trained language model according to claim 10, characterized in that, The part name information, described in text form, is obtained using the text question-and-answer model based on the names of each 3D content, including: Based on the user's input of questions and auxiliary questions related to the 3D content components, generate a set of questions and a set of auxiliary questions; The set of auxiliary questions is input into the text question-answering model, and the text question-answering model is used to obtain the answers to the auxiliary questions corresponding to the set of auxiliary questions. The set of questions and the answers to the auxiliary questions are input into the text question-answering model to obtain the component name information.

12. The dataset generation method based on a pre-trained language model according to any one of claims 1 to 11, characterized in that, The step of obtaining two-dimensional content images from different perspectives corresponding to at least one three-dimensional content in the three-dimensional content dataset includes: The pose information of the virtual camera is calculated based on its position and orientation; the virtual camera is fixedly pointed at each 3D content in the 3D content dataset, and the virtual camera is deployed in multiple locations to cover the acquisition of data from multiple perspectives of each 3D content; Based on the pose information of each virtual camera, each 3D content is rendered to obtain 2D content images of the same 3D content from different viewpoints.

13. The dataset generation method based on a pre-trained language model according to claim 12, characterized in that, Before rendering each 3D content based on each pose information of the virtual camera, the process further includes: All 3D component points of each 3D content in the 3D content dataset are aligned with the target coordinate system, and the dimensions of each 3D content are normalized to a preset standard size.

14. The dataset generation method based on a pre-trained language model according to claim 12, characterized in that, The calculation of the virtual camera's pose information based on its position and orientation includes: Set the virtual camera in a spherical coordinate system; The positions of multiple virtual cameras can be obtained by changing the radius and / or polar angle and / or azimuth angle; Based on the orientation of the virtual camera, multiple pose information is calculated according to different positions of the virtual camera.

15. A method for generating 3D content based on a pre-trained language model, characterized in that, include: Based on the steps of the dataset generation method based on a pre-trained language model as described in any one of claims 1 to 14, a dataset corresponding to text and three-dimensional content components is obtained; Based on the dataset corresponding to the text and 3D content components, a 3D content generation network model is trained to automatically generate 3D content based on user input data.

16. A dataset generation device based on a pre-trained language model, characterized in that, include: The question-answering model building module is used to fine-tune the pre-trained language model based on the execution of the question-answering task using a three-dimensional content question-answering sample set to obtain a text question-answering model. The image and text data acquisition module is used to acquire the component name information described in text form in the three-dimensional content dataset using the text question-and-answer model; and to acquire two-dimensional content images of at least one three-dimensional content in the three-dimensional content dataset from different perspectives. The image and text data correspondence module is used to determine all three-dimensional points corresponding to each component name based on the component name information and each two-dimensional content image, and based on the correspondence between pixels, component names, and each three-dimensional point. The dataset generation module is used to obtain a dataset corresponding to text and three-dimensional content components based on the component names and three-dimensional points of each three-dimensional content in the three-dimensional content dataset. The image-text data correspondence module is further configured to: determine the component name corresponding to at least one target pixel in at least one two-dimensional content image based on the component name information and each two-dimensional content image; perform fusion processing on the component names corresponding to each target pixel based on the three-dimensional content dataset, and determine all three-dimensional points corresponding to each component name based on the fusion result, including: obtaining the component category corresponding to at least one target pixel in at least one two-dimensional content image; training a pre-constructed data fusion network based on the probability information of each three-dimensional point belonging to each component category, using the component name corresponding to each target pixel as the prediction result and the component category corresponding to each target pixel as the ground truth; obtaining the probability information of each three-dimensional point belonging to each component category based on the three-dimensional content dataset; determining all three-dimensional points corresponding to each component name based on the component name corresponding to each three-dimensional point and the probability information; the component name corresponding to each three-dimensional point is determined by the component name corresponding to each target pixel based on the correspondence between pixels and three-dimensional points, and the component category and component name are in one-to-one correspondence.

17. A three-dimensional content generation device based on a pre-trained language model, characterized in that, include: The dataset acquisition module is used to obtain a dataset corresponding to text and three-dimensional content components based on the steps of the dataset generation method based on a pre-trained language model as described in any one of claims 1 to 14. The model training module is used to train a 3D content generation network model based on the dataset corresponding to the text and 3D content components. The content generation module is used to automatically generate 3D content based on user input data using a trained 3D content generation network model.

18. An electronic device, characterized in that, The system includes a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the dataset generation method based on a pre-trained language model as described in any one of claims 1 to 14 and / or the three-dimensional content generation method based on a pre-trained language model as described in claim 15.

19. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dataset generation method based on a pre-trained language model as described in any one of claims 1 to 14 and / or the three-dimensional content generation method based on a pre-trained language model as described in claim 15.

Images