AI large model-based mixed reality content generation method and apparatus, and terminal device

Abstract

The embodiment of the invention provides a mixed reality content generation method and device based on an AI large model and terminal equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps that initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located are collected; performing voice recognition on the initial voice information to obtain target text information; performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information; performing image segmentation on the target image information according to the initial image information to obtain initial object information, which does not appear in the target image information, of the initial image information; performing virtual construction on the initial object information to obtain second three-dimensional information corresponding to the initial object information; and performing mixed reality content generation according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and terminal device for generating mixed reality content based on an AI big model. Background Art

[0002] With the rapid progress of virtual reality technology, people have been able to reconstruct and simulate objects and scenes in the real world through this technology, providing unprecedented possibilities for various applications such as education, entertainment, medical care, engineering, etc. However, despite the remarkable achievements of virtual reality in creating virtual environments and experiences, existing virtual reality generated content often faces a key problem: they are usually separated from physical objects and environments in the real world, and it is difficult to directly utilize objects that exist in reality. This limitation leads to a sense of disconnection in user experience and restricts the comprehensive application and development of virtual reality technology. In many virtual reality applications, although users can see, touch, and even interact with virtual objects, these objects lack effective association with the real environment around the users. For example, in augmented reality applications, virtual objects may appear to be superimposed on the real environment, but their interaction with actual physical objects appears stiff and unnatural. This sense of disconnection not only destroys the coherence of the environment, but also affects the user's perception of the integration of the virtual and real worlds. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a mixed reality content generation method, apparatus and terminal device based on an AI large model, aiming to solve the problem in the related technology that although virtual objects are superimposed on the real environment, their interaction with actual physical objects appears stiff and unnatural. This sense of disconnection not only destroys the coherence of the environment, but also affects the user's perception of the fusion of the virtual and real worlds.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating mixed reality content based on an AI big model, comprising:

[0005] Collecting initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located;

[0006] Performing speech recognition on the initial speech information to obtain target text information;

[0007] Performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information;

[0008] Performing image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information;

[0009] Performing virtual construction on the initial object information to obtain second three-dimensional information corresponding to the initial object information;

[0010] Mixed reality content is generated according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

[0011] In a second aspect, an embodiment of the present invention provides a mixed reality content generation device based on an AI big model, comprising:

[0012] A data acquisition module, used for acquiring initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located;

[0013] A speech recognition module, used for performing speech recognition on the initial speech information to obtain target text information;

[0014] An image generation module, used for performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information;

[0015] An image segmentation module, configured to perform image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information;

[0016] A virtual construction module, used to virtually construct the initial object information to obtain second three-dimensional information corresponding to the initial object information;

[0017] A result generation module is used to generate mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

[0018] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the methods for generating mixed reality content based on an AI large model provided in the specification of the present invention are implemented.

[0019] The embodiment of the present invention provides a method, apparatus and terminal device for generating mixed reality content based on an AI large model. The method comprises: collecting initial voice information of a target user and initial image information and first three-dimensional information of the environment in which the target user is located, and then recognizing the initial voice information to obtain target text information, thereby obtaining virtual information that the target user expects to generate, and then performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information, and then performing image segmentation on the target image information according to the initial image information to obtain initial object information that does not appear in the target image information, that is, to obtain objects that do not exist in the target environment in which the target user is located, and then virtually constructing the initial object information to obtain the second three-dimensional information corresponding to the initial object information. dimensional information, thereby generating mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information, thereby being able to better understand the target environment of the target user, so that the generated mixed reality content is not only visually realistic, but also highly consistent with the real environment in space, thereby providing the target user with a more immersive and realistic experience, thereby reducing the sense of separation and disconnection with the real world, thereby greatly improving the quality and effect of the user's interaction with the digital world, thereby further solving the problem in related technologies that although virtual objects are superimposed on the real environment, their interaction with actual physical objects appears stiff and unnatural. This sense of disconnection not only destroys the coherence of the environment, but also affects the user's perception of the fusion of the virtual and real worlds. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a method for generating mixed reality content based on an AI big model provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the module structure of a mixed reality content generation device based on an AI big model provided in an embodiment of the present invention;

[0023] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] The embodiment of the present invention provides a method, apparatus and terminal device for generating mixed reality content based on a large AI model. The method for generating mixed reality content based on a large AI model can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The terminal device can be a server or a server cluster.

[0028] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A flowchart of a method for generating mixed reality content based on an AI big model is provided in an embodiment of the present invention.

[0030] like Figure 1 As shown, the mixed reality content generation method based on the AI ​​big model includes steps S101 to S106.

[0031] Step S101: collecting initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located.

[0032] For example, a high-quality microphone or microphone array is used to ensure that the initial voice information of the target user can be clearly captured, and a high-resolution camera is used to ensure that the initial image information corresponding to the target environment where the target user is located can be captured. The camera should be placed in a position that can fully cover the target environment where the target user is located.

[0033] Exemplarily, a depth sensor (such as a laser radar, a structured light sensor, or a ToF sensor) or a stereo camera is used to ensure that the first three-dimensional information of the target environment can be captured. In addition, the acquisition time of the initial voice information, the initial image information, and the first three-dimensional information is synchronized to avoid time sequence confusion between the data. Synchronous triggers or timestamp technology can be used to achieve time alignment of multi-sensor data.

[0034] Step S102: Perform speech recognition on the initial speech information to obtain target text information.

[0035] Exemplarily, a pre-trained speech recognition model (such as a deep learning-based model, such as RNN, LSTM, Transformer, etc.) or a traditional statistical model (such as a hidden Markov model, HMM) is used to perform speech recognition on the initial speech information to obtain the target text information.

[0036] In some embodiments, performing speech recognition on the initial speech information to obtain target text information includes: performing pre-emphasis, frame windowing and denoising preprocessing operations on the initial speech information to obtain the target speech information; using a feature extraction model to perform speech feature extraction on the obtained target speech information to obtain initial Mel-frequency cepstrum parameters, initial short-time average energy parameters and initial spectrum mean parameters corresponding to the target speech information; standardizing the initial Mel-frequency cepstrum parameters, the initial short-time average energy parameters and the initial spectrum mean parameters to obtain target Mel-frequency cepstrum parameters, target short-time average energy parameters and target spectrum mean parameters; using a speech recognition model in combination with the target Mel-frequency cepstrum parameters, the target short-time average energy parameters and the target spectrum mean parameters to perform speech recognition to obtain the target text information.

[0037] Exemplarily, the continuous initial speech signal is segmented into short time frames (usually 20-30 milliseconds), and a window function (such as a Hamming window) is applied to each frame to reduce spectral leakage and edge effects, so that the initial speech signal of each frame is fast Fourier transformed to obtain a spectral representation, and then the initial Mel-frequency cepstral coefficients are extracted from the spectral representation to capture the spectral characteristics of the speech signal.

[0038] Exemplarily, an initial short-time average energy parameter of each frame of the initial speech signal is calculated, which can reflect the energy change of the speech signal, and an initial spectrum mean parameter of each frame of the initial speech signal is calculated, which can capture the average level of the spectrum.

[0039] Exemplarily, the mean and standard deviation of the initial Mel frequency cepstrum parameter, the initial short-time average energy parameter and the initial spectrum mean parameter are calculated respectively, and then each parameter is normalized by the mean, that is, each parameter is subtracted from its mean, and then divided by the standard deviation to obtain the target Mel frequency cepstrum parameter, the target short-time average energy parameter and the target spectrum mean parameter. The standardization process makes all feature parameters on the same scale, avoids the problem of different dimensions between features, and improves the training effect and recognition accuracy of the model.

[0040] Exemplarily, a speech recognition model, such as a deep learning-based model (such as RNN, LSTM, Transformer, etc.) or a traditional statistical model (such as a hidden Markov model, HMM), is used to input the standardized target Mel frequency cepstrum parameters, target short-time average energy parameters, and target spectrum mean parameters into the speech recognition model, and the model is used to decode the input feature parameters to generate the corresponding target text information. The decoding process can be combined with a language model to improve the accuracy and fluency of text generation.

[0041] Specifically, the combination of multiple feature parameters and efficient models can significantly improve the accuracy of speech recognition, thereby achieving efficient and accurate conversion from target speech information to target text information, providing a reliable foundation for subsequent image generation and mixed reality content generation.

[0042] In some embodiments, obtaining the feature extraction model includes: obtaining a training speech signal and target feature information corresponding to the training speech signal, wherein the training speech signal is the audio pronunciation corresponding to each character; determining initial model parameters corresponding to the speech recognition model, and performing operations based on the initial speech parameters and the training speech signal through the input layer, hidden layer, and output layer corresponding to the initial model parameters to obtain predicted feature information; performing difference calculation based on the target feature information and the predicted feature information to obtain difference information, and comparing the difference information with a preset threshold to obtain a comparison result; when the comparison result is that the difference information is greater than or equal to the preset threshold, then updating the initial model parameters and continuing to perform model training on the speech recognition model using the training speech signal and the target feature information until the target model parameters corresponding to the speech recognition model are obtained; when the comparison result is that the difference information is less than the preset threshold, then obtaining the target model parameters corresponding to the speech recognition model, and then determining the speech recognition model based on the target model parameters.

[0043] Exemplarily, training data for training a speech recognition model is prepared, including a training speech signal and its corresponding target feature information. The training speech signal is an audio pronunciation corresponding to each word. For example, the target feature information includes Mel frequency cepstral coefficients (MFCC), short-time average energy, spectrum mean, etc. corresponding to the training speech signal.

[0044] Exemplarily, the initial model parameters corresponding to the speech recognition model are randomly generated, and then the training speech signal is input into the input layer of the model. After being processed by the hidden layer, the predicted feature information is finally obtained at the output layer, and then the difference between the target feature information and the predicted feature information is calculated, such as mean square error (MSE), cross entropy, etc. Then, the calculated error value information is compared with the preset threshold to determine whether the error meets the requirements.

[0045] Exemplarily, when the comparison result is that the difference information is greater than or equal to the preset threshold, the initial model parameters are updated and the training speech signal and the target feature information are continuously used to perform model training on the speech recognition model until the target model parameters corresponding to the speech recognition model are obtained. For example, according to the error calculation result, the initial model parameters are adjusted by the back propagation algorithm to reduce the error information, and the forward propagation and back propagation processes are repeated until the error information is lower than the preset threshold or the predetermined number of training times is reached.

[0046] Exemplarily, when the comparison result is that the difference information is less than a preset threshold, the target model parameters corresponding to the speech recognition model are obtained, and then the speech recognition model is determined according to the target model parameters. That is, the trained target model parameters are saved for subsequent use.

[0047] Specifically, a high-performance speech recognition model is trained, which can accurately convert speech signals into text information, providing accurate text input for subsequent image generation and mixed reality content generation.

[0048] Step S103: generating an image according to the target text information and the initial image information to obtain target image information corresponding to the target text information.

[0049] Exemplarily, natural language processing technology is applied to target text information analysis, including part-of-speech tagging, named entity recognition, and syntactic analysis, to extract key entities in the target text information and the association relationship between key entities. Subsequently, models such as convolutional neural networks (CNNs) are used to extract features such as color, texture, shape, and spatial layout from the initial image information, and then identify the position information corresponding to the key entities in the initial image information. On this basis, the key entities that do not appear in the initial image information are further explored, and the relevant image information of the key entities that do not appear in the initial image information is generated through the target generation model. Finally, based on the identified position information and entity relationships, the newly generated relevant image information is fused with the initial image to obtain target image information that is highly consistent with the target text information. The target image information not only integrates the key entities in the target environment but also includes the new key entities that appear in the target text information, thereby providing support for the subsequent reduction of the sense of separation and disconnection between virtual objects and the real world.

[0050] In some embodiments, the image generation based on the target text information and the initial image information to obtain the target image information corresponding to the target text information includes: performing text-image matching on the target text information and the initial image information to obtain associated image information corresponding to the target text information when it matches the initial image information; determining basic image information corresponding to the target text information based on the associated image information; performing feature extraction on the target text information using the text feature extraction layer of the text-generated image model to obtain text feature information; performing feature extraction on the basic image information using the image feature extraction layer of the text-generated image model to obtain initial image feature information; and performing feature extraction on the basic image information using the text-generated image model. The feature fusion layer of the image model performs sentence feature fusion on the text feature information to obtain sentence feature information; the feature addition layer of the text-to-image model is used to add random noise to the sentence feature information to obtain splicing feature information; the text image affine layer of the text-to-image model is used to perform image processing on the splicing feature information and the initial image feature information to obtain target image feature information; the convolutional network layer of the text-to-image model is used to obtain intermediate image information corresponding to the target text information based on the target image feature information; the image enhancement layer of the text-to-image model is used to perform image refinement on the intermediate image information to obtain the target image information corresponding to the target text information.

[0051] Exemplarily, cross-modal matching technology is used to analyze key elements and features in the target text information and the initial image information, determine the corresponding relationship between them, and thus extract the associated image information that matches the target text information and the initial image information. For example, if there is a key entity in the target text information that is a house, then when the target text information and the initial image information are successfully matched, the associated image information corresponding to the house can be obtained from the initial image information.

[0052] Exemplarily, the basic image information corresponding to the target text information is obtained by splicing according to the position of the associated image information in the initial image information as the basis for image generation. That is, the basic image information only contains the content of the key entity corresponding to the target text information that exists in the initial image information.

[0053] Exemplarily, the text feature extraction layer of the text-to-image model is used to perform word embedding, sentence encoding and other processing on the target text information to extract text feature information, and then the image feature extraction layer of the text-to-image model is used to perform convolution, pooling and other operations on the basic image information to extract the corresponding image feature information under color, texture, shape and spatial layout. Then, the feature fusion layer of the text-to-image model is used to fuse the extracted text feature information to form complete sentence feature information.

[0054] For example, the feature adding layer of the text-to-image model is used to add random noise to the sentence feature information to obtain the splicing feature information. Thus, by introducing noise, the generated image is richer in details and features, and the visual quality is improved.

[0055] Exemplarily, the text image affine layer of the text-to-image model is used to perform affine transformation and fusion processing on the spliced ​​feature information and the initial image feature information to generate target image feature information, so that the fused image feature information is more semantically and visually consistent, thereby improving the accuracy and quality of the generated image.

[0056] Exemplarily, the convolutional network layer of the text-to-image model is used to generate intermediate image information through convolution operations based on the target image feature information. Through the convolution operations, the target image feature information is further enhanced to generate intermediate image information that is more consistent with the target text information.

[0057] Exemplarily, the image enhancement layer of the text-generated image model is used to perform image refinement processing on the intermediate image information to improve image quality and details, and then through image refinement processing, the details of the generated image are optimized to better conform to the description of the target text information, thereby obtaining the target image information corresponding to the target text information.

[0058] In some embodiments, the text image affine layer of the text generation image model performs image processing on the splicing feature information and the initial image feature information to obtain the target image feature information, including: using the relationship recognition network of the text image affine layer to determine the corresponding target relationship between the target objects involved in the target text information based on the splicing feature information; using the position determination network of the text image affine layer to determine the target position information corresponding to the target object that does not exist in the initial image information according to the initial position information of the target object in the initial image information and the target relationship; using the position determination image space layout network of the text image affine layer to adjust the generation position of the initial image feature information using the target position information to obtain the target image feature information.

[0059] Exemplarily, the target relationship between target objects involved in the target text information is determined based on the splicing feature information using the relationship recognition network in the text image affine layer. That is, through the relationship recognition network, the semantic relationship between target objects involved in the target text information can be accurately captured, providing important guidance information for image generation.

[0060] Exemplarily, the position determination network in the text image affine layer is used to determine the target position information corresponding to the target object that does not exist in the initial image information based on the initial position information of the target object in the initial image information and the identified target relationship. Then, through the position determination network, it is ensured that the position of the newly introduced target object in the target image information is reasonable to conform to the relationship described in the target text information, thereby providing support for subsequently maintaining the spatial consistency between the target objects in the target image information, thereby improving the naturalness of the generated image.

[0061] Exemplarily, the position in the text image affine layer is used to determine the image space layout network, and the target position information is used to adjust the generation position of the initial image feature information to obtain the target image feature information, so that the generated image feature information is more reasonable and coordinated in space by adjusting the spatial layout of the image feature information, and ensures that the newly added target object is spatially coordinated with the object in the initial image information, thereby improving the coherence of the overall visual effect.

[0062] Specifically, through the above steps, the target text information and the initial image information can be effectively integrated to generate target image feature information that is consistent with the text description and the initial image style, thereby further generating high-quality target image information.

[0063] In some embodiments, the image enhancement layer of the text generation image model is used to perform image refinement on the intermediate image information to obtain the target image information corresponding to the target text information, including: using the image feature recognition network of the image enhancement layer to extract image features from the intermediate image information to obtain initial image features; using the text-image association network of the image enhancement layer to calculate the information association value between the splicing feature information and the initial image features; using the weight calculation layer of the image enhancement layer to determine the target weight corresponding to the initial image feature according to the information association value; using the feature update network of the image enhancement layer to update the initial image feature according to the target weight to obtain updated target image features; using the detail correction network of the image enhancement layer to supplement the missing features of the target image features to obtain final image features; using the image generation network of the image enhancement layer to generate the target image information corresponding to the target text information according to the final image features; wherein the information association value is obtained according to the following formula:

[0064]

[0065] in, represents the information association value between the i-th splicing feature information and the initial image feature, represents the i-th splicing feature information, represents the matrix required to transform the i-th splicing feature information, represents the tth initial image feature, m represents the number of features corresponding to the initial image feature, Representation conversion The required matrix, represents the activation function, and Represents a 1*1 convolution, which is used to perform dimensional conversion so that the i-th concatenated feature information and have the same feature space.

[0066] Exemplarily, the image feature recognition network in the image enhancement layer is used to extract image features from the intermediate image information to obtain the initial image features, and then the text-image association network in the image enhancement layer is used to calculate the information association value between the spliced ​​feature information and the initial image features. Then, by calculating the information association value, the correlation between the text and image features is quantified, providing a basis for weight calculation, which helps to determine which image features need to be strengthened or weakened to better match the text description. The information association value is obtained according to the following formula:

[0067]

[0068] in, represents the information association value between the i-th splicing feature information and the initial image feature, represents the i-th splicing feature information, Represents the matrix required to transform the i-th splicing feature information, represents the tth initial image feature, m represents the number of features corresponding to the initial image feature, Representation conversion The required matrix, represents the activation function, and Represents a 1*1 convolution, which is used to convert the dimension so that the i-th concatenated feature information and have the same feature space.

[0069] For example, by calculating the specific information association value, we can clearly understand the correlation between the splicing feature information and the initial image features, which provides a specific reference for the subsequent weight calculation, making the feature update more accurate. The nonlinear characteristics of the activation function can capture the complex nonlinear relationship between the splicing feature information and the initial image features, improve the accuracy of the information association value, and then enhance the expression ability of the model through nonlinear transformation, so that it can better process complex data.

[0070] Exemplarily, the weight calculation layer in the image enhancement layer is used to normalize the data according to the information association value to calculate the target weight corresponding to the initial image feature, and then the feature update network in the image enhancement layer is used to update the initial image feature based on the target weight to obtain the updated target image feature.

[0071] Exemplarily, the detail correction network in the image enhancement layer is used to supplement the missing features of the target image features to obtain the final image features. The detail correction network supplements the missing features to make the generated image more complete and rich, thereby making the image more realistic and delicate. Finally, the image generation network in the image enhancement layer is used to generate the target image information based on the final image features, and the generated image accurately matches the target text information to achieve high-quality conversion from text to image. In this way, high-quality generation from intermediate image information to final target image information can be achieved, ensuring that the generated image not only conforms to the text description, but also has a high degree of visual realism and delicacy.

[0072] Step S104: performing image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information.

[0073] Exemplarily, target segmentation is performed on the initial image information to obtain a plurality of first segmented objects and type recognition is performed on each first segmented object to obtain a first target type. Target segmentation is performed on the target image information to obtain a plurality of second segmented objects and type recognition is performed on each second segmented object to obtain a second target type.

[0074] Exemplarily, the first target type and the second target type are subjected to intersection processing to determine the relationship between the two. Specifically, by performing an intersection operation on the first target type and the second target type, it is possible to identify which second segmented objects in the second target type already exist in the first target type, and which second segmented objects do not exist in the first target type. When some second segmented objects in the second target type do not exist in the first target type, the initial object information of these second segmented objects, that is, the initial image information, will be extracted as a key input for the subsequent generation stage. Since these initial object information do not have corresponding first three-dimensional information in the first target type, when generating the mixed reality generation result, it is necessary to generate corresponding second three-dimensional information specifically for these initial object information. In this way, by generating these missing second three-dimensional information, it can be ensured that the final generated mixed reality result can be perfectly integrated with the real environment, providing a more realistic and coherent user experience. This process not only fills the gap in data, but also provides a solid foundation for generating mixed reality content that is highly consistent with the real environment.

[0075] Step S105: Virtually construct the initial object information to obtain second three-dimensional information corresponding to the initial object information.

[0076] Exemplarily, a feature extraction algorithm or model is used to extract key features, such as shape, texture, material, etc., from the initial object information to provide basic data for virtual construction, and then, based on the characteristics of the initial object information, a suitable 3D modeling technology is selected, such as a deep learning-based generative model, a traditional 3D reconstruction algorithm, etc., and then the model parameters are adjusted as needed, such as resolution, detail level, lighting conditions, etc., to obtain an ideal 3D effect, and then the extracted features are input into the selected model, the generation process is executed, and the second 3D information corresponding to the initial object information is output.

[0077] In some embodiments, the virtual construction of the initial object information to obtain the second three-dimensional information corresponding to the initial object information includes: filtering the initial object information to obtain a target filtering result, and performing color space conversion on the initial object information to obtain target color information corresponding to the target color space; determining a target segmentation block, and performing image segmentation on the initial object information according to the target segmentation block to obtain a target segmentation result corresponding to the target segmentation block; determining a target feature vector corresponding to the target segmentation result according to the target filtering result and the target color information; performing feature brightness enhancement processing according to the target feature vector using the brightness enhancement layer of the image reconstruction model to obtain a first enhanced feature vector; performing detail feature enhancement processing according to the first enhanced feature vector using the detail reconstruction layer of the image reconstruction model to obtain a second enhanced feature vector; performing image generation on the second enhanced feature vector using the target generation layer of the image reconstruction model to obtain target object information; and performing virtual construction according to the target object information to obtain the second three-dimensional information.

[0078] Exemplarily, a filtering algorithm (such as Gaussian filtering, median filtering, etc.) is used to process the initial object information to obtain a target filtering result, thereby removing unnecessary details and reducing interference information in subsequent processing.

[0079] For example, using a color space conversion algorithm (such as RGB to HSV, Lab, etc.) to convert the target color information, and then converting it to a more suitable color space, can better extract and analyze the color information, thereby providing richer color features, which is helpful for subsequent feature extraction and image segmentation.

[0080] Exemplarily, the initial object information is segmented according to the target segmentation block to obtain the target segmentation result under the target segmentation block. The size of the target segmentation block can be 2*2 or 4*4.

[0081] Exemplarily, a feature extraction algorithm (such as HOG, SIFT, CNN, etc.) is used to determine the target feature vector corresponding to the target segmentation result according to the target filtering result and the target color information. Then, the target feature vector is subjected to feature brightness enhancement processing using the brightness enhancement layer of the image reconstruction model to obtain a first enhanced feature vector. For example, a brightness enhancement algorithm (such as histogram equalization, contrast adjustment, etc.) is used for processing, thereby improving the visual quality of the image and facilitating the generation of a clearer image.

[0082] Exemplarily, the detail reconstruction layer of the image reconstruction model is used to perform detail feature enhancement processing according to the first enhanced feature vector to obtain a second enhanced feature vector. For example, a detail enhancement algorithm (such as sharpening, detail filtering, etc.) is used for processing, thereby enhancing the detail information of the image, making the generated image more realistic and delicate, and improving the detail quality of the image, making the image more in line with expectations.

[0083] Exemplarily, the target generation layer of the image reconstruction model is used to generate an image of the second enhanced feature vector to obtain target object information. For example, an image generation algorithm (such as GAN, VAE, etc.) is used for generation, and then a high-quality image is generated based on the enhanced feature vector to ensure the realism and details of the image.

[0084] Exemplarily, three-dimensional modeling technology (such as Mesh generation, point cloud reconstruction, etc.) is used to virtually construct the target object information to obtain corresponding second three-dimensional information, so that the generated second three-dimensional information can truly reflect the target object.

[0085] Specifically, by effectively converting the initial object information into high-quality second 3D information, the generated 3D model not only meets the requirements of the initial object information, but also has a high degree of realism and detail quality. This provides a solid data foundation for the subsequent mixed reality generation results, ensuring that the generated content can be perfectly integrated with the real environment and enhance the user experience.

[0086] Step S106: generate mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

[0087] Exemplarily, the first three-dimensional information and the second three-dimensional information are aligned using techniques such as coordinate system conversion and scale normalization, and then a feature fusion algorithm, such as weighted fusion and deep learning fusion, is used to integrate the features of the aligned first three-dimensional information and the second three-dimensional information to obtain target three-dimensional information, and then the fused target three-dimensional information is rendered using a three-dimensional rendering engine, such as OpenGL or Unity, according to the color of the first three-dimensional information in the initial image information and the color of the second three-dimensional image in the target image information to generate a visual mixed reality generation result.

[0088] Through these steps, the first three-dimensional information and the second three-dimensional information can be effectively fused and combined with the initial voice information to generate high-quality mixed reality content, providing users with a real and immersive experience. This not only improves the quality of the mixed reality content, but also enhances the interactivity between the user and the virtual environment, improving the user experience.

[0089] In some embodiments, the generating of mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information includes: performing information splicing on the first three-dimensional information and the second three-dimensional information to obtain initial overlap information corresponding to the first three-dimensional information and the second three-dimensional information; performing information fusion on the initial overlap information to obtain target overlap information corresponding to the initial overlap information; and generating mixed reality content according to the first three-dimensional information, the second three-dimensional information, and the target overlap information to obtain the mixed reality generation result corresponding to the initial voice information.

[0090] Exemplarily, the first three-dimensional information and the second three-dimensional information are spliced ​​to obtain initial overlap information, for example, by using coordinate alignment and data splicing technology, the first three-dimensional information and the second three-dimensional information are spliced ​​in space to form a unified three-dimensional data set.

[0091] Exemplarily, an information fusion algorithm (such as weighted average, deep learning fusion, etc.) is used to process the initial overlapping information after splicing, eliminate redundant information, enhance key features, and thus obtain target overlapping information.

[0092] Exemplarily, the initial overlap information is removed from the first three-dimensional information to obtain the third three-dimensional information, and the initial overlap information is removed from the second three-dimensional information to obtain the fourth three-dimensional information, and then data is merged according to the third three-dimensional information, the fourth three-dimensional information and the target overlap information to obtain the target three-dimensional information.

[0093] Exemplarily, the fused target 3D information is rendered using a 3D rendering engine, such as OpenGL or Unity, according to the color of the third 3D information in the initial image information, the color of the fourth 3D image in the target image information, and the color of the target coincidence information after the average color processing in the initial image information and the target image information, to generate a visualized mixed reality generation result. Thus, by fusing multi-source 3D information, the generated mixed reality content is more realistic and delicate, and can be seamlessly integrated with the real environment.

[0094] Specifically, through information splicing, fusion and mixed reality content generation, the first three-dimensional information and the second three-dimensional information can be integrated into high-quality mixed reality content. This process not only improves the integrity and accuracy of the information, but also enhances the realism and immersion of the mixed reality content, providing users with a richer and more realistic experience. The final mixed reality generation result can be perfectly integrated with the real environment to meet the needs of various application scenarios.

[0095] See also Figure 2 , Figure 2A mixed reality content generation device 200 based on an AI big model is provided in an embodiment of the present application. The mixed reality content generation device 200 based on the AI ​​big model includes a data acquisition module 201, a speech recognition module 202, an image generation module 203, an image segmentation module 204, a virtual construction module 205, and a result generation module 206, wherein the data acquisition module 201 is used to collect initial speech information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located; the speech recognition module 202 is used to perform speech recognition on the initial speech information to obtain target text information; the image generation module 203, Used to perform image generation based on the target text information and the initial image information to obtain target image information corresponding to the target text information; an image segmentation module 204, used to perform image segmentation on the target image information based on the initial image information to obtain initial object information of the target image information that does not appear in the initial image information; a virtual construction module 205, used to perform virtual construction on the initial object information to obtain second three-dimensional information corresponding to the initial object information; a result generation module 206, used to perform mixed reality content generation based on the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

[0096] In some implementations, the speech recognition module 202 performs, during the process of performing speech recognition on the initial speech information to obtain the target text information:

[0097] The target voice information is obtained after pre-processing operations of pre-emphasis, frame windowing and denoising are performed on the initial voice information;

[0098] Performing speech feature extraction on the target speech information using a feature extraction model to obtain initial Mel frequency cepstrum parameters, initial short-time average energy parameters, and initial spectrum mean parameters corresponding to the target speech information;

[0099] Standardizing the initial Mel frequency cepstrum parameter, the initial short-time average energy parameter and the initial spectrum mean parameter to obtain target Mel frequency cepstrum parameter, target short-time average energy parameter and target spectrum mean parameter;

[0100] The target text information is obtained by performing speech recognition using a speech recognition model in combination with the target Mel-frequency cepstrum parameter, the target short-time average energy parameter and the target spectrum mean parameter.

[0101] In some implementations, the speech recognition module 202 performs the following steps in the process of obtaining the feature extraction model:

[0102] Obtaining a training speech signal and target feature information corresponding to the training speech signal, wherein the training speech signal corresponds to an audio pronunciation of each character;

[0103] Determine initial model parameters corresponding to the speech recognition model, and perform operations through an input layer, a hidden layer, and an output layer corresponding to the initial model parameters according to the initial speech parameters and the training speech signal to obtain prediction feature information;

[0104] Performing a difference calculation based on the target feature information and the predicted feature information to obtain difference information, and comparing the difference information with a preset threshold to obtain a comparison result;

[0105] When the comparison result is that the difference information is greater than or equal to the preset threshold, the initial model parameters are updated and then the speech recognition model is continuously trained using the training speech signal and the target feature information until the target model parameters corresponding to the speech recognition model are obtained;

[0106] When the comparison result is that the difference information is less than the preset threshold, the target model parameters corresponding to the speech recognition model are obtained, and then the speech recognition model is determined according to the target model parameters.

[0107] In some implementations, the image generation module 203, in the process of performing image generation according to the target text information and the initial image information to obtain the target image information corresponding to the target text information, performs:

[0108] Performing text-image matching on the target text information and the initial image information to obtain associated image information corresponding to the matching of the target text information and the initial image information;

[0109] Determining basic image information corresponding to the target text information according to the associated image information;

[0110] Using the text feature extraction layer of the text generation image model to extract features of the target text information to obtain text feature information;

[0111] Using the image feature extraction layer of the text-generated image model to extract features from the basic image information to obtain initial image feature information;

[0112] Using the feature fusion layer of the text-to-image model to perform sentence feature fusion on the text feature information to obtain sentence feature information;

[0113] Using the feature adding layer of the text-to-image model, random noise is added to the sentence feature information to obtain splicing feature information;

[0114] Using the text image affine layer of the text generation image model to perform image processing on the splicing feature information and the initial image feature information to obtain target image feature information;

[0115] Obtaining intermediate image information corresponding to the target text information according to the target image feature information using the convolutional network layer of the text-to-image model;

[0116] The image enhancement layer of the text generation image model is used to perform image refinement on the intermediate image information to obtain the target image information corresponding to the target text information.

[0117] In some implementations, the image generation module 203 performs, during the process of performing image processing on the splicing feature information and the initial image feature information using the text image affine layer of the text generation image model to obtain the target image feature information:

[0118] Determine the corresponding target relationship between the target objects involved in the target text information based on the splicing feature information using the relationship recognition network of the text image affine layer;

[0119] Determine the target position information corresponding to the target object that does not exist in the initial image information according to the initial position information of the target object in the initial image information and the target relationship using the position determination network of the text image affine layer;

[0120] The image space layout network is determined by utilizing the position of the text image affine layer, and the target position information is used to adjust the generation position of the initial image feature information to obtain the target image feature information.

[0121] In some implementations, the image generation module 203, in the process of performing image refinement on the intermediate image information using the image enhancement layer of the text generation image model to obtain the target image information corresponding to the target text information, performs:

[0122] Using the image feature recognition network of the image enhancement layer to extract image features from the intermediate image information to obtain initial image features;

[0123] Calculating the information association value between the splicing feature information and the initial image feature using the text-image association network of the image enhancement layer;

[0124] Determine the target weight corresponding to the initial image feature according to the information association value using the weight calculation layer of the image enhancement layer;

[0125] Using the feature update network of the image enhancement layer to update the initial image features according to the target weights to obtain updated target image features;

[0126] Using the detail correction network of the image enhancement layer to supplement the missing features of the target image features to obtain the final image features;

[0127] Using the image generation network of the image enhancement layer to generate the target image information corresponding to the target text information according to the final image features;

[0128] The information association value is obtained according to the following formula:

[0129]

[0130] in, represents the information association value between the i-th splicing feature information and the initial image feature, represents the i-th splicing feature information, represents the matrix required to transform the i-th splicing feature information, represents the tth initial image feature, m represents the number of features corresponding to the initial image feature, Representation conversion The required matrix, represents the activation function, and Represents a 1*1 convolution, which is used to perform dimensional conversion so that the i-th concatenated feature information and have the same feature space.

[0131] In some implementations, the virtual construction module 205, during the process of virtually constructing the initial object information to obtain the second three-dimensional information corresponding to the initial object information, executes:

[0132] Performing filtering processing on the initial object information to obtain a target filtering result, and performing color space conversion on the initial object information to obtain corresponding target color information in a target color space;

[0133] Determine a target segmentation block, and perform image segmentation on the initial object information according to the target segmentation block to obtain a target segmentation result corresponding to the target segmentation block;

[0134] Determine a target feature vector corresponding to the target segmentation result according to the target filtering result and the target color information;

[0135] Using the brightness enhancement layer of the image reconstruction model to perform feature brightness enhancement processing according to the target feature vector to obtain a first enhanced feature vector;

[0136] Using the detail reconstruction layer of the image reconstruction model, performing detail feature enhancement processing according to the first enhanced feature vector to obtain a second enhanced feature vector;

[0137] Using the target generation layer of the image reconstruction model to perform image generation on the second enhanced feature vector to obtain target object information;

[0138] The second three-dimensional information is obtained by performing virtual construction according to the target object information.

[0139] In some implementations, the result generation module 206, in the process of generating mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information, performs:

[0140] Splicing the first three-dimensional information and the second three-dimensional information to obtain initial overlap information corresponding to the first three-dimensional information and the second three-dimensional information;

[0141] Performing information fusion on the initial overlap information to obtain target overlap information corresponding to the initial overlap information;

[0142] Mixed reality content is generated according to the first three-dimensional information, the second three-dimensional information and the target coincidence information to obtain the mixed reality generation result corresponding to the initial voice information.

[0143] In some embodiments, the mixed reality content generation device 200 based on the AI ​​big model can be applied to a terminal device.

[0144] It should be noted that those skilled in the art can clearly understand that, for the sake of convenience and conciseness of description, the specific working process of the mixed reality content generation device 200 based on the AI ​​big model described above can refer to the corresponding process in the aforementioned embodiment of the mixed reality content generation method based on the AI ​​big model, and will not be repeated here.

[0145] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0146] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected via a bus 303 , such as an I2C (Inter-integrated Circuit) bus.

[0147] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0148] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0149] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] Among them, the processor is used to run the computer program stored in the memory, and implement any one of the mixed reality content generation methods based on the AI ​​big model provided by the embodiments of the present invention when executing the computer program.

[0151] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0152] Collecting initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located;

[0153] Performing speech recognition on the initial speech information to obtain target text information;

[0154] Performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information;

[0155] Performing image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information;

[0156] Performing virtual construction on the initial object information to obtain second three-dimensional information corresponding to the initial object information;

[0157] Mixed reality content is generated according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

[0158] In some implementations, the processor 301, during the process of performing speech recognition on the initial speech information to obtain target text information, executes:

[0159] The target voice information is obtained after pre-processing operations of pre-emphasis, frame windowing and denoising are performed on the initial voice information;

[0160] Performing speech feature extraction on the target speech information using a feature extraction model to obtain initial Mel frequency cepstrum parameters, initial short-time average energy parameters, and initial spectrum mean parameters corresponding to the target speech information;

[0161] Standardizing the initial Mel frequency cepstrum parameter, the initial short-time average energy parameter and the initial spectrum mean parameter to obtain target Mel frequency cepstrum parameter, target short-time average energy parameter and target spectrum mean parameter;

[0162] The target text information is obtained by performing speech recognition using a speech recognition model in combination with the target Mel-frequency cepstrum parameter, the target short-time average energy parameter and the target spectrum mean parameter.

[0163] In some implementations, the processor 301, in the process of obtaining the feature extraction model, executes:

[0164] Obtaining a training speech signal and target feature information corresponding to the training speech signal, wherein the training speech signal corresponds to an audio pronunciation of each character;

[0165] Determine initial model parameters corresponding to the speech recognition model, and perform operations through an input layer, a hidden layer, and an output layer corresponding to the initial model parameters according to the initial speech parameters and the training speech signal to obtain prediction feature information;

[0166] Performing a difference calculation based on the target feature information and the predicted feature information to obtain difference information, and comparing the difference information with a preset threshold to obtain a comparison result;

[0167] When the comparison result is that the difference information is greater than or equal to the preset threshold, the initial model parameters are updated and then the speech recognition model is continuously trained using the training speech signal and the target feature information until the target model parameters corresponding to the speech recognition model are obtained;

[0168] When the comparison result is that the difference information is less than the preset threshold, the target model parameters corresponding to the speech recognition model are obtained, and then the speech recognition model is determined according to the target model parameters.

[0169] In some implementations, during the process of performing image generation according to the target text information and the initial image information to obtain the target image information corresponding to the target text information, the processor 301 executes:

[0170] Performing text-image matching on the target text information and the initial image information to obtain associated image information corresponding to the matching of the target text information and the initial image information;

[0171] Determining basic image information corresponding to the target text information according to the associated image information;

[0172] Using the text feature extraction layer of the text generation image model to extract features of the target text information to obtain text feature information;

[0173] Using the image feature extraction layer of the text-generated image model to extract features from the basic image information to obtain initial image feature information;

[0174] Using the feature fusion layer of the text-to-image model to perform sentence feature fusion on the text feature information to obtain sentence feature information;

[0175] Using the feature adding layer of the text-to-image model, random noise is added to the sentence feature information to obtain splicing feature information;

[0176] Using the text image affine layer of the text generation image model to perform image processing on the splicing feature information and the initial image feature information to obtain target image feature information;

[0177] Obtaining intermediate image information corresponding to the target text information according to the target image feature information using the convolutional network layer of the text-to-image model;

[0178] The image enhancement layer of the text generation image model is used to perform image refinement on the intermediate image information to obtain the target image information corresponding to the target text information.

[0179] In some implementations, the processor 301, in the process of performing image processing on the splicing feature information and the initial image feature information using the text image affine layer of the text generation image model to obtain the target image feature information, executes:

[0180] Determine the corresponding target relationship between the target objects involved in the target text information based on the splicing feature information using the relationship recognition network of the text image affine layer;

[0181] Determine the target position information corresponding to the target object that does not exist in the initial image information according to the initial position information of the target object in the initial image information and the target relationship using the position determination network of the text image affine layer;

[0182] The image space layout network is determined by utilizing the position of the text image affine layer, and the target position information is used to adjust the generation position of the initial image feature information to obtain the target image feature information.

[0183] In some implementations, the processor 301, in the process of performing image refinement on the intermediate image information using the image enhancement layer of the text generation image model to obtain the target image information corresponding to the target text information, executes:

[0184] Using the image feature recognition network of the image enhancement layer to extract image features from the intermediate image information to obtain initial image features;

[0185] Calculating the information association value between the splicing feature information and the initial image feature using the text-image association network of the image enhancement layer;

[0186] Determine the target weight corresponding to the initial image feature according to the information association value using the weight calculation layer of the image enhancement layer;

[0187] Using the feature update network of the image enhancement layer to update the initial image features according to the target weights to obtain updated target image features;

[0188] Using the detail correction network of the image enhancement layer to supplement the missing features of the target image features to obtain the final image features;

[0189] Using the image generation network of the image enhancement layer to generate the target image information corresponding to the target text information according to the final image features;

[0190] The information association value is obtained according to the following formula:

[0191]

[0192] in, represents the information association value between the i-th splicing feature information and the initial image feature, represents the i-th splicing feature information, represents the matrix required to transform the i-th splicing feature information, represents the tth initial image feature, m represents the number of features corresponding to the initial image feature, Representation conversion The required matrix, represents the activation function, and Represents a 1*1 convolution, which is used to perform dimensional conversion so that the i-th concatenated feature information and have the same feature space.

[0193] In some implementations, the processor 301, during the process of virtually constructing the initial object information to obtain the second three-dimensional information corresponding to the initial object information, executes:

[0194] Performing filtering processing on the initial object information to obtain a target filtering result, and performing color space conversion on the initial object information to obtain corresponding target color information in a target color space;

[0195] Determine a target segmentation block, and perform image segmentation on the initial object information according to the target segmentation block to obtain a target segmentation result corresponding to the target segmentation block;

[0196] Determine a target feature vector corresponding to the target segmentation result according to the target filtering result and the target color information;

[0197] Using the brightness enhancement layer of the image reconstruction model to perform feature brightness enhancement processing according to the target feature vector to obtain a first enhanced feature vector;

[0198] Using the detail reconstruction layer of the image reconstruction model, performing detail feature enhancement processing according to the first enhanced feature vector to obtain a second enhanced feature vector;

[0199] Using the target generation layer of the image reconstruction model to perform image generation on the second enhanced feature vector to obtain target object information;

[0200] The second three-dimensional information is obtained by performing virtual construction according to the target object information.

[0201] In some implementations, the processor 301, in the process of generating mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information, executes:

[0202] Splicing the first three-dimensional information and the second three-dimensional information to obtain initial overlap information corresponding to the first three-dimensional information and the second three-dimensional information;

[0203] Performing information fusion on the initial overlap information to obtain target overlap information corresponding to the initial overlap information;

[0204] Mixed reality content is generated according to the first three-dimensional information, the second three-dimensional information and the target coincidence information to obtain the mixed reality generation result corresponding to the initial voice information.

[0205] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned embodiment of the mixed reality content generation method based on the AI ​​big model, and will not be repeated here.

[0206] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of the mixed reality content generation method based on the AI ​​big model provided in the description of the embodiment of the present invention.

[0207] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0208] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0209] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0210] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating mixed reality content based on an AI big model, characterized in that: The method comprises: Collecting initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located; Performing speech recognition on the initial speech information to obtain target text information; Performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information; Performing image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information; Performing virtual construction on the initial object information to obtain second three-dimensional information corresponding to the initial object information; Mixed reality content is generated according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

2. The method according to claim 1, characterized in that The performing speech recognition on the initial speech information to obtain target text information includes: The target voice information is obtained after pre-processing operations of pre-emphasis, frame windowing and denoising are performed on the initial voice information; Performing speech feature extraction on the target speech information using a feature extraction model to obtain initial Mel frequency cepstrum parameters, initial short-time average energy parameters, and initial spectrum mean parameters corresponding to the target speech information; Standardizing the initial Mel frequency cepstrum parameter, the initial short-time average energy parameter and the initial spectrum mean parameter to obtain target Mel frequency cepstrum parameter, target short-time average energy parameter and target spectrum mean parameter; The target text information is obtained by performing speech recognition using a speech recognition model in combination with the target Mel-frequency cepstrum parameter, the target short-time average energy parameter and the target spectrum mean parameter.

3. The method according to claim 2, characterized in that Obtaining the feature extraction model includes: Obtaining a training speech signal and target feature information corresponding to the training speech signal, wherein the training speech signal corresponds to an audio pronunciation of each character; Determine initial model parameters corresponding to the speech recognition model, and perform operations through an input layer, a hidden layer, and an output layer corresponding to the initial model parameters according to the initial speech parameters and the training speech signal to obtain prediction feature information; Performing a difference calculation based on the target feature information and the predicted feature information to obtain difference information, and comparing the difference information with a preset threshold to obtain a comparison result; When the comparison result is that the difference information is greater than or equal to the preset threshold, the initial model parameters are updated and then the speech recognition model is continuously trained using the training speech signal and the target feature information until the target model parameters corresponding to the speech recognition model are obtained; When the comparison result is that the difference information is less than the preset threshold, the target model parameters corresponding to the speech recognition model are obtained, and then the speech recognition model is determined according to the target model parameters.

4. The method according to claim 1, characterized in that: The step of generating an image according to the target text information and the initial image information to obtain target image information corresponding to the target text information includes: Performing text-image matching on the target text information and the initial image information to obtain associated image information corresponding to the matching of the target text information and the initial image information; Determining basic image information corresponding to the target text information according to the associated image information; Using the text feature extraction layer of the text generation image model to extract features of the target text information to obtain text feature information; Using the image feature extraction layer of the text-generated image model to extract features from the basic image information to obtain initial image feature information; Using the feature fusion layer of the text-to-image model to perform sentence feature fusion on the text feature information to obtain sentence feature information; Using the feature adding layer of the text-to-image model, random noise is added to the sentence feature information to obtain splicing feature information; Using the text image affine layer of the text generation image model to perform image processing on the splicing feature information and the initial image feature information to obtain target image feature information; Obtaining intermediate image information corresponding to the target text information according to the target image feature information using the convolutional network layer of the text-to-image model; The image enhancement layer of the text generation image model is used to perform image refinement on the intermediate image information to obtain the target image information corresponding to the target text information.

5. The method according to claim 4, characterized in that The step of performing image processing on the splicing feature information and the initial image feature information using the text image affine layer of the text generation image model to obtain target image feature information includes: Determine the corresponding target relationship between the target objects involved in the target text information based on the splicing feature information using the relationship recognition network of the text image affine layer; Determine the target position information corresponding to the target object that does not exist in the initial image information according to the initial position information of the target object in the initial image information and the target relationship using the position determination network of the text image affine layer; The image space layout network is determined by utilizing the position of the text image affine layer, and the target position information is used to adjust the generation position of the initial image feature information to obtain the target image feature information.

6. The method according to claim 4, characterized in that The step of performing image refinement on the intermediate image information by using the image enhancement layer of the text generation image model to obtain the target image information corresponding to the target text information includes: Using the image feature recognition network of the image enhancement layer to extract image features from the intermediate image information to obtain initial image features; Calculating the information association value between the splicing feature information and the initial image feature using the text-image association network of the image enhancement layer; Determine the target weight corresponding to the initial image feature according to the information association value using the weight calculation layer of the image enhancement layer; Using the feature update network of the image enhancement layer to update the initial image features according to the target weights to obtain updated target image features; Using the detail correction network of the image enhancement layer to supplement the missing features of the target image features to obtain the final image features; Using the image generation network of the image enhancement layer to generate the target image information corresponding to the target text information according to the final image features; The information association value is obtained according to the following formula: ； in, represents the information association value between the i-th splicing feature information and the initial image feature, represents the i-th splicing feature information, represents the matrix required to transform the i-th splicing feature information, represents the tth initial image feature, m represents the number of features corresponding to the initial image feature, Representation conversion The required matrix, represents the activation function, and Represents a 1*1 convolution, which is used to perform dimensional conversion so that the i-th concatenated feature information and have the same feature space.

7. The method according to claim 1, characterized in that The virtually constructing the initial object information to obtain the second three-dimensional information corresponding to the initial object information includes: Performing filtering processing on the initial object information to obtain a target filtering result, and performing color space conversion on the initial object information to obtain corresponding target color information in a target color space; Determine a target segmentation block, and perform image segmentation on the initial object information according to the target segmentation block to obtain a target segmentation result corresponding to the target segmentation block; Determine a target feature vector corresponding to the target segmentation result according to the target filtering result and the target color information; Using the brightness enhancement layer of the image reconstruction model to perform feature brightness enhancement processing according to the target feature vector to obtain a first enhanced feature vector; Using the detail reconstruction layer of the image reconstruction model, performing detail feature enhancement processing according to the first enhanced feature vector to obtain a second enhanced feature vector; Using the target generation layer of the image reconstruction model to perform image generation on the second enhanced feature vector to obtain target object information; The second three-dimensional information is obtained by performing virtual construction according to the target object information.

8. The method according to claim 1, characterized in that The generating mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information includes: Splicing the first three-dimensional information and the second three-dimensional information to obtain initial overlap information corresponding to the first three-dimensional information and the second three-dimensional information; Performing information fusion on the initial overlap information to obtain target overlap information corresponding to the initial overlap information; Mixed reality content is generated according to the first three-dimensional information, the second three-dimensional information and the target coincidence information to obtain the mixed reality generation result corresponding to the initial voice information.

9. A mixed reality content generation device based on an AI big model, characterized in that: include: A data acquisition module, used for acquiring initial voice information corresponding to a target user and initial image information and first three-dimensional information corresponding to a target environment where the target user is located; A speech recognition module, used for performing speech recognition on the initial speech information to obtain target text information; An image generation module, used for performing image generation according to the target text information and the initial image information to obtain target image information corresponding to the target text information; An image segmentation module, configured to perform image segmentation on the target image information according to the initial image information to obtain initial object information of the target image information that does not appear in the initial image information; A virtual construction module, used to virtually construct the initial object information to obtain second three-dimensional information corresponding to the initial object information; A result generation module is used to generate mixed reality content according to the first three-dimensional information and the second three-dimensional information to obtain a mixed reality generation result corresponding to the initial voice information.

10. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the mixed reality content generation method based on the AI ​​big model as described in any one of claims 1 to 8 when executing the computer program.

Images