Personalized audio-visual fusion intelligent interaction system based on multi-modal data and use method thereof

Abstract

The invention discloses a personalized audio-visual fusion intelligent interaction system based on multi-modal data and a use method thereof. The system comprises a multi-modal data receiving module, a facial feature extraction module, a three-dimensional model generation module, an audio feature extraction module, a virtual sound generation module, a virtual image integration module and an interaction module. A user uploads pictures, videos and audios, the system extracts face and audio features, virtual images and virtual sounds are generated, and the virtual images, the virtual sounds and the virtual sounds are adjusted and matched. And the interaction module displays the virtual image, the user inputs the dialogue content, and the system generates a reply through an algorithm and performs video and audio interaction through the virtual image. The virtual image and sound generation and interaction based on the multi-modal data can be realized, so that the highly personalized and intelligent user experience is provided. According to the invention, not only can highly vivid virtual images and sounds be generated, but also natural and smooth user interaction can be realized through the intelligent interaction module, so that the quality and satisfaction of user experience are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a personalized audio-visual fusion intelligent interaction system based on multimodal data and a method for using the system. Background Art

[0002] In the current digital age, avatar generation and interaction technology has become a bridge connecting the real world and the digital world, providing users with an unprecedented immersive experience. However, traditional avatar generation and interaction systems mostly rely on pre-defined templates and static rules, which greatly limits the personalization and intelligence level of the system. When using these systems, users often find that the avatar lacks sufficient personality characteristics, and the interaction process seems mechanical and monotonous, which makes it difficult to meet users' needs for highly realistic, natural and personalized interactive experiences.

[0003] With the rapid development of deep learning, computer vision, speech recognition, and natural language processing, the accuracy and efficiency of image and audio processing technologies have been significantly improved, providing strong technical support for the innovation of virtual image generation and interaction technologies. In particular, the application of deep learning algorithms enables the system to more accurately capture the user's facial features, voice characteristics, and even emotional changes, laying a solid foundation for generating highly personalized virtual images. Therefore, the development of a new system based on deep learning technology and multimodal fusion interaction not only has important technical value, but will also bring users a new digital experience and promote the rapid development of virtual reality, artificial intelligence, and other fields. Summary of the invention

[0004] The embodiments of the present application provide a personalized audio-visual fusion intelligent interaction system based on multimodal data and a method of using the system, thereby solving the problem in the prior art of lacking sufficient personalized features and being difficult to meet users' needs for highly realistic, natural and personalized interactive experiences.

[0005] In order to achieve the above object, the technical solution of the embodiment of the present invention is:

[0006] In a first aspect, an embodiment of the present invention provides a personalized audio-visual fusion intelligent interaction system based on multimodal data, including: a multimodal data receiving module, used to receive multimodal data uploaded by a first target user; the multimodal data at least includes pictures, videos and audio files containing portrait information and audio information of the first target user; a facial feature extraction module, used to perform facial recognition and analysis on the first target user according to the multimodal data, and obtain facial feature data of the first target user; the facial feature data at least includes the facial contour and facial features of the first target user; a three-dimensional model generation module, used to generate a corresponding three-dimensional facial model according to the extracted facial feature data of the first target user, and perform detail optimization and rendering to generate a virtual image of the first target user; an audio feature extraction module, used to extract the voice features of the first target user through multimodal data; the voice features at least include pitch, timbre, and speaking speed; a virtual sound generation module, used to generate a virtual sound matching the original sound of the first target user based on the voice features of the first target user. The virtual image integration module is used to adjust and match the virtual image and virtual voice of the first target user so that the virtual image's lip shape, expression and audio are synchronized in the video interaction; the interaction module is used to display the virtual image through the interactive interface, and receive the dialogue content input by the second target user through the interactive interface, generate reply content according to the dialogue content through a preset intelligent algorithm, and use the virtual image integration module to provide vivid and natural interactive feedback on the reply content through the virtual image in the form of video and audio integration.

[0007] In some possible implementations, the multimodal data receiving module also has a data preprocessing function, including but not limited to data compression, denoising, and format conversion, to improve data processing efficiency and accuracy.

[0008] In some possible implementations, the system also includes: a personalized customization module for adjusting the appearance characteristics of the virtual image and / or the pitch, timbre, and speaking speed of the virtual voice according to the personal preferences of the first target user to achieve a more personalized virtual image generation and interactive experience; wherein the appearance characteristics include but are not limited to skin color, hairstyle, clothing, and accessories.

[0009] In some possible implementations, the system also includes: an emotion recognition and response module, which uses emotion recognition technology to adjust the pitch, timbre, and speaking speed parameters of the virtual sound according to the emotional changes in the voice of the first target user, so that the virtual sound is emotionally expressive.

[0010] In a second aspect, an embodiment of the present invention provides a method for using a personalized audio-visual fusion intelligent interaction system based on multimodal data, which is applied to the personalized audio-visual fusion intelligent interaction system based on multimodal data in the first aspect, including: a first target user uploads pictures, videos and audio files containing portrait information and audio information of the first target user through an interactive interface; facial features of the first target user in the uploaded image and / or video are extracted; a three-dimensional facial model is generated based on the extracted facial features and optimized to obtain a virtual image of the first target user; voice features of the uploaded audio and / or video are extracted to obtain voice features of the first target user; a virtual voice of the first target user is generated based on the voice features; the virtual model is coordinated with the virtual voice; the virtual image is displayed, and input from a second target user is received, reply content is generated through a preset intelligent algorithm, and the reply content is fed back through video and audio interaction through the virtual image.

[0011] In some possible implementations, after acquiring the multimodal data uploaded by the first target user, the method further includes: preprocessing the multimodal data, including but not limited to data compression, denoising, and format conversion, to improve data processing efficiency and accuracy.

[0012] In some possible implementations, after obtaining the virtual image of the first target user, the method further includes: adjusting the appearance characteristics of the virtual image and / or the pitch, timbre, and speaking speed of the virtual voice according to the personal preferences of the first target user to achieve a more personalized virtual image generation and interactive experience; wherein the appearance characteristics include but are not limited to skin color, hairstyle, clothing, and accessories.

[0013] In some possible implementations, after generating a virtual voice of the first target user based on voice features, the method further includes: using emotion recognition technology to adjust the pitch, timbre, and speaking speed parameters of the virtual voice according to emotional changes in the voice of the first target user, so that the virtual voice is emotionally expressive.

[0014] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0015] In the embodiments of the present invention, it is possible to realize the generation and interaction of virtual images and sounds based on multimodal data, thereby providing a highly personalized and intelligent user experience. The system can not only generate highly realistic virtual images and sounds, but also realize natural and smooth user interaction through intelligent interaction modules, greatly improving the quality and satisfaction of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention, the accompanying drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0017] Figure 1 A structural schematic diagram of a personalized audio-visual fusion intelligent interaction system based on multimodal data provided for the implementation of the present invention;

[0018] Figure 2 The present invention is a flowchart of a method for using a personalized audio-visual fusion intelligent interaction system based on multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In the relevant description of this embodiment, the terms "including, containing, having" and the like are open terms and are generally understood to include but not be limited to; the term "at least one" is generally understood to mean one or more, where "plurality" refers to two or more; the term "at least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items, for example, "at least one of a, b or c", or "at least one of a, b and c", can all represent: a, b, c, ab (i.e., a and b), ac, bc, or abc, where a, b, c can be single or multiple, respectively; the symbol "A / B" is used to describe the selection relationship of associated objects, generally indicating an "or" relationship before and after.

[0021] In the following description of the present embodiment, the terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0022] Those skilled in the art should understand that in the following description of the embodiments of the present application, the order of serial numbers does not mean the order of execution, some or all of the steps can be executed in parallel or sequentially, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] Those skilled in the art will appreciate that the numerical ranges in the embodiments of the present application are to be construed as also specifically disclosing each intermediate value between the upper and lower limits of the scope. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the range is also included in the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded in the scope.

[0024] Unless otherwise specified, the technical / scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this application belongs. Although this application only describes preferred methods and materials, any methods and materials similar or equivalent to these may also be used in the implementation or testing of this application. All documents mentioned in this specification are incorporated by reference to disclose and describe methods and / or materials related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.

[0025] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0026] In the current digital age, avatar generation and interaction technology has become a bridge connecting the real world and the digital world, providing users with an unprecedented immersive experience. However, traditional avatar generation and interaction systems mostly rely on pre-defined templates and static rules, which greatly limits the personalization and intelligence level of the system. When using these systems, users often find that the avatar lacks sufficient personality characteristics, and the interaction process seems mechanical and monotonous, which makes it difficult to meet users' needs for highly realistic, natural and personalized interactive experiences.

[0027] With the rapid development of deep learning, computer vision, speech recognition and natural language processing, the accuracy and efficiency of image and audio processing technologies have been significantly improved, providing strong technical support for the innovation of virtual image generation and interaction technology. In particular, the application of deep learning algorithms enables the system to more accurately capture the user's facial features, voice characteristics and even emotional changes, laying a solid foundation for the generation of highly personalized virtual images.

[0028] In this context, it is particularly urgent to build a new virtual image generation and interaction system that can combine image processing and audio processing. The system aims to break the limitations of traditional systems and realize the intelligent generation and highly personalized customization of virtual images through the empowerment of deep learning technology. At the same time, through the application of multimodal fusion interaction technology, the system will be able to provide a more natural, smooth and emotional interactive experience, meeting users' pursuit of higher interactivity and personalized needs.

[0029] Therefore, developing a new system based on deep learning technology and multimodal fusion interaction not only has important technical value, but will also bring a new digital experience to users and promote the rapid development of virtual reality, artificial intelligence and other fields.

[0030] Based on this, the embodiments of the present application solve the problem that the prior art lacks sufficient personalized features and is difficult to meet users' needs for highly realistic, natural and personalized interactive experiences by providing a personalized audio-visual fusion intelligent interaction system based on multimodal data and a method of using the system.

[0031] Figure 1 A structural diagram of a personalized audio-visual fusion intelligent interactive system based on multimodal data provided for the implementation of the present invention, see Figure 1 As shown, the personalized audio-visual fusion intelligent interaction system 10 based on multimodal data may include:

[0032] The multimodal data receiving module 11 is used to receive the multimodal data uploaded by the first target user;

[0033] The multimodal data at least includes pictures, videos and audio files containing the portrait information and audio information of the first target user;

[0034] In some embodiments, the picture file may contain the portrait information of the first target user, which is usually used for facial recognition, expression analysis or generating personalized virtual images. A wide range of file formats are supported, such as JPEG, PNG, etc., to meet the uploading habits and device compatibility requirements of different users. Video files are media that can dynamically display user behavior, emotions and environmental information. The reception of mainstream video formats such as MP4 can be supported. These video data will be used to extract image information from continuous frames, motion analysis, and voice synchronization. Audio files can contain key information such as the user's voice content, intonation, and speaking speed, which is crucial for applications such as speech recognition, sentiment analysis, and personalized speech synthesis. Supported audio formats may include WAV, MP3, etc., to ensure that the system can capture and parse high-quality audio data.

[0035] In some embodiments, in order to improve the user experience, the multimodal data receiving module 11 can provide an intuitive and friendly user interface module, so that the user can easily upload the required multimodal data. The user interface design can follow the principle of simplicity and clarity, and can have at least the following characteristics:

[0036] Multi-format support: The user interface can automatically identify and accept uploads in multiple file formats, reducing the obstacles users encounter due to format inconsistencies.

[0037] Convenient upload method: Users can upload files by dragging and dropping them to the designated area, or by clicking a button to select files to upload. Both methods can improve the convenience of uploading operations.

[0038] Real-time feedback: During the upload process, the system will provide immediate progress feedback and give clear prompts after the upload is completed, so that users can understand the processing status of the data.

[0039] Security assurance: When designing the user interface, attention should be paid to the security of data transmission. Encrypted transmission technology can be used to protect user data and ensure that data is not leaked or tampered with during transmission.

[0040] In some embodiments, when the multimodal data is successfully uploaded, the multimodal data receiving module 11 will be responsible for receiving and storing the data to prepare for subsequent further processing. These data need to be pre-processed (such as data compression, denoising, format conversion, etc.) before they can be effectively used by other functional modules. Therefore, the design of the multimodal data receiving module 11 can also consider the efficient storage and fast retrieval mechanism of data, so that the required data resources can be quickly accessed in the processing flow.

[0041] A facial feature extraction module 12 is used to perform facial recognition and analysis on the first target user according to the multimodal data to obtain facial feature data of the first target user; the facial feature data at least includes the facial contour and facial features of the first target user;

[0042] After obtaining the multimodal data uploaded by the first target user, the facial feature extraction module 12 can first pre-process the multimodal data. This step includes but is not limited to image enhancement (such as brightness adjustment, contrast optimization), noise filtering (using Gaussian filtering, median filtering and other methods to remove image noise), and image size normalization, etc., to ensure the consistency and high quality of the input data. Afterwards, advanced facial detection algorithms, such as Haar feature classifiers, deep learning-based detection networks, etc., are used to quickly locate the facial area in the pre-processed pictures or video frames. It can serve as the basis for subsequent feature extraction and can greatly reduce the interference of non-facial areas. Once the facial area is successfully detected, the system will further identify and mark the key feature points of the face, including facial contours, facial features such as corners of the eyes, corners of the mouth, and the tip of the nose.

[0043] Then, feature extraction is performed through a convolutional neural network (CNN). A convolutional neural network model is used to extract deep features from the marked facial area. CNN automatically learns and abstracts high-level facial feature representations, such as texture, shape, and contour, through multi-layer convolution, pooling, activation, and other operations. These features not only contain macro information of facial contours, but also go deep into the subtle differences in the shapes of facial features. The facial features extracted by CNN are integrated to form a complete set of facial feature datasets for the first target user. This dataset includes but is not limited to the precise description of facial contours, specific parameters of facial features, and other high-level features that may be extracted, such as skin texture, expression features, etc.

[0044] The three-dimensional model generation module 13 is used to generate a corresponding three-dimensional facial model according to the extracted facial feature data of the first target user, and perform detail optimization and rendering to generate a virtual image of the first target user;

[0045] The three-dimensional model generation module 13 is a key link in constructing a personalized virtual image. It creates a highly realistic and detailed three-dimensional facial model based on the detailed facial feature data of the first target user obtained from the facial feature extraction module.

[0046] Specifically, after acquiring the facial features of the first target user, the three-dimensional model generation module 13 first analyzes the received data to ensure the accuracy of each feature point and its relative position in space, providing precise guidance for subsequent three-dimensional modeling. After that, the three-dimensional space mapping and geometric structure creation can be carried out, and the two-dimensional facial feature points are mapped into the three-dimensional space using an algorithm to form the basic outline of the face and the preliminary position of the facial features. This step is the basis for building a three-dimensional model and determines the accuracy and authenticity of the model. Based on the results of the spatial mapping, the three-dimensional geometric structure of the face is created through the principles of computational geometry and computer graphics. This includes defining the curved surface, edges, and connection relationships between various parts of the face to form a preliminary three-dimensional model framework.

[0047] Mesh generation algorithms (such as Delaunay triangulation, Marching Cubes, etc.) can then be used to generate dense meshes on the 3D geometry. These meshes are the basic units that make up the 3D model, and their density and quality directly affect the detailed performance of the model. The generated mesh is refined, and the mesh is made smoother and more natural through smoothing, edge enhancement and other techniques, while retaining important facial feature details. Texture mapping technology is used to apply material information such as skin and hair to the 3D model. This includes selecting appropriate texture images, adjusting the texture mapping method (such as UV mapping), and performing texture fusion and transition processing to ensure that the texture fits the model surface perfectly. The model can then be further optimized in detail, including adjusting the proportions and shapes of the facial features, enhancing facial expression features, and adding personalized details such as moles and wrinkles. These optimization measures are aimed at improving the realism and personalization of the model.

[0048] Finally, advanced rendering techniques (such as ray tracing, global illumination, etc.) can be used to render the 3D model. By simulating the lighting and shadow effects in the real world, the model presents a more realistic and vivid visual effect. The rendered 3D model is output in an appropriate format for use by subsequent application modules. The output format may include but is not limited to 3D model files (such as OBJ, FBX, etc.), scene files in real-time rendering engines, and compressed formats for network transmission.

[0049] In the embodiment of the present invention, the three-dimensional model generation module 13 can successfully generate a three-dimensional model of a virtual image that is highly realistic and detailed with the first target user through a series of complex and sophisticated operations such as accurate analysis of facial feature data, three-dimensional space mapping, geometric structure creation, mesh generation and refinement, texture mapping and detail optimization, rendering and output.

[0050] The audio feature extraction module 14 is used to extract the voice features of the first target user through the multimodal data; the voice features include at least pitch, timbre, and speech speed;

[0051] The audio feature extraction module 14 can obtain the audio files and audio contained in the video files uploaded by the user, which may contain the user's natural voice for subsequent feature extraction. In order to ensure the quality of the extracted voice features, the system can pre-process the received audio signal. This includes but is not limited to noise reduction processing to reduce the interference of background noise on feature extraction; and normalization processing to adjust the amplitude of the audio signal to a uniform range for subsequent processing.

[0052] Before extracting speech features, the preprocessed audio signal is first divided into smaller frames, for example, in tens of milliseconds. Since many characteristics of speech (such as pitch and timbre) are relatively stable in a short period of time, frame-by-frame analysis can be performed. Before formally using the deep learning algorithm, some traditional signal processing methods can be applied to preliminarily screen out information that may be helpful for subsequent feature extraction, such as energy envelope, zero-crossing rate, etc. Afterwards, given the advantages of long short-term memory network (LSTM) in processing sequence data, the LSTM algorithm can be selected to extract key features in audio. LSTM can capture long-term dependencies in audio signals. In the LSTM network, the sequence of audio frames is taken as input, and the network extracts key audio features such as pitch (pitch), timbre (sound quality features) and sound intensity (volume changes) by learning the temporal dependencies between audio frames. These features not only reflect the acoustic properties of the user's voice, but also contain information about the dynamic changes of the voice. In the feature extraction process, some optimization strategies can also be applied, such as feature selection, selecting the most useful features for virtual sound generation, feature dimensionality reduction, reducing the number of features to improve processing efficiency, etc., to further improve the quality and effectiveness of the features. The extracted audio features are sorted and output as the basic data for the subsequent virtual voice generation module. These features contain the uniqueness of the user's voice and are the key to generating personalized virtual voices.

[0053] The virtual voice generating module 15 is used to generate a virtual voice matching the original voice of the first target user based on the voice features of the first target user.

[0054] The virtual sound generation module 15 first receives the audio feature data from the audio feature extraction module 14, including but not limited to key sound attributes such as pitch, timbre, and speech speed. The received feature data is parsed to understand the sound characteristics represented by each type of feature and its role in the process of generating virtual sound.

[0055] Based on the results of feature analysis, the speech synthesis technology that best suits the current audio features can be selected. There are many modern speech synthesis technologies, including but not limited to synthesis based on hidden Markov models (HMM), concatenated synthesis, and neural network synthesis (such as WaveNet, Tacotron, etc.). In order to ensure that the generated virtual voice matches the user's original voice, a speech synthesis model that has or can learn the user's specific voice characteristics can be selected or trained.

[0056] The parsed audio features are used as input to drive the selected speech synthesis model to generate virtual voices. During the generation process, the parameters of the synthesis model can be fine-tuned as needed to ensure that the generated virtual voice is more consistent with the user's original voice in detail. In addition to accuracy, you can also pay attention to the fluency and naturalness of the generated virtual voice. By optimizing the model algorithm or using post-processing technology, you can reduce the synthesis traces and make the virtual voice sound more natural.

[0057] The generated virtual sound can be checked by an automatic quality assessment system to evaluate whether its timbre, pitch, speaking speed, etc. are consistent with the extracted audio features, as well as the overall naturalness and fluency. Or in some cases, the module can also invite human listeners to evaluate the generated virtual sound to provide more subjective feedback. Based on the results of automatic evaluation and manual feedback, its internal algorithms and model parameters can be continuously optimized to improve the quality and effect of virtual sound generation. After the quality assessment, the virtual sound that meets the requirements will be prepared for output for subsequent applications.

[0058] A virtual image integration module 16 is used to adjust and match the virtual image and virtual voice of the first target user so that the lip shape, expression and audio of the virtual image are synchronized in the video interaction;

[0059] The virtual image integration module 16 is responsible for perfectly integrating the virtual image and virtual sound of the first target user in building a personalized audio-visual fusion intelligent interactive experience, ensuring that the lip shape, expression and audio output of the virtual image are highly synchronized during the video interaction process. The module can first load the virtual image of the first target user that has been created or specified, which includes the image's facial appearance, hairstyle and other visual elements. At the same time, the module receives virtual sound data from the virtual sound generation module 15, which contains the user's voice content and its corresponding audio features.

[0060] In order to ensure the synchronization of the virtual image and the audio, the virtual image integration module 16 needs to record the timestamp of the audio data so that each frame of audio can accurately correspond to the action of the virtual image in subsequent processing. Based on the results of the audio feature analysis, the audio-driven facial animation technology is used to map the voice content in the audio to the lip movements of the virtual image. This includes adjusting the shape and opening degree of the virtual image's lips according to the different syllables and words to achieve accurate synchronization of the lip shape. In addition to lip synchronization, the expression of the virtual image will also be adjusted according to the emotional color of the audio (such as happiness, sadness, surprise, etc.) to make it more consistent with the emotions expressed by the audio content. All facial animation adjustments are performed in real time to ensure that during the video interaction process, the lip shape, expression and audio content of the virtual image are always kept synchronized and updated.

[0061] Based on this, in some embodiments, the system may also include an emotion recognition and response module, which uses emotion recognition technology to adjust the pitch, timbre, and speaking speed parameters of the virtual sound according to the emotional changes in the voice of the first target user, so that the virtual sound is emotionally expressive.

[0062] In some embodiments, in order to improve the realism of the interaction, the avatar integration module 16 will continuously optimize the smoothness and naturalness of the avatar's movements to reduce the mechanical or unnatural performance. By collecting user feedback, the module will continuously adjust the optimization algorithm to adapt to the preferences and needs of different users and improve the overall user experience.

[0063] After completing all the above steps, the module will output the integrated virtual image and virtual sound to the video interaction platform for users to interact in real time.

[0064] In some embodiments, in order to expand the scope of application, the virtual image integration module 16 can also ensure that the generated video interaction content can be played smoothly on multiple platforms and devices, including PCs, mobile devices, VR / AR devices, etc.

[0065] In some embodiments, in order to meet the needs of different users, the system may also include a personalized customization module.

[0066] A personalized customization module is used to adjust the appearance characteristics of the virtual image and / or the pitch, timbre, and speaking speed of the virtual voice according to the personal preferences of the first target user to achieve a more personalized virtual image generation and interactive experience; wherein the appearance characteristics include but are not limited to skin color, hairstyle, clothing, and accessories.

[0067] For example, users can select or fine-tune the skin color of the avatar according to their skin color or preferences. The system provides a variety of skin color options, supporting continuous changes from light to dark, and may even include special effects such as sunburn makeup and healthy wheat color. Hairstyle is one of the important ways to show personality. The personalized customization module allows users to select their favorite hairstyle from the preset hairstyle library, including short hair, long hair, curly hair, straight hair and other styles. In addition, users can also adjust the depth and glossiness of hair color, and even add hair accessories such as hairpins and hairbands. Alternatively, users can also adjust the pitch of the virtual voice according to their preferences or needs. Whether it is a low magnetic voice, a crisp and pleasant female voice, or even a special sound effect, it can be achieved by adjusting the pitch. The timbre affects the texture and expressiveness of the sound. The personalized customization module allows users to select or fine-tune the timbre of the virtual voice, such as warm, cool, sweet, mature, etc., to adapt to different communication scenarios and emotional expressions. The speed of speech will also affect the comfort and efficiency of communication. Users can adjust the speed of the virtual voice according to their own speaking habits or the needs of the communication object.

[0068] The interactive module 17 is used to display the virtual image through the interactive interface, and receive the conversation content input by the second target user through the interactive interface, generate reply content according to the conversation content through a preset intelligent algorithm, and use the virtual image integration module to integrate the reply content through the virtual image in the form of video and audio for vivid and natural interactive feedback.

[0069] In the embodiment of the present invention, the interaction module 17 is the core component for building an intelligent and interactive experience. It integrates multiple key functions such as user interface display, natural language processing, intelligent reply generation, and video and audio integrated feedback, providing a vivid and natural dialogue and communication platform between users and virtual images.

[0070] The interaction module 17 can firstly intuitively display the generated virtual image of the first target user through a user interface (UI), ensuring that the user can intuitively see and interact with the virtual image. On the interface, the virtual image can be presented in the form of a high-fidelity image or 3D model, including details such as its appearance and expression, so that the user can experience a communication experience similar to that of a real person.

[0071] The second target user can input the conversation content in a variety of ways, including:

[0072] Text input: The second target user inputs the conversation content he wants to express through the text box or voice input function on the interface.

[0073] Voice input: the second target user can also directly input the conversation content in a voice wake-up manner. The interaction module 17 recognizes the voice input of the second target user through a built-in voice recognition module and converts it into corresponding text content.

[0074] Once the conversation content input by the second target user is received, the interaction module 17 can immediately start the natural language processing (NLP) engine for analysis. NLP technology can understand the meaning, context and emotion of the user input, laying the foundation for the subsequent generation of reasonable replies. Based on the results of NLP analysis, the system can deeply understand the semantic information input by the user, including elements such as subject, intention, and emotion. Using preset intelligent algorithms (such as machine learning models, dialogue management systems, etc.), corresponding reply content is generated according to the semantic information input by the user. These reply contents not only conform to language specifications, but also accurately respond to the needs and emotions of users.

[0075] The generated text reply can be passed to the virtual sound generation module 15 and converted into a realistic audio signal through speech synthesis technology. At the same time, the reply content is also passed to the virtual image integration module 16 to drive the facial animation and body language of the virtual image. The virtual image integration module 16 adjusts the facial expressions and lip movements of the virtual image according to the reply content to ensure that they are synchronized with the audio signal. In this way, when the virtual image speaks, its lip shape, expression and voice content can be perfectly matched, presenting a natural and smooth conversation effect. Finally, the integrated video and audio signals are displayed to the user through the user interface. Through the interactive interface, the user can see the virtual image vividly respond to his input and hear the matching voice reply, thereby obtaining an immersive conversation experience.

[0076] In some embodiments, the entire interactive dialogue process with the virtual image can be real-time and continuous, and the user can get feedback from the virtual image immediately after input, and the dialogue can be continued for multiple times. This immediacy can enhance the fluency and interactivity of the dialogue.

[0077] In some embodiments, the interaction module 17 may also have a dialogue management capability, and may be able to track the history of the dialogue, understand contextual information, and take these factors into account when generating a reply, thereby making the dialogue more coherent and meaningful.

[0078] In some embodiments, the second target user and the first target user may be the same user.

[0079] In the embodiment of the present invention, the system not only provides a personalized experience of interacting with the virtual image, but also further enhances the user's self-expression and immersive feeling. When the second target user is the same person as the first target user, the user can conduct self-reflection and introspection through dialogue with the virtual image. As a customizable and personalized dialogue object, the virtual image can guide the user to explore his or her thoughts, emotions, and behaviors in depth. This self-dialogue method helps users better understand themselves, discover potential problems or advantages, and thus make more informed decisions or adjust behavioral strategies.

[0080] In this mode, users can treat the avatar as their learning partner or mentor. By customizing the specific appearance and voice characteristics of the avatar, users can create an ideal learning environment for themselves. For example, users can set the avatar as an expert in a certain field and obtain professional knowledge or skill guidance by talking to it. In addition, the system can dynamically adjust the avatar's reply content and teaching methods based on the user's learning progress and feedback to achieve a more personalized learning experience. For users who like to create or express, talking to a customized avatar can be a way to inspire inspiration. Users can explore new ideas, storylines or art forms by interacting with the avatar. At the same time, users can also use the system's personalized customization function to continuously adjust and improve the image design of the avatar to better match their own creative style and theme. This creative interactive process not only helps users improve their creative ability, but also brings great fun and a sense of accomplishment to users.

[0081] The personalized audio-visual fusion intelligent interaction system based on multimodal data provided by the embodiment of the present invention can realize the generation and interaction of virtual images and sounds based on multimodal data, thereby providing a highly personalized and intelligent user experience. The system can not only generate highly realistic virtual images and sounds, but also realize natural and smooth user interaction through the intelligent interaction module, greatly improving the quality and satisfaction of user experience.

[0082] Based on the same inventive concept, an embodiment of the present application also provides a method for using a personalized audio-visual fusion intelligent interaction system based on multimodal data. Figure 2 FIG. 1 is a flow chart of an embodiment of a method for using a personalized audio-visual fusion intelligent interaction system based on multimodal data in an embodiment of the present invention, see Figure 2 As shown, the method may include:

[0083] S201, a first target user uploads multimodal data through an interactive interface, where the multimodal data at least includes a picture, a video, and an audio file containing portrait information and audio information of the first target user;

[0084] S202, extracting facial features from the face of the first target user in the uploaded image and / or video;

[0085] S203, generating a three-dimensional facial model according to the extracted facial features and optimizing it to obtain a virtual image of the first target user;

[0086] S204, extracting voice features from the uploaded audio and / or video to obtain voice features of the first target user;

[0087] S205, generating a virtual voice of the first target user based on the voice feature;

[0088] S206, matching the virtual model with the virtual sound;

[0089] S207, displaying the virtual image, and receiving the second target user input, generating reply content through a preset intelligent algorithm, and providing interactive video and audio feedback of the reply content through the virtual image.

[0090] In some possible implementations, after acquiring the multimodal data uploaded by the first target user, the method further includes: preprocessing the multimodal data, including but not limited to data compression, denoising, and format conversion, to improve data processing efficiency and accuracy.

[0091] In some possible implementations, after obtaining the virtual image of the first target user, the method further includes: adjusting the appearance characteristics of the virtual image and / or the pitch, timbre, and speaking speed of the virtual voice according to the personal preferences of the first target user to achieve a more personalized virtual image generation and interactive experience; wherein the appearance characteristics include but are not limited to skin color, hairstyle, clothing, and accessories.

[0092] In some possible implementations, after generating a virtual voice of the first target user based on voice features, the method further includes: using emotion recognition technology to adjust the pitch, timbre, and speaking speed parameters of the virtual voice according to emotional changes in the voice of the first target user, so that the virtual voice is emotionally expressive.

[0093] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments.

[0094] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A personalized audio-visual fusion intelligent interactive system based on multimodal data, characterized in that: include: A multimodal data receiving module, used to receive multimodal data uploaded by a first target user; The multimodal data at least includes pictures, videos and audio files containing the portrait information and audio information of the first target user; A facial feature extraction module, configured to perform facial recognition and analysis on the first target user according to the multimodal data to obtain facial feature data of the first target user; the facial feature data at least includes a facial contour and a shape of facial features of the first target user; A three-dimensional model generation module, used to generate a corresponding three-dimensional facial model according to the extracted facial feature data of the first target user, and perform detail optimization and rendering to generate a virtual image of the first target user; An audio feature extraction module, configured to extract voice features of the first target user through the multimodal data; The speech features include at least pitch, timbre, and speech speed; A virtual voice generation module, configured to generate a virtual voice matching an original voice of the first target user based on the voice features of the first target user; A virtual image integration module, used to adjust and match the virtual image of the first target user with the virtual voice, so that the lip shape, expression and audio of the virtual image are synchronized in the video interaction; The interactive module is used to display the virtual image through an interactive interface, receive the conversation content input by the second target user through the interactive interface, generate reply content according to the conversation content through a preset intelligent algorithm, and use the virtual image integration module to integrate the reply content with the virtual image in the form of video and audio for vivid and natural interactive feedback.

2. The intelligent interactive system according to claim 1, characterized in that: The multimodal data receiving module also has data preprocessing functions, including but not limited to data compression, denoising, and format conversion, to improve data processing efficiency and accuracy.

3. The intelligent interactive system according to claim 2, characterized in that: The system further comprises: A personalized customization module is used to adjust the appearance characteristics of the virtual image and / or the pitch, timbre, and speaking speed of the virtual voice according to the personal preferences of the first target user to achieve a more personalized virtual image generation and interactive experience; wherein the appearance characteristics include but are not limited to skin color, hairstyle, clothing, and accessories.

4. The intelligent interactive system according to claim 3, characterized in that: The system further comprises: The emotion recognition and response module is used to use emotion recognition technology to adjust the pitch, timbre, and speech speed parameters of the virtual sound according to the emotional changes in the first target user's voice, so that the virtual sound is emotionally expressive.

5. A method for using a personalized audio-visual fusion intelligent interaction system based on multimodal data, applied to the personalized audio-visual fusion intelligent interaction system based on multimodal data according to any one of claims 1 to 4, characterized in that: include: The first target user uploads multimodal data through the interactive interface, wherein the multimodal data at least includes a picture, a video, and an audio file containing the portrait information and audio information of the first target user; Extracting facial features from the face of the first target user in the uploaded image and / or video; Generating a three-dimensional facial model according to the extracted facial features and optimizing the model to obtain a virtual image of the first target user; Extracting voice features from the uploaded audio and / or video to obtain voice features of the first target user; generating a virtual voice of the first target user based on the voice feature; Adjusting and matching the virtual model with the virtual sound; The virtual image is displayed, and input from a second target user is received, reply content is generated through a preset intelligent algorithm, and the reply content is interactively fed back through video and audio through the virtual image.

Images