Man-machine interaction system of high-real-time multi-mode unified model

Abstract

The invention relates to the technical field of man-machine interaction, and discloses a man-machine interaction system of a high-real-time multi-mode unified model, which comprises a video input module, an audio extraction module, a multi-mode processing module and a video output module, and is characterized in that the video input module receives and processes a video of a user and then inputs the video into the audio extraction module; the audio extraction module extracts audio data from a video and then sends the audio data to the multi-mode processing module, the multi-mode processing module directly generates a finally synthesized video through a unified model and then sends the finally synthesized video to the video output module, and the finally synthesized video is output through the video output module. According to the method, related functions are integrated through the multi-modal unified model, so that the problem that in an existing man-machine interaction system, the whole system possibly needs to be comprehensively tested and adjusted when one module is updated, and consequently the complexity of the system is increased is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a human-computer interaction system with a high real-time multi-modal unified model. Background Art

[0002] A human-computer interaction system is a system that enables information exchange between humans and computers through input and output devices in an effective way. It covers hardware (input devices such as keyboards, mice, touch screens, and handles, and output devices such as displays and speakers) and software (including operating systems, application interfaces, etc.), and uses various interactive technologies (such as speech recognition, gesture recognition, and eye tracking) to lower the threshold for interaction between users and computers, improve interaction efficiency and naturalness, and enable users to easily and accurately convey instructions to computers, while computers can clearly and intuitively provide feedback to users.

[0003] In existing human-computer interaction systems, speech recognition and text-to-speech modules usually rely on different technologies and algorithms. Updating one of the modules may require comprehensive testing and adjustment of the entire system, which will increase the complexity of the system. Summary of the invention

[0004] In order to make up for the above shortcomings, the present invention provides a human-computer interaction system with a high real-time multimodal unified model, aiming to improve the existing human-computer interaction system. Speech recognition and text-to-speech modules usually rely on different technologies and algorithms. Updating one of the modules may require comprehensive testing and adjustment of the entire system, which will increase the complexity of the system.

[0005] In the first aspect, the present invention provides the following technical solutions: a human-computer interaction system of a high real-time multimodal unified model, comprising a video input module, an audio extraction module, a multimodal processing module and a video output module, wherein the video input module receives and processes the user's video and inputs the video into the audio extraction module, the audio extraction module extracts audio data from the video and then sends the audio data to the multimodal processing module, the multimodal processing module directly generates a final synthesized video through a unified model, and then sends the final synthesized video to the video output module, and the final synthesized video is output through the video output module, and the multimodal processing module is used to integrate speech recognition, text processing, text-to-speech and video generation.

[0006] Preferably, the video input module can process video inputs of different formats, provide video data, and perform pre-processing on the video, including removing noise, adjusting brightness and contrast.

[0007] Preferably, the audio extraction module performs pre-processing on the audio through an efficient audio processing algorithm, including noise reduction, echo removal and sound quality enhancement.

[0008] Preferably, the multimodal processing module can directly process audio data through a deep learning algorithm to generate corresponding video content.

[0009] Preferably, the video output module can process the synchronization of video and audio, and can encode, compress and convert the format of the video.

[0010] Preferably, the video input module includes a video encryption unit and a video source verification unit, the video encryption unit is used to encrypt the received video input, and the video source verification unit is used to verify the legitimacy of the source of the video input; the audio extraction module includes an audio source positioning unit and an audio classification unit, the audio source positioning unit is used to determine the position information of the audio in the video while extracting the audio data, and the audio classification unit is used to classify the audio according to features.

[0011] Preferably, the multimodal processing module includes a semantic understanding enhancement unit, a sentiment analysis unit and a model updating unit, wherein the semantic understanding enhancement unit is used to adopt a semantic knowledge base and language rules to enhance the semantic understanding ability when processing audio and video data, the sentiment analysis unit is used to perform sentiment analysis on the input audio and video data, and the model updating unit is used to update and optimize the multimodal unified model; the video output module includes a video watermark adding unit and a video playback test unit, wherein the video watermark adding unit is used to add a video watermark when generating the final video output, and the video playback test unit is used to perform a playback test on the video before outputting the video.

[0012] In a second aspect, the present invention provides the following technical solution, a method for using a human-computer interaction system of a high real-time multimodal unified model comprises the following steps: S1. Video input: Input the video into the interactive system through the video input module; S2. Video preprocessing: The format conversion unit converts videos of different formats into a unified format, and then the video source verification unit verifies the legitimacy of the source of the video input. After the verification is completed, the video is encrypted by the video encryption unit; S3, audio extraction: The audio extraction module extracts audio data from the video; S4. Audio preprocessing and analysis: The audio is subjected to noise reduction, echo removal and sound quality enhancement processing through the audio extraction module, the audio features are further extracted through the audio feature extraction unit, the extracted audio quality is evaluated through the audio quality evaluation unit, the audio is classified according to the audio features through the audio classification unit to distinguish different types of speech, ambient sound and music, and the audio location information of the audio in the video is determined through the audio source positioning unit; S5. Semantic and sentiment analysis: The semantic understanding enhancement unit understands the semantics in the audio, uses the semantic knowledge base and language rules to process complex semantic situations, and uses the sentiment analysis unit to perform sentiment analysis on the input audio and video data; S6. Data fusion and processing: The data fusion unit is used to more effectively fuse audio and video data. The multimodal unified model processes the fused data through a deep learning algorithm and directly outputs the corresponding video content. The model update unit continuously updates and optimizes the multimodal unified model based on the system operation status and new data. S7. Video generation and adjustment: The processed data is integrated into the final video output through the video output module, the video format is adapted to different output devices and requirements through the video format adaptation unit, the video effect enhancement unit is used to enhance the video effect, and the video watermark adding unit is used to add a watermark to the video to identify the copyright information and system source; S8. Video quality check: The video playback test unit is used to perform a video playback test.

[0013] In the third aspect, the invention provides the following technical solution: a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for using the above-mentioned high real-time multimodal unified model human-computer interaction system when executing the computer program.

[0014] In a fourth aspect, the present invention provides the following technical solution: a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, a method for using the above-mentioned high real-time multimodal unified model human-computer interaction system is implemented.

[0015] The present invention has the following beneficial effects: 1. In the present invention, relevant functions are integrated through a multimodal unified model, and the internal connections are learned through end-to-end training based on a deep learning algorithm, so that speech recognition and text-to-speech become internal related parts of the model to avoid technical differences. Through standardized interfaces and data formats, common interfaces and standard data formats are defined for each module. For example, the output of text data by the speech recognition module must follow the standard. Therefore, when the module is updated, as long as the interface remains unchanged, it will not affect the operation of other modules, thereby improving the existing human-computer interaction system. Speech recognition and text-to-speech modules usually rely on different technologies and algorithms. Updating one of the modules may require comprehensive testing and adjustment of the entire system, which will increase the complexity of the system.

[0016] 2. In the present invention, relevant functions are integrated through a multimodal unified model to reduce the number of data conversion and transmission times between modules, thereby avoiding the delay caused by transferring data between modules in the traditional step-by-step processing method, thereby improving the problem of increased delay in conversion and data transmission between each step and reducing the system response speed, while being able to improve efficiency and simplify the process.

[0017] 3. In the present invention, by training a multimodal unified model, the functions of speech recognition, text processing, text-to-speech and audio-to-video generation are integrated into one model, which simplifies the system process, reduces intermediate links, reduces the independence and complexity of modules, and at the same time enhances the scalability and stability of the model, further reducing the system maintenance and update costs. By reducing intermediate links, the stability and reliability of the system are improved, and the possibility of failure is reduced, thereby improving the independence and complexity of each module, which increases the maintenance and update costs of the system, and enhancing robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a system architecture diagram of the human-computer interaction system with a high real-time multi-modal unified model proposed by the present invention; Figure 2 This is a method flow chart of the human-computer interaction system with high real-time multimodal unified model proposed by the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1 Reference Figure 1In the first embodiment of the present invention, the present invention provides a human-computer interaction system of a high real-time multimodal unified model, including a video input module, an audio extraction module, a multimodal processing module and a video output module. The video input module receives and processes the user's video and inputs the video into the audio extraction module. The audio extraction module extracts audio data from the video and then sends the audio data to the multimodal processing module. The multimodal processing module directly generates a final synthesized video through a unified model, and then sends the final synthesized video to the video output module, and outputs the final synthesized video through the video output module. The multimodal processing module is used to integrate speech recognition, text processing, text-to-speech and video generation.

[0021] Specifically, the video input module can receive and process the user's video input, perform video preprocessing, ensure the quality of the video data, and pass it to the audio extraction module. The audio extraction module can extract audio data from the video, providing a basis for subsequent multimodal processing. At the same time, audio preprocessing is performed to ensure the clarity and effectiveness of the audio data. The multimodal processing module can integrate speech recognition, text processing, text-to-speech and video generation functions, process audio data through deep learning algorithms and directly generate output videos, thereby improving system efficiency and stability.

[0022] The video input module can process video inputs of different formats, provide video data, and perform pre-processing on the video, including removing noise, adjusting brightness and contrast.

[0023] Specifically, by processing video inputs of different formats, compatibility and adaptability are ensured; by removing noise and adjusting brightness and contrast, clarity and stability of video data are ensured, thereby providing high-quality video data and ensuring the accuracy and effectiveness of subsequent processing. The video input module can quickly process video input and ensure that video data can be delivered to subsequent modules in a timely manner.

[0024] The audio extraction module pre-processes the audio through efficient audio processing algorithms, including noise reduction, echo removal and sound quality enhancement.

[0025] Specifically, efficient audio processing algorithms are used to ensure accurate extraction of audio data, and preprocessing work is used to ensure the clarity and effectiveness of audio data. The accuracy and quality of the audio extraction module directly affect the subsequent multimodal processing effect. Therefore, the design and implementation of this module are crucial. This module must be able to quickly extract audio data to ensure that the audio data can be delivered to subsequent modules in a timely manner.

[0026] The multimodal processing module can directly process audio data through deep learning algorithms and generate corresponding video content.

[0027] Specifically, the multimodal processing module needs to process multiple data types, including audio and video, to ensure effective conversion and integration between data, and to be trained with a large amount of multimodal data to ensure the accuracy and generalization ability of the multimodal processing module. The multimodal processing module uses a deep learning algorithm to ensure real-time processing of user input, thereby achieving rapid response, realizing efficient and accurate multimodal processing, reducing intermediate links, and improving the response speed and stability of the system.

[0028] The video output module can handle the synchronization of video and audio, and can encode, compress and convert the format of the video.

[0029] Specifically, it ensures the smoothness and consistency of the output video by processing the synchronization of video and audio, and adapts to different output requirements and devices by encoding, compressing and format converting the video.

[0030] The video input module includes a video encryption unit and a video source verification unit. The video encryption unit is used to encrypt the received video input, and the video source verification unit is used to verify the legitimacy of the source of the video input; the audio extraction module includes an audio source positioning unit and an audio classification unit. The audio source positioning unit is used to determine the position information of the audio in the video while extracting the audio data, and the audio classification unit is used to classify the audio according to the features.

[0031] Specifically, the video encryption unit is used to ensure the security and privacy of data and prevent video data from being illegally obtained or tampered with during transmission and processing. The video source verification unit is used to ensure that the system only processes videos from authorized channels and avoid malicious input that causes interference or security threats to the system. The audio source positioning unit is used to better combine video information in multimodal processing. The audio classification unit is used to facilitate the multimodal processing module to adopt more appropriate processing strategies for different types of audio.

[0032] The multimodal processing module includes a semantic understanding enhancement unit, a sentiment analysis unit and a model updating unit. The semantic understanding enhancement unit is used to enhance the semantic understanding ability by using a semantic knowledge base and language rules when processing audio and video data. The sentiment analysis unit is used to perform sentiment analysis on the input audio and video data. The model updating unit is used to update and optimize the multimodal unified model. The video output module includes a video watermark adding unit and a video playback test unit. The video watermark adding unit is used to add a video watermark when generating the final video output. The video playback test unit is used to perform a playback test on the video before outputting the video.

[0033] Specifically, the semantic understanding enhancement unit can be used to combine more semantic knowledge bases and language rules to improve the accuracy of processing complex semantic situations, especially when processing ambiguous or polysemous natural languages. The sentiment analysis unit can be used to judge the user's emotional state so that the system's response can better adapt to different emotional atmospheres and improve the humanization and accuracy of the interaction. The model updating unit can ensure that the accuracy and generalization ability of the model continue to improve to adapt to different application scenarios and user needs. The video watermark adding unit can identify the copyright information and system source of the video to protect the intellectual property rights of the system and the traceability of the output video. The video playback test unit can be used to check whether the video has problems such as freezes and audio and video asynchrony, and make timely adjustments and repairs to ensure that the output video quality meets the requirements.

[0034] Embodiment 2: Reference Figure 1 In a second embodiment of the present invention, the present invention provides a method for using a human-computer interaction system of a high real-time multimodal unified model, comprising the following steps: S1. Video input: Input the video into the interactive system through the video input module; S2. Video preprocessing: The format conversion unit converts videos of different formats into a unified format, and then the video source verification unit verifies the legitimacy of the source of the video input. After the verification is completed, the video is encrypted by the video encryption unit; S3, audio extraction: The audio extraction module extracts audio data from the video; S4. Audio preprocessing and analysis: The audio is subjected to noise reduction, echo removal and sound quality enhancement processing through the audio extraction module, the audio features are further extracted through the audio feature extraction unit, the extracted audio quality is evaluated through the audio quality evaluation unit, the audio is classified according to the audio features through the audio classification unit to distinguish different types of speech, ambient sound and music, and the audio location information of the audio in the video is determined through the audio source positioning unit; S5. Semantic and sentiment analysis: The semantic understanding enhancement unit understands the semantics in the audio, uses the semantic knowledge base and language rules to process complex semantic situations, and uses the sentiment analysis unit to perform sentiment analysis on the input audio and video data; S6. Data fusion and processing: The data fusion unit is used to more effectively fuse audio and video data. The multimodal unified model processes the fused data through a deep learning algorithm and directly outputs the corresponding video content. The model update unit continuously updates and optimizes the multimodal unified model based on the system operation status and new data. S7. Video generation and adjustment: The processed data is integrated into the final video output through the video output module, the video format is adapted to different output devices and requirements through the video format adaptation unit, the video effect enhancement unit is used to enhance the video effect, and the video watermark adding unit is used to add a watermark to the video to identify the copyright information and system source; S8. Video quality check: The video playback test unit is used to perform a video playback test.

[0035] Embodiment 3 The third embodiment of the present invention is based on the same inventive concept and proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for using the high real-time multimodal unified model human-computer interaction system of the above-mentioned embodiment.

[0036] Embodiment 4 The fourth embodiment of the present invention is based on the same inventive concept. The present invention proposes a computer device, the terminal includes: a processor, a memory; the processor and the memory communicate with each other; the memory is used to store instructions; the processor is used to execute the instructions in the memory, and execute the method for using the high real-time multimodal unified model human-computer interaction system of the above embodiment.

[0037] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0038] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A human-computer interaction system with a high real-time multi-modal unified model, characterized by: It includes a video input module, an audio extraction module, a multimodal processing module and a video output module. The video input module receives and processes the user's video and then inputs the video into the audio extraction module. The audio extraction module extracts audio data from the video and then sends the audio data to the multimodal processing module. The multimodal processing module directly generates the final synthesized video through a unified model, and then sends the final synthesized video to the video output module, and outputs the final synthesized video through the video output module. The multimodal processing module is used to integrate speech recognition, text processing, text-to-speech and video generation.

2. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized in that: The video input module can process video inputs in different formats, provide video data, and perform preprocessing on the video, including removing noise, adjusting brightness and contrast.

3. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized in that: The audio extraction module performs preprocessing on the audio through an efficient audio processing algorithm, including noise reduction, echo removal and sound quality enhancement.

4. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized by: The multimodal processing module can directly process audio data through a deep learning algorithm to generate corresponding video content.

5. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized in that: The video output module can process the synchronization of video and audio, and can encode, compress and convert the format of the video.

6. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized by: The video input module includes a video encryption unit and a video source verification unit, wherein the video encryption unit is used to encrypt the received video input, and the video source verification unit is used to verify the legitimacy of the source of the video input; The audio extraction module includes an audio source positioning unit and an audio classification unit. The audio source positioning unit is used to determine the position information of the audio in the video while extracting the audio data, and the audio classification unit is used to classify the audio according to the features.

7. The human-computer interaction system of the high real-time multi-modal unified model according to claim 1 is characterized by: The multimodal processing module includes a semantic understanding enhancement unit, a sentiment analysis unit and a model updating unit, wherein the semantic understanding enhancement unit is used to enhance the semantic understanding capability by using a semantic knowledge base and language rules when processing audio and video data, the sentiment analysis unit is used to perform sentiment analysis on the input audio and video data, and the model updating unit is used to update and optimize the multimodal unified model; The video output module includes a video watermark adding unit and a video playback test unit. The video watermark adding unit is used to add a video watermark when generating a final video output, and the video playback test unit is used to perform a playback test on the video before outputting the video.

8. A method for using a human-computer interaction system with a high real-time multi-modal unified model, characterized in that: A human-computer interaction system using the high real-time multimodal unified model according to any one of claims 1 to 7 comprises the following steps: S1. Video input: Input the video into the interactive system through the video input module; S2. Video preprocessing: The format conversion unit converts videos of different formats into a unified format, and then the video source verification unit verifies the legitimacy of the source of the video input. After the verification is completed, the video is encrypted by the video encryption unit; S3, audio extraction: The audio extraction module extracts audio data from the video; S4. Audio preprocessing and analysis: The audio is subjected to noise reduction, echo removal and sound quality enhancement processing through the audio extraction module, the audio features are further extracted through the audio feature extraction unit, the extracted audio quality is evaluated through the audio quality evaluation unit, the audio is classified according to the audio features through the audio classification unit to distinguish different types of speech, ambient sound and music, and the audio location information of the audio in the video is determined through the audio source positioning unit; S5. Semantic and sentiment analysis: The semantic understanding enhancement unit understands the semantics in the audio, uses the semantic knowledge base and language rules to process complex semantic situations, and uses the sentiment analysis unit to perform sentiment analysis on the input audio and video data; S6. Data fusion and processing: The data fusion unit is used to more effectively fuse audio and video data. The multimodal unified model processes the fused data through a deep learning algorithm and directly outputs the corresponding video content. The model update unit continuously updates and optimizes the multimodal unified model based on the system operation status and new data. S7. Video generation and adjustment: The processed data is integrated into the final video output through the video output module, the video format is adapted to different output devices and requirements through the video format adaptation unit, the video effect enhancement unit is used to enhance the video effect, and the video watermark adding unit is used to add a watermark to the video to identify the copyright information and system source; S8. Video quality check: The video playback test unit is used to perform a video playback test.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for using the human-computer interaction system of the high real-time multimodal unified model as claimed in claim 8 is implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for using the human-computer interaction system of the high real-time multimodal unified model as claimed in claim 8 is implemented.

Images