Intelligent video playing control system based on multi-mode interaction

Abstract

The invention discloses an intelligent video playing control system based on multi-mode interaction, and the system comprises four core modules: a voice instruction collection and preprocessing module which collects voice through a microphone array and extracts clear signals through adaptive filtering noise reduction and acoustic model endpoint detection; the multistage speech recognition and intention analysis module is used for converting speech into text by using a deep learning model, judging an instruction category in combination with an intention classification model, and extracting key parameters through a named entity recognition model; the context perception and instruction optimization module is used for maintaining a video playing state in real time based on a state machine model and eliminating instruction ambiguity in combination with a historical instruction; and the precise video playing control module calls a player API (Application Program Interface) to realize precise operations such as frame-level fast forward / fast backward, multi-speed adjustment, playing / pause switching and playing exit. The voice control accuracy and the control refinement degree can be remarkably improved, dependence on entity operation components is not needed, and the method is suitable for various devices such as mobile phones, televisions and vehicle-mounted systems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and human-computer interaction, and particularly relates to an intelligent video playing control system based on multi-modal interaction. BACKGROUND

[0002] With the increasing richness of multimedia content, users have higher requirements for the interactive experience of video players. Traditional video players mainly rely on mouse, keyboard and touch screen for operation, and have problems such as low interactive efficiency, non-intuitive operation, strict hardware condition limitation and the like.

[0003] The existing voice control scheme has three major defects: Firstly, the recognition accuracy is low, the ability to distinguish similar semantic instructions is poor, and misrecognition is easy to occur; Secondly, the control granularity is single, and usually only simple instructions such as playing and pausing can be recognized, and there is a lack of precise control over complex instructions such as fast forward / fast backward and speed playing; Thirdly, the context understanding ability is weak, and the specific meaning of the instruction under different playing states cannot be accurately understood. Therefore, there is an urgent need for a system that can realize accurate voice recognition, understand complex instruction intent and perform efficient video playing control. SUMMARY

[0004] The present application aims to provide an intelligent video playing control system based on multi-modal interaction, which aims to solve the problems of low recognition accuracy, single control granularity and weak context understanding ability in the prior art voice control video playing, and to provide an intelligent video playing control system based on multi-modal interaction which is efficient in interaction, accurate in control and suitable for various playing scenarios.

[0005] To solve the above technical problems, the present application provides an intelligent video playing control system based on multi-modal interaction, which is composed of four core modules, and the functions and technical implementations of each module are as follows: 1. Voice instruction acquisition and preprocessing module function: The voice instructions of the user are collected through a microphone array, the collected voice signals are subjected to noise reduction, echo cancellation and endpoint detection, and clear voice signals are extracted to provide high-quality input for subsequent recognition. Technical implementation: adaptive filtering algorithm is used for noise reduction processing to effectively remove environmental noise; acoustic model is used for endpoint detection to accurately locate the start and end positions of the voice signal and eliminate invalid signals.

[0006] 2. Multi-level voice recognition and intent analysis module function: The preprocessed speech signal is recognized, converted into text, and the specific intent of the command is deeply analyzed to clarify the command category and corresponding key parameters. Technical implementation: The first level is front-end speech recognition, using a deep learning-based model to convert the speech signal into text; the second level is intent classification and slot filling. An intent classification model determines the command category, including time control, speed control, playback control, and exit playback control, while a named entity recognition model extracts key parameters from the command, such as time parameters and speed parameters.

[0007] 3. Context awareness and instruction optimization module functions: By combining the current video playback status and historical command records, the parsed commands are optimized and adjusted to eliminate ambiguity and ensure that the command intent aligns with user needs. Technical implementation: A state machine model is constructed to maintain the current state of the video player in real time, including normal playback, playback speed adjustment, pause, etc. When parsing commands, the current playback state is input into the model as contextual information to verify and correct the command intent.

[0008] 4. Precise video playback control module functions: Based on the optimized instructions, the corresponding API of the video player is called to execute precise video playback control operations. Technical implementation: This module includes several sub-functions: fast forward / rewind control, playback speed control, playback status control, and exit playback control. Fast forward / rewind control achieves precise frame-level skipping based on the parsed time parameters; playback speed control adjusts the playback rate based on the speed parameter; playback status control switches between play and pause states based on instructions; and exit playback control closes the current video based on instructions.

[0009] In summary, due to the adoption of the above-mentioned technologies, the beneficial effects of this invention are: 1. Significantly improved recognition accuracy: Through multi-level speech recognition and intent parsing mechanisms, combined with deep learning models, it effectively distinguishes semantically similar instructions, greatly reducing the false recognition rate.

[0010] 2. Rich and diverse control granularity: Supports various command types such as fast forward / rewind, speed adjustment, play / pause, and exit playback, meeting users' needs for fine-grained control over video playback.

[0011] 3. Strong contextual understanding: It maintains the playback state in real time through a state machine model, optimizes instructions by combining historical instructions, accurately understands the specific meaning of instructions in different scenarios, and eliminates ambiguity.

[0012] 4. Efficient and convenient interaction: Without relying on mouse, keyboard or touch screen operation, users can accurately control video playback through voice. The operation is intuitive and responsive, improving the user interaction experience.

[0013] 5. Strong compatibility: Supports multiple playback kernels such as ExoPlayer and MediaPlayer, adapts to different video playback platforms and devices, and has a wide range of applications. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention, making other features, objects, and advantages of the invention more apparent. The illustrative embodiments of the invention illustrated in the drawings and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a system diagram of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] In the description of this invention, it should be understood that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing the invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0017] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific context of the specification.

[0018] This invention provides an intelligent video playback control system based on multimodal interaction.

[0019] Voice command acquisition and preprocessing: The user issues a voice command through the microphone, the system acquires the voice signal through the microphone array, uses an adaptive filtering algorithm to remove environmental noise, uses an acoustic model to perform endpoint detection, and extracts a clear voice signal.

[0020] Multi-level speech recognition and intent parsing: The front-end speech recognition model converts the preprocessed speech signal into text; the intent classification model determines that the instruction category is time control; the named entity recognition model extracts the action fast forward and the time parameter of ten seconds.

[0021] Context awareness and instruction optimization: The state machine model detects that the current video is in a normal playback state. Combined with this context information, it confirms that the instruction intent is clear and no optimization adjustment is required.

[0022] Precise video playback control: The system calls the video player API to obtain the current video playback progress, moves the playback progress pointer forward by the corresponding number of frames ten seconds, achieves precise fast forward at the frame level, and maintains normal playback.

[0023] Voice command acquisition and preprocessing: When the video is playing at double speed, the user issues a pause command through the microphone. The microphone array acquires the voice signal, which is then denoised by an adaptive filtering algorithm and subjected to acoustic model endpoint detection to obtain a clear voice signal.

[0024] Multi-level speech recognition and intent parsing: The front-end speech recognition model converts the speech signal into text pause; the intent classification model determines the instruction category as playback control; since the instruction has no additional parameters, the slot filling step did not extract the key parameters.

[0025] Context awareness and instruction optimization: The state machine model maintains the playback state in real time as double speed playback. The system combines this context information to clarify that the instruction intent is to pause the current double speed playback state, rather than pausing the video itself, thus completing instruction optimization.

[0026] Precise video playback control: The system calls the video player API to perform a pause operation, switching the video playback status from double speed playback to paused state, while retaining the current playback progress and double speed setting so that the original speed can be maintained when the user resumes playback later.

[0027] Voice command acquisition and preprocessing: The video is currently paused. The user issues a one-minute rewind command through the microphone. After the microphone array acquires the voice signal, it is processed by noise reduction, echo cancellation and endpoint detection to extract the valid voice signal.

[0028] Multi-level speech recognition and intent parsing: The front-end speech recognition model converts speech signals into text; the intent classification model determines the instruction category as time control; the named entity recognition model extracts the action rewind and the time parameter of one minute, and converts them into a unified time unit of six thousand milliseconds.

[0029] Context awareness and instruction optimization: The state machine model detects that the current video is paused. Based on this context information, the system determines that the user's intention is to rewind for a specified time and then resume playback, and optimizes and supplements the instruction accordingly.

[0030] Precise video playback control: The system calls the video player API to obtain the current playback progress, calculates the rewind target position, and performs boundary protection to ensure that the target position is not less than zero; after moving the playback progress pointer to the target position, the playback operation is automatically executed to switch the video from paused state to normal playback state.

[0031] The three embodiments described above cover different instruction control scenarios under three typical playback states: normal playback, accelerated playback, and pause, comprehensively demonstrating the core functions and technical advantages of the present invention. The time and acceleration parameters involved in the embodiments can be flexibly adjusted according to the user's actual needs, and the system can achieve precise control through multi-level recognition and context optimization. Furthermore, the system supports adaptation to multiple playback kernels and can run stably on different hardware devices and operating systems, further verifying the practicality and compatibility of the present invention.

[0032] To further demonstrate the system's practicality and robustness in complex scenarios, abnormal boundaries, and continuous interactions, the following embodiments are included:

[0033] Voice command acquisition and preprocessing: The user is currently watching a video at "1.5x speed playback" and issues a "fast forward 30 seconds" command via microphone. The system acquires the voice signal through a microphone array, uses an adaptive filtering algorithm to remove background noise from the TV video playback, and then uses an acoustic model to locate the start ("fast") and end ("second") endpoints of the voice signal to extract a clean voice signal free from noise interference.

[0034] The multi-level speech recognition and intent parsing front-end deep learning model converts the speech signal into the text "fast forward 30 seconds"; the intent classification model combines the instruction features (including the "fast forward" action and the "30 seconds" time parameter) to determine the instruction category as "time control"; the named entity recognition model extracts key information—the action "fast forward" and the time parameter "30 seconds" (automatically converted to 30,000 milliseconds, a unified time unit for easy subsequent calculation).

[0035] The context-aware and instruction-optimized state machine model maintains the current playback state as "1.5x speed playback" in real time. The system combines this context to determine that the core of the user instruction is "time progress adjustment" rather than "speed switching". Fast forward should be executed while maintaining 1.5x speed to avoid misinterpreting "fast forward" as "speed increase" (such as misjudging it as "2x speed"). This confirms that the instruction intent is unambiguous and no additional optimization is needed.

[0036] The precise video playback control system calls the player API to first obtain the current playback progress (assuming it is 120,000 milliseconds, or 2 minutes), and calculates the target progress as 120,000 + 30,000 = 150,000 milliseconds (2 minutes and 30 seconds). The progress pointer is positioned at 150,000 milliseconds through the frame-level jump interface, while retaining the "1.5x speed" playback parameter. After fast-forwarding, playback continues at 1.5x speed without requiring secondary speed adjustments from the user.

[0037] Voice command acquisition and preprocessing (first round of commands): The user's initial state is "video paused," and they issue a "play" command. After the system acquires the voice signal, it performs noise reduction and endpoint detection processing to extract the core voice segment for "play," with no redundant interference signals.

[0038] The multi-level speech recognition and intent parsing (first round of instructions) front-end model converts speech into the text "play"; the intent classification model determines the instruction category as "play control", and since there are no additional parameters, the slot filling result is "no parameters"; the system outputs the intent result "execute playback operation".

[0039] Context awareness and instruction optimization (first round of instructions): The state machine model detects that the current state is "paused", confirms that the "play" instruction is intended to "resume video playback", which is unambiguous, and executes it directly.

[0040] Precise video playback control (first round of instructions) calls the player's "play()" interface, and the video starts playing normally at "1x speed" from the paused progress (assuming it is 80000 milliseconds, i.e., 1 minute and 20 seconds).

[0041] Voice command acquisition and preprocessing (second round of commands, 2-second interval): After the user watches for 5 seconds, they issue the command "Switch to 2x speed". The system repeats the acquisition-noise reduction-endpoint detection process to extract a clear voice signal.

[0042] The multi-level speech recognition and intent parsing (second round of instructions) front-end model converts the text into "switch to 2x speed"; the intent classification model determines the category as "speed control" based on the keywords "switch" and "speed"; the named entity recognition model extracts the speed parameter "2x speed".

[0043] In addition to recording the current "normal playback" state, the context-aware and instruction optimization (second round of instructions) state machine model also stores historical instructions (the "play" instruction was executed 2 seconds ago, restoring 1x speed). The system combines the context of "historical speed is 1x speed, and the current speed needs to be switched to 2x speed" to confirm that the instruction intent is "speed increase" without having to ask the user (e.g., there is no need to confirm whether to "switch from the current speed" or "switch from 1x speed by default").

[0044] Precise video playback control (second round of instructions) calls the player's "setPlaybackSpeed(2.0f)" interface to adjust the playback speed from 1x to 2x while maintaining the current playback progress (80000+5000=85000 milliseconds, i.e. 1 minute and 25 seconds), achieving seamless connection of continuous instructions.

[0045] Voice command acquisition and preprocessing: The user is currently watching a 5-minute (300,000 milliseconds) video, and the current progress is 280,000 milliseconds (4 minutes and 40 seconds). The user issues the command "Fast forward 30 seconds". After the system acquires the voice signal, it preprocesses it to obtain the clear text "Fast forward 30 seconds".

[0046] The result of multi-level speech recognition and intent parsing is "Time Control - Fast Forward - 30 seconds (30000 milliseconds)", and the parameter extraction is correct.

[0047] The context-aware and instruction optimization state machine model obtains the current total video duration as 300,000 milliseconds and calculates the target progress as 280,000 + 30,000 = 310,000 milliseconds, which exceeds the total duration (300,000 milliseconds). The system determines that this instruction has a potential "parameter out of bounds" problem and needs to optimize the target progress to the end of the video to avoid the player reporting an error due to "progress out of range".

[0048] When the precise video playback control system calls the player API, it automatically triggers boundary protection logic: corrects the target progress to "total duration - 1 millisecond" (i.e., 299999 milliseconds to avoid triggering the "playback ended" pop-up due to "equal to total duration"), and performs a fast-forward operation; after fast-forwarding, it maintains the current playback state (e.g., 1x speed), and at the same time confirms through the player callback interface that the progress has reached "4 minutes 59 seconds 999 milliseconds" to ensure no abnormal errors and improve user experience.

Claims

1. An intelligent video playback control system based on multimodal interaction, characterized in that: include: The system includes a voice command acquisition and preprocessing module, a multi-level voice recognition and intent parsing module, a context awareness and command optimization module, and a precise video playback control module. The voice command acquisition and preprocessing module is used to acquire user voice commands and optimize the signal; the multi-level voice recognition and intent parsing module is used to convert the optimized voice signal into text and parse the command intent and key parameters; the context awareness and command optimization module is used to eliminate command ambiguity by combining video playback status and historical commands; the precise video playback control module is used to achieve precise control of video playback based on the optimized commands.

2. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, Includes the following steps: S1 collects user voice commands and performs noise reduction, echo cancellation, and endpoint detection processing. S2 converts the processed speech signal into text, identifies the command category, and extracts key parameters; S3 optimizes the parsed instructions by combining the current video playback status and historical instructions; S4 executes the corresponding video playback control operation based on the optimized instructions.

3. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, The voice command acquisition and preprocessing module uses a microphone array to acquire voice commands, implements noise reduction through an adaptive filtering algorithm, and completes endpoint detection using an acoustic model.

4. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, The multi-level speech recognition and intent parsing module uses a deep learning model to convert speech signals into text, employs an intent classification model to determine the instruction category, and extracts key parameters through a named entity recognition model.

5. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, The instruction categories include time control, speed control, playback control, and exit playback control, and the key parameters include time parameters and speed parameters.

6. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, The context awareness and instruction optimization module maintains the video playback state in real time through a state machine model. The playback state includes normal playback, speed-up playback, and pause.

7. The intelligent video playback control system based on multimodal interaction according to claim 1, characterized in that, The precise video playback control module implements frame-level fast forward / rewind control, playback speed adjustment, state switching play / pause control, and video exit control by calling the video player API.

8. The intelligent video playback control system based on multimodal interaction according to claim 2, characterized in that, In S2, instruction category recognition and key parameter extraction adopt a hierarchical processing method, first completing the conversion from speech to text, and then performing intent classification and slot filling.

9. The intelligent video playback control system based on multimodal interaction according to claim 2, characterized in that, In S3, when an ambiguous instruction is detected, the intent of the instruction is corrected by combining the current playback status and historical instruction records.

10. The intelligent video playback control system based on multimodal interaction according to claim 2, characterized in that, In S4, when executing fast forward / rewind control, boundary protection is performed on the target playback position to ensure that it does not exceed the start and end range of the video.

Images