Audio-video player control method based on voice instruction

Abstract

The application relates to the technical field of audio and video control, and discloses an audio and video player control method based on a voice instruction. The method comprises the following steps: collecting original voice instruction streams of a user, the instruction streams containing a time domain audio signal sequence, environmental noise spectrum and user pronunciation characteristic parameters, so that voice information can be comprehensively captured; performing multi-modal instruction analysis processing on the original voice instruction streams, generating a structured control instruction set containing an acoustic control intention identifier, a semantic operation object description and context association parameters, and improving analysis accuracy; then performing player state adaptation based on the set, generating a dynamic control response sequence containing device state adjustment commands, media content positioning parameters and interface interaction logic identifiers, driving the player to perform multi-dimensional control operations and generating real-time playing control effect feedback data; and finally optimizing multi-modal analysis parameters according to the feedback data, generating an adaptive instruction analysis strategy, and optimizing the control experience of the user on the audio and video player.

Description

Technical Field

[0001] This invention relates to the field of audio and video control technology, specifically to a voice command-based audio and video player control method. Background Technology

[0002] With the widespread adoption of smart devices, audio and video players have become essential tools for daily entertainment and information access, and their control methods have evolved from traditional physical buttons and touchscreen operations to voice control. Voice control, with its convenience of eliminating manual operation, has demonstrated significant application value in in-vehicle scenarios, smart home scenarios, and handheld device usage scenarios that free up users' hands. However, current mainstream audio and video player voice control solutions still have several technical limitations, making it difficult to meet users' diverse and highly stable control needs.

[0003] In the initial voice command acquisition stage, most existing technologies focus only on acquiring the time-domain audio signal sequence, neglecting the impact of environmental noise spectrum and user pronunciation feature parameters on subsequent command parsing. In complex environments such as homes and vehicles, environmental noise (such as television sound, engine noise, and external human voices) can superimpose with user voice commands, leading to a decrease in the signal-to-noise ratio of the acquired voice signal. Furthermore, different users have different pronunciation habits (such as speaking speed, pitch, and regional accents), and these differences in pronunciation feature parameters further increase the difficulty of command parsing, making subsequent parsing processes prone to misjudgment or omission.

[0004] In terms of command parsing and processing, existing technologies mostly adopt a single-modal parsing approach, such as relying solely on semantic analysis of speech signals for command recognition, lacking comprehensive utilization of acoustic features and contextual information. This single-modal parsing approach struggles to accurately capture the user's true control intent: when the user's voice command is semantically ambiguous (e.g., "faster" could mean either increasing playback speed or increasing volume), semantic analysis alone cannot determine the specific operation target; it also cannot combine the player's historical playback status (e.g., whether it was previously paused, the type of media currently being played) for contextual association, resulting in a mismatch between the parsed command and the actual playback requirements, thus affecting the control effect.

[0005] At the level of control response and adaptive optimization, existing player state adaptation operations are often relatively fixed, and the generated control response sequences lack flexibility, making it difficult to dynamically adjust according to different playback scenarios and user needs. For example, users' needs for "fast forward" differ between movie playback and music playback scenarios, but existing technologies typically use a uniform fast forward parameter, failing to meet scenario-specific requirements. Furthermore, existing technologies lack parameter optimization mechanisms based on control effect feedback data, making it impossible to self-adjust based on parsing errors that occur during historical control processes (such as mistakenly identifying "pause" as "play"). This results in difficulty improving command parsing accuracy over long-term use, leading to a poor user experience. Summary of the Invention

[0006] The purpose of this invention is to provide a voice command-based audio and video player control method to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a voice command-based audio and video player control method, the method comprising:

[0008] The system collects the raw voice command stream issued by the user, which includes a time-domain audio signal sequence, an environmental noise spectrum, and pronunciation feature parameters.

[0009] The original voice command stream is subjected to multimodal command parsing processing to generate a structured control command set, which includes acoustic control intent identifiers, semantic operation object descriptions, and context association parameters.

[0010] Based on the structured control instruction set, the player state adaptation operation is executed to generate a dynamic control response sequence, which includes device state adjustment commands, media content positioning parameters, and interface interaction logic identifiers.

[0011] Based on the dynamic control response sequence, the audio and video player is driven to perform multi-dimensional control operations, generating real-time playback control effect feedback data;

[0012] Based on the real-time playback control effect feedback data, the parameters of the multimodal instruction parsing and processing are optimized and adjusted to generate an adaptive instruction parsing strategy.

[0013] Preferably, the step of performing multimodal instruction parsing processing on the original voice instruction stream to generate a structured control instruction set includes:

[0014] The time-domain audio signal sequence is subjected to acoustic feature extraction processing to generate acoustic fingerprint feature vectors and phoneme segmentation boundary data;

[0015] The acoustic fingerprint feature vector is subjected to noise robustness enhancement processing based on the ambient noise spectrum to generate an enhanced acoustic feature representation;

[0016] The enhanced acoustic feature representation is subjected to deep semantic decoding to generate initial semantic parsing results and semantic confidence scores;

[0017] The initial semantic parsing results are subjected to dialect-adaptive correction using the pronunciation feature parameters to generate a standardized semantic operation object description;

[0018] By integrating the phoneme segmentation boundary data, semantic confidence scores, and standardized semantic operation object descriptions, the acoustic control intent identifier, semantic operation object description, and context association parameters are generated.

[0019] Preferably, the step of performing acoustic feature extraction processing on the time-domain audio signal sequence to generate acoustic fingerprint feature vectors and phoneme segmentation boundary data includes:

[0020] The time-domain audio signal sequence is converted into a Mel frequency spectrum coefficient sequence to generate a time-frequency feature representation matrix;

[0021] The time-frequency feature representation matrix is ​​subjected to dynamic time warping to generate a warped time-frequency feature sequence;

[0022] Extract the formant trajectory features and fundamental frequency variation curve from the normalized time-frequency feature sequence to generate an acoustic fingerprint feature vector;

[0023] The regularized time-frequency feature sequence is segmented at the phoneme level based on the Hidden Markov Model to generate phoneme segmentation boundary data and phoneme category identifiers.

[0024] Preferably, the step of performing deep semantic decoding on the enhanced acoustic feature representation to generate an initial semantic parsing result and a semantic confidence score includes:

[0025] The enhanced acoustic feature representation is input into a bidirectional long short-term memory neural network for context encoding to generate a context-aware feature representation.

[0026] The context-aware feature representation is processed by an attention mechanism to focus key information, generating an attention-weighted feature vector.

[0027] The attention-weighted feature vector is sequence-to-sequence mapping processed using a connectionist temporal classification decoder to generate initial semantic parsing results.

[0028] The semantic confidence score of the initial semantic parsing result is calculated based on the softmax classifier.

[0029] Preferably, the step of performing player state adaptation operation based on the structured control instruction set to generate a dynamic control response sequence includes:

[0030] The basic operation type identifier and operation intensity parameters are obtained by parsing the acoustic control intent identifier;

[0031] Based on the semantic operation object description, the media resource database is retrieved to generate target media resource identifiers and resource location information;

[0032] Calculate the player state transition path based on the context association parameters, and generate state transition constraints.

[0033] By integrating the basic operation type identifier, operation intensity parameter, target media resource identifier, and state transition constraints, a device state adjustment command, media content positioning parameter, and interface interaction logic identifier are generated.

[0034] Preferably, the step of calculating the player state transition path and generating state transition constraints by combining the context association parameters includes:

[0035] Extract the historical operation sequence records and current device status snapshot from the context association parameters;

[0036] Based on the state machine model, the historical operation sequence records are processed by pattern mining to generate a set of common state transition patterns.

[0037] A state transition probability distribution is generated based on the degree of matching between the current device state snapshot and a set of common state transition patterns.

[0038] The state transition probability distribution is optimized and filtered based on the transition cost function to generate state transition constraints.

[0039] Preferably, the step of driving the audio / video player to perform multi-dimensional control operations based on the dynamic control response sequence and generating real-time playback control effect feedback data includes:

[0040] The volume adjustment parameters, playback rate parameters, and channel balance parameters are obtained by parsing the device status adjustment command.

[0041] Perform media file location operation based on the media content location parameters to generate accurate timestamp location data;

[0042] Based on the interface interaction logic identifier, control the user interface element update operation and generate an interface rendering instruction sequence.

[0043] The volume adjustment parameters, playback rate parameters, channel balance parameters, precise timestamp positioning data, and interface rendering instruction sequence are executed synchronously to generate real-time playback control effect feedback data.

[0044] Preferably, after generating the real-time playback control effect feedback data, the method further includes:

[0045] Collect user's subsequent interaction behavior data to generate behavioral feedback feature vectors;

[0046] By comparing the deviation between the behavioral feedback feature vector and the expected control effect, a control accuracy evaluation index is generated.

[0047] The parameter configuration of the multimodal instruction parsing and processing is updated based on the control accuracy evaluation index to generate optimized parsing model parameters.

[0048] Preferably, updating the parameter configuration of the multimodal command parsing and processing based on the control accuracy evaluation index to generate optimized parsing model parameters includes:

[0049] Calculate the acoustic model adaptation coefficient and the language model correction amount based on the control accuracy evaluation index;

[0050] The acoustic model adaptation coefficients are optimized by gradient descent to generate updated acoustic feature extraction parameters.

[0051] The semantic decoding dictionary weights are adjusted using the language model correction values ​​to generate optimized language model parameters.

[0052] By combining the updated acoustic feature extraction parameters and the optimized language model parameters, optimized analytical model parameters are generated.

[0053] Preferably, after generating the optimized analytical model parameters, the method further includes:

[0054] The optimized parsed model parameters are deployed to an online learning framework for incremental learning processing.

[0055] The execution logic of the adaptive instruction parsing strategy is updated based on the incremental learning results;

[0056] The raw speech command stream of subsequent inputs is processed according to the updated adaptive instruction parsing strategy.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] In the initial voice command acquisition stage, this method simultaneously acquires time-domain audio signal sequences, environmental noise spectra, and pronunciation feature parameters. Compared to existing schemes that only acquire a single audio signal, it can obtain more comprehensive information related to voice commands. The acquisition of the environmental noise spectrum provides a data foundation for noise suppression in the subsequent command parsing process. By analyzing the noise spectrum, environmental interference can be filtered in a targeted manner, reducing the impact of noise on the voice command signal, improving the signal-to-noise ratio of the voice signal, and thus reducing parsing errors caused by noise interference. The acquisition of pronunciation feature parameters can adapt to the differences in pronunciation habits among different users. By recording and analyzing the user's speech rate, pitch, accent, and other characteristics, the subsequent parsing process is more closely aligned with the individual user's pronunciation characteristics, avoiding misrecognition of commands due to differences in user pronunciation. This is particularly suitable for scenarios where multiple users share a player in a home setting, meeting the usage needs of different users.

[0059] In the multimodal command parsing and processing stage, this method overcomes the limitations of existing single-modal parsing. It comprehensively utilizes acoustic control intent identifiers, semantic operation object descriptions, and contextual parameters to generate a structured set of control commands, significantly improving the accuracy of command parsing and the precision of intent capture. By extracting acoustic control intent identifiers, the acoustic features of the speech signal (such as volume changes and intonation fluctuations) can be used to help determine the user's control intent. For example, when a user raises the volume to issue a "pause" command, the acoustic features can be combined to further confirm the urgency of the command and avoid confusion with other semantically similar commands. The clear description of the semantic operation object can effectively solve the problem of semantic ambiguity in commands. When a user issues an ambiguous command, semantic analysis can be used to determine the specific operation object (such as clarifying "faster" as "increase playback speed" or "increase volume"). The introduction of contextual parameters can be combined with the player's historical playback status (such as the current media type and historical operation records) for comprehensive judgment. For example, when a user issues a "start" command in a paused state, the context can be used to determine that it is a "play" operation, rather than an operation such as "start recording" in other scenarios, ensuring that the parsed command is highly matched with the user's actual needs.

[0060] In the player state adaptation and control response stage, this method generates a dynamic control response sequence based on a structured set of control commands. This allows for flexible adjustment of device state adjustment commands, media content positioning parameters, and interface interaction logic identifiers according to different playback scenarios, media types, and user needs, achieving scenario-based and personalized control responses. For example, when playing a movie, if the user issues a "fast forward" command, the system can generate positioning parameters that jump by the minute based on the movie's playback duration and scene characteristics; when playing music, if the user issues a "fast forward" command, it generates positioning parameters that jump by the second, meeting the user control needs under different media types. Simultaneously, the dynamic adjustment of the interface interaction logic identifiers allows the player interface to display corresponding prompts when performing control operations (such as pause or fast forward) (e.g., displaying a "paused" icon when paused, and a progress bar animation when fast forwarded), enhancing the user's interactive perception with the player and improving operational convenience.

[0061] At the adaptive optimization level, this method optimizes and adjusts the parameters of multimodal command parsing based on real-time playback control effect feedback data, generating an adaptive command parsing strategy that enables the system to continuously self-optimize. After each control operation, the system collects control effect feedback data (such as whether the user corrected the previous control command through a second operation, and whether the parsed command is consistent with the actual execution result), and adjusts the parameters in the command parsing process (such as noise filtering threshold, semantic matching weight, and context association coefficient) based on this data. For example, if "reduce volume" is repeatedly misidentified as "reduce fast forward," the system can adjust the matching weight of the semantic analysis module through feedback data to enhance the distinction between "volume" related words and "playback speed" related words; if the parsing accuracy decreases under specific environmental noise, the system can adjust the noise filtering parameters to improve the signal-to-noise ratio of the speech signal in that environment. Through this continuous parameter optimization, the system's command parsing accuracy will gradually improve with the increase in usage, maintaining a stable control effect throughout long-term use, avoiding a decline in control experience due to environmental changes or changes in user habits, and further enhancing users' trust and reliance on the voice control function of the audio and video player. Attached Figure Description

[0062] Figure 1 This is a schematic diagram illustrating the working principle of the voice command-based audio and video player control method described in this invention.

[0063] Figure 2 A flowchart for multimodal instruction parsing and processing;

[0064] Figure 3 This is a flowchart of deep semantic decoding processing. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Please see Figure 1 This invention provides a voice command-based audio / video player control method. The method includes: acquiring a raw voice command stream containing a time-domain audio signal sequence, environmental noise spectrum, and pronunciation feature parameters; performing multimodal command parsing processing on this stream to generate a structured control command set, which includes acoustic control intent identifiers, semantic operation object descriptions, and context-related parameters. Based on the structured control command set, a player state adaptation operation is performed to generate a dynamic control response sequence containing device state adjustment commands, media content positioning parameters, and interface interaction logic identifiers. This dynamic control response sequence is used to drive the audio / video player to perform multi-dimensional control operations and generate real-time playback control effect feedback data. The system uses the real-time playback control effect feedback data to optimize and adjust the parameters of the multimodal command parsing processing, generating an adaptive command parsing strategy, thereby achieving continuous improvement in system performance.

[0067] Example 1: See Figure 2 The process of performing multimodal instruction parsing on the raw speech command stream to generate a structured control command set constitutes the core of the entire control method. Its processing quality directly determines the accuracy and smoothness of subsequent control actions. The implementation of this process begins with the fine deconstruction and feature mining of the input raw speech command stream. The raw speech command stream is not a single audio data, but a composite data stream containing a time-domain audio signal sequence, an environmental noise spectrum, and pronunciation feature parameters. The time-domain audio signal sequence is the most direct carrier of the speech command. It records the original fluctuation of the sound wave with the amplitude values ​​at discrete time points. The environmental noise spectrum is obtained by parallel acquisition or analysis of background sound. It describes the distribution of noise energy in the command acquisition environment at different frequencies. The pronunciation feature parameters may come from user registration information or real-time analysis. It contains characteristics related to the individual speaker, such as fundamental frequency range, formant preference, and even regional accent characteristics.

[0068] Acoustic feature extraction processing of time-domain audio signal sequences aims to extract feature representations that effectively characterize speech content and are robust to irrelevant variations from the original waveform vibrations. This process first transforms the continuous time-domain audio signal sequence into a Mel frequency spectrum coefficient sequence that better reflects auditory perception characteristics. The transformation process simulates the nonlinear response of the human cochlea to sounds of different frequencies. A set of triangular filters distributed according to the Mel scale is used to filter the short-time power spectrum of the signal. Then, the logarithm of the filter output energy is taken and a discrete cosine transform is performed to compress the data and decorrelate it. The resulting Mel frequency spectrum coefficient sequence constitutes a time-frequency feature representation matrix. The horizontal axis of this matrix represents the time frame, and the vertical axis represents the Mel frequency cepstral coefficient dimension. However, due to differences in the user's pronunciation rate, directly comparing the time-frequency feature representation matrices of different commands introduces distortion in the time dimension. Therefore, dynamic time warping of the matrix is ​​required. This algorithm finds an optimal nonlinear bending path to align the test feature sequence with a reference template sequence on the time axis, thereby eliminating the influence of pronunciation speed and generating a more regularized time-frequency feature sequence in terms of time scale.

[0069] From the regularized time-frequency feature sequence, it is necessary to further extract more discriminative acoustic fingerprint feature vectors. These vectors aim to uniquely or with high recognizability represent specific speech segments. The extraction work focuses on relatively stable individual features in the speech signal, such as formant trajectory features and fundamental frequency variation curves. Formants are spectral peaks generated by vocal tract resonance; their frequency positions and directions are closely related to the shape and movement of the vocal organs. These formants can be tracked over time using methods such as linear predictive coding or cepstral analysis. The fundamental frequency variation curve reflects the frequency of vocal cord vibration, constituting the tone information of speech. The fundamental frequency value for each time frame can be estimated using autocorrelation methods or algorithms combined with harmonic structures. These trajectories and curves... By concatenating features frame by frame or calculating their statistics, a high-dimensional acoustic fingerprint feature vector is formed. At the same time, in order to understand the temporal structure of speech, phoneme-level segmentation processing is required on the normalized time-frequency feature sequence based on the Hidden Markov Model (HMM). The HMM treats the speech signal as a process of generating an observable feature sequence from a sequence of hidden states. Each phoneme is usually modeled as a chain of multiple states. By training the HMM parameters in advance using a large amount of labeled data, and then using the Viterbi decoding algorithm to find the most likely hidden state sequence on the given observation sequence, the start and end positions of each phoneme are determined, phoneme segmentation boundary data is generated, and a phoneme category label is assigned to each segmented segment.

[0070] While acoustic fingerprint feature vectors are representative, they inevitably become mixed with environmental noise during actual acquisition. Therefore, it is necessary to combine them with the environmental noise spectrum in the original speech command stream for noise robustness enhancement. The noise spectrum provides the characteristics of the interference signal. Enhancement can be performed using traditional methods such as spectral subtraction and Wiener filtering, or by using deep learning models such as denoising autoencoders. The core idea is to estimate and subtract the influence of noise components from noisy speech features, or to learn a mapping function from noisy features to clean features, generating an enhanced acoustic feature representation that better reflects the essential information of the speech. This step significantly improves the stability of features in complex acoustic environments. Deep semantic decoding of enhanced acoustic feature representations represents a crucial leap in transforming low-level acoustic features into high-level symbolic instructions. This process typically relies on deep neural network models. First, the sequence of enhanced acoustic feature representations is input into a bidirectional long short-term memory (BiLSTM) neural network for context encoding. BiLSTM can traverse the input sequence in both forward and backward directions, ensuring that the feature representation at each time step incorporates past and future contextual information, thereby generating a context-aware feature representation rich in contextual information. However, for long sequences, the hidden state at the last time step of BiLSTM may not effectively carry the information from the beginning of the sequence. Therefore, an attention mechanism is needed to focus key information in the context-aware feature representation. The attention mechanism dynamically calculates the attention weights of each time step to the hidden states of the encoder at each time step during the decoding process. Then, based on these weights, the hidden states of the encoder are weighted and summed to generate an attention-weighted feature vector most relevant to the current decoding state. This vector can adaptively focus on the part of the input sequence most relevant to the current output.

[0071] Using a connectionist temporal classification decoder to perform sequence-to-sequence mapping on attention-weighted feature vectors is an effective way to solve problems such as inconsistent input and output sequence lengths. The CTC decoder predicts a probability distribution of a set of labels at each time step in the output layer. This set contains all possible phonemes or characters and a special whitespace character. The dynamic programming algorithm is used to integrate all possible label sequences, allowing duplicate labels and whitespace characters in the input sequence. Finally, the initial semantic parsing result is obtained by merging duplicate labels and removing whitespace characters. To evaluate the reliability of this result, a semantic confidence score needs to be calculated based on a softmax classifier. The softmax function transforms the raw scores of the CTC output layer into a probability distribution. The semantic confidence score can be the probability of the entire output sequence or the product of the probabilities of the key label sequences. A high score usually means that the parsing result is highly consistent with the acoustic evidence.

[0072] Because users may have different accents or pronunciation habits, the generated initial semantic parsing results may contain non-standard expressions. Therefore, it is necessary to use pronunciation feature parameters to perform dialect-adaptive correction on the initial semantic parsing results. Pronunciation feature parameters can indicate the user's potential dialect background. The correction process can be based on a dialect pronunciation dictionary or dialect language model to map the words or pronunciations in the initial parsing results to standard forms, or to correct possible misidentifications according to dialect rules, thereby generating a standardized semantic operation object description that is easy for the system to understand and process. In the final stage, all intermediate results are integrated to generate the final structured control instruction set. The system needs to integrate phoneme segmentation boundary data, semantic confidence scores, and standardized semantic operation object descriptions. Phoneme segmentation boundary data provides temporal structure information of the instructions, which helps to determine the boundaries of instruction words. Semantic confidence scores, as weights, give higher confidence to the parts with higher confidence during the fusion process. Standardized semantic operation object descriptions clarify the operation objects of the instructions, such as specific song names, actor names, or operation objects such as "volume" and "brightness". Through a rule engine or lightweight inference model, the logical relationship between these information is analyzed to infer the user's final intention and generate a clear acoustic control intention identifier, such as PLAY, SEEK, or VOLUME_UP. At the same time, the semantic operation object descriptions are refined and context-related parameters are generated by associating the current dialogue context and device status. These three together constitute a machine-readable and semantically rich structured control instruction set, providing accurate input for subsequent state adaptation and control execution.

[0073] Example 2: See Figure 3The process of extracting acoustic features from time-domain audio signal sequences to generate acoustic fingerprint feature vectors and phoneme segmentation boundary data forms the foundation of speech command parsing. This process transforms the original waveform data into a parameterized representation more suitable for machine recognition and analysis. The time-domain audio signal sequence first needs to undergo a series of preprocessing operations to improve the signal-to-noise ratio and stability of subsequent processing. Preprocessing includes a pre-emphasis stage, which uses a first-order high-pass filter to enhance the high-frequency components of the signal, thereby compensating for the high-frequency attenuation effect caused by lip radiation during speech, making the signal spectrum flatter. The pre-emphasis processed signal is then segmented into a series of short time frames. The framing operation usually uses a fixed-length Hamming window or Hanning window for windowing. The purpose of windowing is to reduce the signal discontinuity at the edges of each frame. After framing, the signal changes from a continuous waveform to a series of overlapping short time segments, preparing for subsequent frequency domain analysis. Transforming the short-time signal segments after frame-segmentation and windowing into the frequency domain is a crucial step in obtaining the Mel frequency spectrum coefficients. Each short-time frame signal is converted into a corresponding linear spectrum through a Fast Fourier Transform (FFT). This spectrum reflects the amplitude information of each frequency component of the signal within that time segment. However, the scale of the linear spectrum differs from the nonlinear perceptual characteristics of the human auditory system. Therefore, it is necessary to map the linear frequency scale to the Mel frequency scale, which better matches the characteristics of human hearing. The Mel frequency scale has high resolution in the low-frequency region, but its resolution gradually decreases in the high-frequency region. The mapping process uses a set of triangular filters covering the entire analysis frequency band and distributed according to the Mel frequency scale. This is achieved using a filter bank. This filter bank convolves the linear power spectrum, calculates the sum of signal energy within each Mel band, and thus converts the linear spectrum into filter bank energy at the Mel scale. Taking the logarithm of these energy values ​​can compress the dynamic range of the data, making its distribution closer to the human ear's perception of loudness. Finally, a discrete cosine transform is performed on the logarithmic Mel filter bank energy to decorrelate and compress the data. The first few coefficients after the transformation constitute the Mel frequency cepstral coefficients characterizing the signal characteristics of that frame. Arranging the MFCCs of all time frames in chronological order generates a time-frequency feature representation matrix describing the entire speech command stream.

[0074] Because different users may have significant differences in speech rate when issuing the same voice command, directly comparing the time-frequency feature representation matrices corresponding to two commands of different durations will introduce distortion on the time axis. Dynamic time warping is designed to solve this problem of nonlinear change on the time scale. The algorithm aims to find an optimal warping path, which defines the correspondence between each feature vector of the test feature sequence and a reference template sequence. The search for the warping path is usually constrained by endpoint constraints, monotonicity constraints, and continuity constraints to ensure the physical rationality of the alignment. By calculating the distance between each pair of feature vectors in the test sequence and the reference sequence, and applying a dynamic programming algorithm to accumulate the minimum cumulative distance, the path that minimizes the total distance is finally found by backtracking. Based on this path, the original time-frequency feature representation matrix is ​​stretched or compressed to generate a warped time-frequency feature sequence with the same length as the reference template in the time dimension and a clearer feature correspondence. Extracting acoustic fingerprint feature vectors from the regularized time-frequency feature sequence requires focusing on relatively stable acoustic properties that can effectively distinguish different speech content. Formant trajectory features reflect the direction of spectral peak frequencies determined by changes in the vocal tract shape during articulation. Formant extraction typically employs linear predictive coding analysis or cepstral analysis. Linear predictive coding predicts the current signal sample by establishing a full-pole vocal tract model; the frequencies corresponding to the model poles are related to the formant frequencies. By tracking the changes in these pole frequencies over time, the trajectories of the first few formants (e.g., F1, F2, F3) can be obtained. The fundamental frequency variation... The fundamental frequency curve reflects the basic frequency of vocal cord vibration, which constitutes the prosody and tone information of speech. Fundamental frequency estimation can be performed using the autocorrelation method, which finds periodicity by calculating the correlation between the signal and its delayed replicas, with the peak position corresponding to the fundamental frequency period. Alternatively, algorithms based on harmonic structures, such as the weighted correlation function method, can be used to improve the estimation robustness in noisy environments. Multiple formant frequency values, fundamental frequency values, and other possible acoustic parameters extracted from each time frame are combined to form a high-dimensional feature vector. This vector sequence or its statistical summary constitutes the acoustic fingerprint feature vector characterizing the uniqueness of the speech segment.

[0075] Phoneme-level segmentation relies on the ability of Hidden Markov Models (HMMs) to model the temporal structure of speech. HMMs treat the speech signal as a doubly stochastic process: one is the implicit sequence of phoneme states, and the other is a stochastic process that generates observed feature vectors from each state. HMMs used for phoneme segmentation typically have a left-to-right topology, with each phoneme represented by an HMM containing several states (usually 3 or 5). Self-jumps and transitions to the next state are allowed between states. Model parameters (including initial state probabilities, state transition probabilities, and emission probabilities of observed feature vectors in each state) need to be pre-defined. A large amount of speech data that has been annotated at the phoneme level is used for training. The training usually uses the Baum-Welch algorithm for expectation maximization estimation. When processing new normalized time-frequency feature sequences, the Viterbi decoding algorithm is used to find the hidden state sequence that is most likely to produce the observation sequence. The algorithm calculates the maximum probability path to each state through dynamic programming and records the path source. Finally, it backtracks to obtain the global optimal path. The state change points on this path correspond to the boundaries between phonemes, thereby generating phoneme segmentation boundary data. At the same time, the HMM state of each segment on the path also identifies the phoneme category to which the segment belongs.

[0076] The core task of deep semantic decoding is to map the enhanced acoustic feature representation into a meaningful symbol sequence. Bidirectional long short-term memory neural networks serve as encoders, with complex gating mechanisms that enable them to effectively learn long-term dependencies in sequences. BiLSTM processes the input sequence from both forward and backward directions and concatenates or sums the hidden states of the two directions at each time step to generate context-aware feature representations. These representations capture the intrinsic relationship between the features at each time step and their contextual information. The attention mechanism was introduced to address the information bottleneck problem caused by the encoder needing to compress the information of the entire input sequence into a fixed-dimensional vector. In the attention mechanism, when the decoder generates each output symbol, it calculates an attention weight vector. This vector represents the degree of attention the decoder pays to the hidden states of the encoder at each time step at the current time. The weights are usually calculated by a trainable feedforward neural network. The input is the hidden state of the decoder at the previous time step and the hidden state of the encoder at a certain time step. After the calculated weights are normalized by the softmax function, they are used to perform a weighted summation of the hidden states of the encoder at all time steps to generate an attention weighted feature vector that is most relevant to the current decoding step. This vector dynamically focuses on the part of the input sequence that is most relevant to the current output.

[0077] Connectionist temporal classification decoders (CTCs) provide an end-to-end sequence learning framework, particularly suitable for tasks with unequal input and output sequence lengths and unknown alignment relationships. At the output layer, CTCs define an expanded label set for each time step, containing all possible output symbols (such as phonemes or characters) and a special symbol representing space (blank). CTCs' forward-backward algorithm efficiently calculates the sum of probabilities for all possible output sequences given an input sequence. During the decoding phase, a beam search algorithm is typically used to find the output sequence with the highest probability. CTCs allow duplicate labels and blank symbols in the input sequence, ultimately obtaining the initial semantic parsing result by merging consecutive identical non-blank labels and removing all blank symbols. The Softmax classifier operates at each time step of the CTC output layer, transforming the non-normalized score of the CTC output into a probability distribution. This distribution represents the probability of each label appearing at that time step. The semantic confidence score can be calculated based on this probability distribution. For example, the logarithm of the conditional probability of the entire output sequence can be taken as the overall confidence score, or the product of the probabilities of the key labels in the output sequence can be taken, or the average label probability of the entire sequence can be calculated. This score quantifies the uncertainty of the parsing result.

[0078] Example 3: The process of executing player state adaptation operations based on a structured control instruction set to generate a dynamic control response sequence is a key bridge connecting voice understanding and actual device control. This operation transforms abstract semantic instructions into specific executable device commands, while fully considering the current state of the player and the user's operating habits. The structured control instruction set includes acoustic control intent identifiers, semantic operation object descriptions, and context-related parameters. The acoustic control intent identifier is usually an encoded symbol, such as a code representing a basic operation type. The first step in parsing this identifier is to map it to an element in a predefined set of operation types. This mapping process is accomplished by querying an intent mapping table, which defines the correspondence between identified intent words or phrases and machine-executable operation types. While parsing the basic operation type identifier, it is also necessary to extract the operation intensity parameter from the parameters or modifiers attached to the intent identifier. For example, for the "volume up" instruction, the intensity parameter might be a small increment value corresponding to "slightly" or an absolute value corresponding to "turn to maximum".

[0079] Retrieving media resource databases based on semantic operation object descriptions requires converting natural language descriptions into database query conditions. These descriptions might include media names, creators, album names, program types, etc., mentioned in the command. The system first standardizes and segments the descriptions, then matches them against metadata fields in the media resource database. Each resource record in the database contains a unique target media resource identifier and resource location information for locating the actual media file. The retrieval process may employ fuzzy matching algorithms to handle cases where the user's pronunciation is not standard or the description is incomplete. Retrieval results may return a single match or multiple candidates. When multiple candidates exist, it may be necessary to combine contextual parameters or perform disambiguation through subsequent interactions. Contextual parameters provide crucial constraints and background information for state adaptation. Historical operation sequence records need to be extracted from these parameters. These records store a series of recent user commands executed on the player and their timestamps in chronological order. Simultaneously, a snapshot of the current device state needs to be obtained. This snapshot records the player's various state variable values ​​at the moment the current command is received, such as the currently playing media resource identifier, playback progress, volume level, loop mode, and playback rate.

[0080] Pattern mining is performed on historical operation sequence records based on the state machine model to discover user habitual operation patterns or common state transition paths. The state machine model explicitly defines all possible states of the player and the allowed transition conditions between states. For example, the "play" state can transition to the "pause", "stop" or "fast forward" state, but cannot directly transition to a settings menu state. By analyzing historical operation sequences, the frequency of various subsequent operations starting from a specific state can be statistically determined, or certain operation sequences can be found to frequently appear consecutively, thereby generating a set of common state transition patterns. This set contains frequently occurring state transition paths and their support or confidence. Based on the degree of matching between the current device state snapshot and a set of common state transition patterns, the system can predict the state transition that the user may expect next. The degree of matching is evaluated by calculating the similarity between the current state and the starting state of each pattern in the pattern set, as well as the degree of matching between the current context and the context in which the pattern occurred. For example, if the current state is "playing a movie", and the historical patterns show that the user often performs the "adjust subtitles" operation when the movie is playing, then even if the user's current instruction is ambiguous, the system may prioritize operations related to subtitles. By calculating the probability of all possible next states, a state transition probability distribution is generated.

[0081] To select the optimal state transition path from multiple possible paths, a transition cost function needs to be defined for optimization and selection. This function takes into account a variety of factors:

[0082]

[0083] Where: symbol This represents a candidate control action, such as play, pause, or skip, which is a potential response that the system is evaluating. (Symbol) This represents the current state of the player, including all dynamic variables such as media ID, playback progress, and volume settings. (Symbol) It includes current context information, such as time, user identity, and recent operation history. (Symbol) It is a time cost factor that estimates the amount of time required to perform an action. From state The time delay required to begin. (Symbol) Measuring the action With stored in history The degree of inconsistency between user habit patterns; a higher value indicates a greater deviation from typical user behavior. (Symbol) The execution actions were evaluated. In context The required computing resources or energy consumption. Weighting coefficient. , , These represent the relative importance of time cost, user habit deviation cost, and resource consumption cost in the total cost calculation, respectively. These weights can be adjusted based on device performance or user settings. By calculating the total cost of each candidate action, the system can select the path with the lowest cost and generate corresponding state transition constraints, such as restricting certain high-cost or low-probability operations.

[0084] The integration process requires combining basic operation type identifiers, operation intensity parameters, target media resource identifiers, and state transition constraints. First, it checks whether the basic operation type is compatible with the target media resource. For example, the "play" operation requires a valid media resource identifier, while the "mute" operation does not. Then, it checks whether the operation is allowed by the state transition constraints in the current device state. If the operation is allowed, the device state adjustment command is specified according to the operation intensity parameter. For example, the intent to "increase volume" is transformed into a specific volume value setting command. The media content positioning parameter is generated based on the operation type and semantic object description. For example, "start playing from the fifth minute" will generate a precise timestamp. The interface interaction logic identifier indicates the user interface elements that need to be updated and their new states. For example, highlighting the operated function button and updating the progress bar display. Finally, all these specific commands and parameters are organized into an ordered dynamic control response sequence, which ensures the logical consistency and execution feasibility of the control instructions.

[0085] The entire state adaptation process is a complex decision-making process. It not only translates the user's direct instructions, but also incorporates an understanding of the user's habits and a consideration of the system's current state. This makes the final generated control response sequence more intelligent and personalized, smoothly guiding the player from the current state to the user's desired target state, while keeping the user interface updated in sync, providing the user with a seamless control experience.

[0086] Example 4: The implementation process of driving the audio and video player to perform multi-dimensional control operations based on the dynamic control response sequence and generating real-time playback control effect feedback data is a key stage in transforming abstract control logic into specific device behavior. This process involves precise calls to underlying hardware interfaces and software components. Assuming that the user voice command received by the system is "start playing the movie 'Interstellar' from the fifth minute", after the aforementioned steps of parsing and adaptation, a structured dynamic control response sequence is generated. This sequence contains specific information such as device status adjustment commands, media content positioning parameters, and interface interaction logic identifiers.

[0087] Parsing device status adjustment commands is the first step in executing control. This command is not a simple instruction name, but a structured data block containing specific parameter values. For example, for playback control, the command explicitly specifies the target playback state as "play" rather than "pause." For volume control, it may contain a specific volume percentage value. Assuming the current volume is 50%, and the user command implicitly sets the volume to 70%, the device status adjustment command will include a parameter to adjust the volume from 50% to 70%. The playback rate parameter is usually a multiplier; for example, 1.0 represents normal speed, and 1.5 represents 1.5x speed. This parameter also needs to be parsed from the command. If the command is "slow playback," the rate parameter might be set to 0.75. The channel balance parameter specifies the relative volume ratio of the left and right channels, used for sound image localization or adaptation to different audio formats. All these parameters must be converted into specific values ​​and instruction formats that can be recognized by the audio / video player's underlying software development kit or the operating system's multimedia interface.

[0088] The media file location operation based on the media content location parameters requires interaction with the media resource management system. The media content location parameters include at least the target media resource identifier and precise timestamp location data. The target media resource identifier is a unique index number or Uniform Resource Locator pointing to a specific media file (such as "Interstellar"). The system first needs to locate the corresponding media file in local storage or network media library based on this identifier and initialize the decoder to prepare for playback. The precise timestamp location data specifies the starting point of playback. For example, the timestamp corresponding to "start from the fifth minute" in the instruction is 300,000 milliseconds. The location operation is not a simple file offset, but requires the use of the media file's internal index information (such as keyframe index) for fast and accurate location to ensure that the picture can be rendered correctly immediately when playback starts, avoiding long black screens or decoding errors.

[0089] The interface interaction logic identifier controls the update operations of user interface elements, aiming to ensure that the user interface state is visually consistent with the internal state of the player and the control actions being executed. The interface interaction logic identifier describes in detail the interface components that need to be updated, the new visual state of the components, and possible transition animation effects. For example, when a playback command is executed, the "play / pause" button icon on the interface needs to switch from the "play" state to the "pause" state, the progress bar needs to immediately jump to the fifth minute position and start moving forward dynamically, the current playback time display needs to be updated to "05:00", and if the media poster or title information changes, the corresponding display area also needs to be updated with the relevant information of "Interstellar". After receiving these update instructions, the interface rendering engine will generate a series of specific interface rendering instruction sequences, which will schedule the graphics processing unit to redraw the relevant user interface elements.

[0090] Synchronous execution is the core element for ensuring a smooth and lag-free user experience. It requires that several relatively independent operations, such as volume adjustment, playback rate setting, channel balance adjustment, media content positioning, and user interface updates, be coordinated and completed within a very short time window. The system usually adopts a multi-threaded programming model to achieve synchronization. The main thread is responsible for parsing the dynamic control response sequence and distributing tasks. One thread is dedicated to calling the underlying audio interface to set volume, rate, and balance parameters. Another thread is responsible for controlling the media decoder to perform positioning and start playback operations. User interface updates are usually triggered in a dedicated UI thread through a message queue mechanism. Through synchronization primitives between threads (such as semaphores and barriers) and precise timestamp management, the system strives to make these changes appear instantaneous to the user. That is, at the same time the user hears the sound start playing from a specified position, the progress bar and button status on the interface are also updated synchronously.

[0091] Generating real-time playback control effect feedback data is a comprehensive record and confirmation of the execution result of this control operation. This feedback data is a structured log. It is not information displayed to the user, but is used for internal system status tracking and subsequent optimization. The feedback data records the final effective device status parameters (such as the actual set volume value and playback rate), the precise start timestamp of media playback, whether the interface element update was successfully completed, and the timestamp and duration of the entire control sequence execution. This data constitutes an objective description of whether the "instruction was executed accurately". After the control operation is completed, the system immediately enters a passive listening phase to collect data on the user's subsequent interaction behavior. These behaviors are the user's most direct evaluation of the control result. Subsequent interaction behavior data includes, but is not limited to: whether the user issued a cancellation or correction command within a short period of time (e.g., within 3 seconds), such as saying "no" or "not here"; whether the user intervened through non-voice means such as touch screen or remote control, such as manually dragging the progress bar; whether the user repeatedly issued the same or semantically similar command, implying that the first execution did not achieve the expected result; and even the system can analyze the user's facial expressions through a camera (if available and authorized) to determine whether the user shows signs of confusion or dissatisfaction. After these multimodal behavioral data are captured by the system, they will be quantified into a series of numerical features to form a behavioral feedback feature vector, as shown in Table 1.

[0092] Table 1: Behavioral Feedback Feature Vectors

[0093] Feature Name Data types Value description Example value Cancel instruction flag Boolean value 1 indicates cancellation, 0 indicates no cancellation. 1 Non-speech intervention time difference Integer (milliseconds) The time interval between control execution and non-voice intervention 2500 Instruction repetition count Plastic Surgery Number of times the same instruction is repeated in a short period of time 0 Confusion score floating-point numbers The level of confusion (0-1) is determined based on facial expression analysis. 0.8 Silence duration after operation is completed Integer (milliseconds) Duration of time during which the user does not interact after the control is completed. 5000

[0094] Comparing the deviation between the behavioral feedback feature vector and the expected control effect is an evaluation process based on rules or a lightweight classifier. The expected control effect refers to the behavior that the user should exhibit after correctly executing the instruction under ideal conditions. For example, after successful playback, the user should enter a silent viewing state without immediately performing a correction operation. The system compares the actual collected behavioral feedback feature vector with this "ideal profile." For example, if the "cancel instruction flag" is 1, or the "non-voice intervention time difference" is less than a threshold (such as 3 seconds), or the "expression confusion score" is too high, it indicates that the actual effect deviates significantly from the expectation. Through a predefined set of evaluation rules or a trained binary classification model, these feature deviations are comprehensively calculated into a quantitative control accuracy evaluation index. This index can be a score between 0 and 1. The lower the score, the worse the accuracy of the control and the lower the user satisfaction may be. This evaluation index provides the system with a quantifiable measure of the quality of this voice control interaction based on real user feedback.

[0095] Example 5: The process of updating the parameter configuration of multimodal instruction parsing and processing based on the control precision evaluation index to generate optimized parsing model parameters reflects the system's self-learning and continuous improvement capabilities. This process converts the indirect feedback of user interaction into specific adjustment amounts of model parameters. Suppose in an interaction, the user issues the instruction "Play Qi Li Xiang by Jay Chou". After the system recognizes it, it plays the song "Qi Li Xiang" but the singer is not Jay Chou. The control precision evaluation index is judged to be a low score through subsequent user behavior, indicating that the current parsing model has deficiencies in the association understanding of specific singers and song names. To calculate the acoustic model adaptation coefficient and the language model correction amount according to the control precision evaluation index, a mapping relationship between the evaluation result and the model parameters needs to be established. The acoustic model adaptation coefficient is a vector or scalar that indicates the adjustment direction and amplitude required for acoustic feature extraction or the acoustic model itself to cope with the current user pronunciation characteristics or the current environmental noise characteristics. The calculation process may be based on error analysis. For example, if it is found that the acoustic characteristics of the character "Jie" in "Jay Chou" do not match well with the original template in the model, resulting in confusion, then the acoustic model parameters corresponding to the phonemes related to the character "Jie" need a positive adaptation coefficient to enhance their weights. The language model correction amount focuses on the lexical and grammatical levels and points out the deviation in the probability distribution of the current language model. In the above example, the joint probability of the phrase "Qi Li Xiang by Jay Chou" in the language model may be too low, or the connection probability between "Qi Li Xiang" and other singers may be too high. Therefore, a correction amount needs to be calculated to increase the probability of the co-occurrence of "Jay Chou" and "Qi Li Xiang" and reduce the connection probability between "Qi Li Xiang" and irrelevant singers.

[0096] Gradient descent optimization processing of the acoustic model adaptation coefficient is one of the most widely used parameter update strategies. The core idea of gradient descent is to fine-tune the model parameters along the direction of the steepest descent of the loss function, thereby gradually reducing the difference between the model output and the expected output. In this process, the negative value of the control precision evaluation index can be regarded as part of the loss function. The acoustic model adaptation coefficient indicates the direction in which the parameters need to be adjusted, and the gradient descent algorithm calculates the specific adjustment step size for each parameter. For example, for a deep neural network acoustic model, the gradient of the loss function with respect to the network weight parameters can be calculated through the backpropagation algorithm, and then stochastic gradient descent or its variants (such as the Adam optimizer) are used to update the weights along the negative gradient direction to generate a set of updated acoustic feature extraction parameters. This process usually only uses the current error sample or a small batch of recent samples, belonging to online learning or small batch learning, to avoid forgetting old knowledge.

[0097] Adjusting the weights of the semantic decoding dictionary using the language model correction directly affects the priority of word selection in the parsing stage. The semantic decoding dictionary usually contains a list of words and the probability (i.e., weight) of each word appearing in a specific context. The language model correction indicates which entries or N-gram combinations need their weights adjusted. For example, for the misrecognition of "Jay Chou's Qi Li Xiang", the system will increase the probability weight of the bigram "Jay Chou_Qi Li Xiang" in the language model, and at the same time may reduce the weight of the combination of "Qi Li Xiang" with the previously misrecognized singer. The adjustment can be achieved by directly modifying the count smoothing parameters of the N-gram model, or for a neural network language model, indirectly changing the probability of the context representation of words by updating the network parameters through backpropagation and gradient descent, thereby generating optimized language model parameters. This adjustment makes the model more likely to output the correct word sequence when encountering similar contexts in the future.

[0098] Fusing the updated acoustic feature extraction parameters and the optimized language model parameters to generate optimized parsing model parameters requires ensuring the compatibility and synergy of the two parts of the parameters. The update of the acoustic model may change the distribution of features, and the update of the language model changes the prior probability of words. The two need to be integrated into a unified parsing framework. The fusion operation may simply package the two new parameter files and replace the old model file. In a more complex system, a quick online joint fine-tuning may be required to check the performance of the fused model using a small amount of reserved validation data to ensure that the balance between the acoustic score and the language model score is not broken. The generated overall optimized parsing model parameter package will be marked with a version number and a timestamp for management and rollback purposes.

[0099] Deploying the optimized parsing model parameters to an online learning framework for incremental learning processing is a key step in realizing the continuous evolution of the model. The online learning framework is responsible for managing the life cycle of the model version, handling concurrent issues of model updates, and controlling the update rhythm to avoid model oscillations. Incremental learning processing means that instead of completely retraining the model with new data, new knowledge (in the form of optimized parameters) is integrated into the existing model in small steps. The framework may adopt an exponential smoothing strategy to perform a weighted average of the old model parameters and the newly calculated parameters, enabling the model to absorb new knowledge while retaining most of the old memories, or adopt a method based on a replay buffer to randomly select a part of the old data to train with the new data together to alleviate the problem of catastrophic forgetting.

[0100] The execution logic of updating the adaptive instruction parsing strategy based on incremental learning results enables the strategy itself to learn. The adaptive instruction parsing strategy includes decision logic such as confidence threshold setting, rejection rules, and timing of clarification questions. Changes in model performance after incremental learning need to be fed back into these strategies. For example, if it is found that the model's confidence in recognizing music-related instructions has generally improved after several optimizations, then the confidence threshold for music instructions in the strategy can be appropriately increased to reduce unnecessary user confirmations. Conversely, if the recognition error rate increases in certain scenarios, the strategy may trigger more frequent confirmations or suggest the use of touch control. The strategy update is usually based on monitoring and analyzing the performance of the model after incremental learning on a series of test cases or recent real interaction data.

[0101] The system processes subsequent input voice command streams based on the updated adaptive command parsing strategy, completing a closed loop of learning and improvement from a single erroneous interaction. When the user says "play Jay Chou's 'Seven Mile Fragrance'" or a similar command, the optimized acoustic model can more accurately capture the pronunciation features of "Jay Chou," and the optimized language model assigns a higher probability to "Jay Chou's 'Seven Mile Fragrance.'" The parsing strategy may directly execute the correct command due to increased confidence, eliminating the need for secondary confirmation from the user. Through this continuous and minute adjustment of parameters and strategies, the entire system gradually adapts to the user's personal pronunciation habits, commonly used command types, and usage environment, thereby providing an increasingly accurate and smooth voice interaction experience. This self-optimization capability makes the system no longer static, but a dynamic intelligent agent that can grow together with the user.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A voice command-based audio / video player control method, characterized in that, The method includes: The system collects the raw voice command stream issued by the user, which includes a time-domain audio signal sequence, an environmental noise spectrum, and pronunciation feature parameters. The original voice command stream is subjected to multimodal command parsing processing to generate a structured control command set, which includes acoustic control intent identifiers, semantic operation object descriptions, and context association parameters. Based on the structured control instruction set, the player state adaptation operation is executed to generate a dynamic control response sequence, which includes device state adjustment commands, media content positioning parameters, and interface interaction logic identifiers. Based on the dynamic control response sequence, the audio and video player is driven to perform multi-dimensional control operations, generating real-time playback control effect feedback data; Based on the real-time playback control effect feedback data, the parameters of the multimodal instruction parsing and processing are optimized and adjusted to generate an adaptive instruction parsing strategy; The step of performing player state adaptation operations based on the structured control instruction set to generate a dynamic control response sequence includes: The basic operation type identifier and operation intensity parameters are obtained by parsing the acoustic control intent identifier; Based on the semantic operation object description, the media resource database is retrieved to generate target media resource identifiers and resource location information; Calculate the player state transition path based on the context association parameters, and generate state transition constraints. By integrating the basic operation type identifier, operation intensity parameter, target media resource identifier, and state transition constraint, a device state adjustment command, media content positioning parameter, and interface interaction logic identifier are generated. The step of calculating the player state transition path and generating state transition constraints by combining the context association parameters includes: Extract the historical operation sequence records and current device status snapshot from the context association parameters; Based on the state machine model, the historical operation sequence records are processed by pattern mining to generate a set of common state transition patterns. A state transition probability distribution is generated based on the degree of matching between the current device state snapshot and a set of common state transition patterns. The state transition probability distribution is optimized and filtered based on the transition cost function to generate state transition constraints.

2. The audio / video player control method based on voice commands according to claim 1, characterized in that, The step of performing multimodal instruction parsing processing on the original voice instruction stream to generate a structured control instruction set includes: The time-domain audio signal sequence is subjected to acoustic feature extraction processing to generate acoustic fingerprint feature vectors and phoneme segmentation boundary data; The acoustic fingerprint feature vector is subjected to noise robustness enhancement processing based on the ambient noise spectrum to generate an enhanced acoustic feature representation; The enhanced acoustic feature representation is subjected to deep semantic decoding to generate initial semantic parsing results and semantic confidence scores; The initial semantic parsing results are subjected to dialect-adaptive correction using the pronunciation feature parameters to generate a standardized semantic operation object description; By integrating the phoneme segmentation boundary data, semantic confidence scores, and standardized semantic operation object descriptions, the acoustic control intent identifier, semantic operation object description, and context association parameters are generated.

3. The voice command-based audio / video player control method according to claim 2, characterized in that, The acoustic feature extraction process for the time-domain audio signal sequence, generating acoustic fingerprint feature vectors and phoneme segmentation boundary data, includes: The time-domain audio signal sequence is converted into a Mel frequency spectrum coefficient sequence to generate a time-frequency feature representation matrix; The time-frequency feature representation matrix is ​​subjected to dynamic time warping to generate a warped time-frequency feature sequence; Extract the formant trajectory features and fundamental frequency variation curve from the normalized time-frequency feature sequence to generate an acoustic fingerprint feature vector; The regularized time-frequency feature sequence is segmented at the phoneme level based on the Hidden Markov Model to generate phoneme segmentation boundary data and phoneme category identifiers.

4. The audio / video player control method based on voice commands according to claim 2, characterized in that, The process of performing deep semantic decoding on the enhanced acoustic feature representation to generate initial semantic parsing results and semantic confidence scores includes: The enhanced acoustic feature representation is input into a bidirectional long short-term memory neural network for context encoding to generate a context-aware feature representation. The context-aware feature representation is processed by an attention mechanism to focus key information, generating an attention-weighted feature vector. The attention-weighted feature vector is sequence-to-sequence mapping processed using a connectionist temporal classification decoder to generate initial semantic parsing results. The semantic confidence score of the initial semantic parsing result is calculated based on the softmax classifier.

5. The voice command-based audio / video player control method according to claim 1, characterized in that, The step of driving the audio / video player to perform multi-dimensional control operations based on the dynamic control response sequence and generating real-time playback control effect feedback data includes: The volume adjustment parameters, playback rate parameters, and channel balance parameters are obtained by parsing the device status adjustment command. Perform media file location operation based on the media content location parameters to generate accurate timestamp location data; Based on the interface interaction logic identifier, control the user interface element update operation and generate an interface rendering instruction sequence. The volume adjustment parameters, playback rate parameters, channel balance parameters, precise timestamp positioning data, and interface rendering instruction sequence are executed synchronously to generate real-time playback control effect feedback data.

6. The voice command-based audio / video player control method according to claim 5, characterized in that, After generating the real-time playback control effect feedback data, the method further includes: Collect user's subsequent interaction behavior data to generate behavioral feedback feature vectors; By comparing the deviation between the behavioral feedback feature vector and the expected control effect, a control accuracy evaluation index is generated. The parameter configuration of the multimodal instruction parsing and processing is updated based on the control accuracy evaluation index to generate optimized parsing model parameters.

7. The voice command-based audio / video player control method according to claim 6, characterized in that, The step of updating the parameter configuration of the multimodal command parsing and processing based on the control accuracy evaluation index to generate optimized parsing model parameters includes: Calculate the acoustic model adaptation coefficient and the language model correction amount based on the control accuracy evaluation index; The acoustic model adaptation coefficients are optimized by gradient descent to generate updated acoustic feature extraction parameters. The semantic decoding dictionary weights are adjusted using the language model correction values ​​to generate optimized language model parameters. By combining the updated acoustic feature extraction parameters and the optimized language model parameters, optimized analytical model parameters are generated.

8. The voice command-based audio / video player control method according to claim 7, characterized in that, After generating the optimized analytical model parameters, the process also includes: The optimized parsed model parameters are deployed to an online learning framework for incremental learning processing. The execution logic of the adaptive instruction parsing strategy is updated based on the incremental learning results; The raw speech command stream of subsequent inputs is processed according to the updated adaptive instruction parsing strategy.

Images