Director intention recognition method based on diffusion type speech enhancement and visual gesture coding

Abstract

The method for director intention recognition based on diffusion speech enhancement and visual gesture coding relates to the technical fields of pattern recognition and image recognition, and the speech intelligibility is improved through the generation diffusion prior and observation consistency guidance, the gesture is explicitly structured through ROI alignment and skeleton key points, the asynchronous and noise level different modal evidences are adaptively integrated through block level alignment, attention convergence and gate weighted, and finally the stable director intention category and related parameters and confidence are output, so that the cooperative understanding and reliable discrimination of the director speech password and gesture signal are provided without relying on specific device control; the application can continuously output more stable, comparable and interpretable director intention results without relying on specific device control, solve the problems of short-time large fluctuation, strong noise interference and cross-modal asynchronization leading to unstable intention, and further reduce the difficulty and cost of live production.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition and image recognition technology, and in particular to a method for director intent recognition based on diffusion-based speech enhancement and visual gesture coding. Background Technology

[0002] In the real-world scenario of entertainment group live streaming, accurately and consistently identifying the director's intentions is crucial for ensuring the smooth flow and effectiveness of the broadcast. However, this field currently faces numerous technical challenges that urgently need to be addressed.

[0003] The live broadcast environment is complex and ever-changing, with strong noise and unstable interference everywhere, frequent camera obstruction, and extremely drastic lighting changes. Under such conditions, relying on a single voice or gesture channel to recognize the director's intentions has significant shortcomings and is easily rendered ineffective by external interference. For example, strong noise can severely interfere with voice signals, making it difficult to accurately recognize voice commands; camera obstruction and drastic lighting changes can affect the capture and interpretation of gesture information, leading to errors in gesture recognition.

[0004] Traditional methods typically employ threshold- and rule-based multi-channel fusion strategies to integrate speech and gesture information to improve recognition accuracy. However, in practical applications, this strategy proves highly vulnerable under complex conditions of asynchronous, multi-source, and uncertain noise. On one hand, precise temporal synchronization of information from different channels is difficult, significantly reducing the fusion effect. On the other hand, due to the uncertainty of noise, pre-set thresholds and rules struggle to adapt to various changes, impacting system stability. Furthermore, traditional methods are costly to maintain, requiring extensive debugging and optimization for different scenarios and program characteristics, undoubtedly increasing the difficulty and cost of live broadcast production.

[0005] In summary, existing director intent recognition technologies for live entertainment group broadcasts struggle to achieve stable, real-time, and low-cost recognition results in complex and ever-changing live environments. Therefore, a director intent recognition method based on diffusion-based speech enhancement and visual gesture coding is urgently needed to address these issues. Summary of the Invention

[0006] To address the aforementioned technical problems in related technologies, this invention proposes a director's intent recognition method based on diffusion-based speech enhancement and visual gesture coding.

[0007] In a first aspect, the present invention provides a method for director intent recognition based on diffusion-based speech enhancement and visual gesture coding, comprising the following steps:

[0008] S1. Speech Enhancement and Diffusion Denoising: An additivity observation model is set for the mixed speech waveform y. An injection noise estimate is given through a diffusion denoising network, and an inverse mean is formed as the speech prior. Then, the inverse mean is corrected by constructing the gradient of the log-conditional likelihood based on the lightweight observation likelihood model to obtain the guided inverse mean. The next reverse state is obtained by randomly sampling it using a Gaussian distribution. The enhanced speech waveform is obtained by continuously performing reverse iterations. ;d represents the time step;

[0009] S2, Speech semantic coding: Enhancing speech waveforms The complex spectrum is obtained by framing and performing a short-time Fourier transform. Then, the power spectrum is calculated based on the complex spectrum, and the logarithm is obtained by Mel filtering. Normalized frame-level features are obtained by mean normalization. Then, normalized frame-level features Gating is performed to obtain the gated frame-level acoustic vector. Then, by aligning the visual temporal granularity through frame stacking and downsampling, block-level acoustic vectors are obtained. Then, it is projected onto the multimodal common dimension through two layers of projection. The speech semantic embedding sequence is obtained by superimposing positional encoding. and voice timestamp sequences ; is the total length of the acoustic downsampling frames; h is the downsampling frame index; i is the frame index of the subframe;

[0010] S3, Visual gesture encoding: Encoding the t-th video frame... Patch unrolling and local attention encoding are performed to obtain frame-level spatial features for each frame. For video frames The gesture bounding boxes detected in the frame-level spatial features Extract the region vector and average it with the video frame. Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Build frame-level gesture tokens Then, for frame-level spatial features... and frame-level gesture tokens Perform exponential smoothing and downsampling by block to obtain the block-level scene token for each frame. and block-level gesture tokens Finally, the block-level scene token and block-level gesture tokens Mapping to multimodal common dimension Then, shared location coding is superimposed to obtain the scene visual embedding sequence. Gesture visual embedding sequence and video timestamp sequence ; t is the video frame index;

[0011] S4. Construct a time raster timestamp sequence The three types of modal block sequences are aligned to the same common temporal index. Then, attention convergence is performed on each modality within each temporal grid to obtain the intra-modal representation. Subsequently, the intra-modal representations are concatenated using an existence mask, linearly transformed, and then weighted and fused into a grid-level fused token. The position and segment embeddings are then input into a unified Transformer encoder to obtain the fused temporal latent variable H and the global fusion vector. ;in, n is the alignment sequence length; n is the time raster index; For time raster timestamps;

[0012] S5. First, use the global fusion vector. The query vector is constructed and attention aggregation is performed on the temporal latent variable H to obtain the context vector c. Then, the global fusion vector is... After concatenation and projection with the context vector c, a parallel multi-task head is constructed to output the classification log odds z and parametric regression. The log-odds distribution z was smoothed in the time domain and then exponentially normalized to obtain a smoothed probability distribution. Finally, based on the smooth probability distribution Calculate the intention confidence q and the predicted category. Predicting entropy confidence margin , and parametric regression and time raster timestamp The director's multimodal intent result is generated for each time grid, resulting in the latest director's multimodal intent result. .

[0013] Specifically, step S1 includes the following steps:

[0014] S11. Noise Generation Hypothesis and Variable Definition: Setting the Additive Observation Model Mixed speech waveforms Restore to target clean voice It is described as a denoising problem affected by noise disturbance; among which, Additive noise; N is the number of sampling points;

[0015] S12. Prior update of diffusion backsampling: At each time step d, the diffusion denoising network is called. Output the reverse state The injected noise is estimated, and the inverse mean is calculated based on the noise estimate. The denoising update, which serves as the diffusion prior, is shown in the following equation:

[0016]

[0017] Among them, reverse mean As a priori update result; For injecting noise estimation; The first diffusion scheduling coefficient, This is the second diffusion scheduling coefficient; The cumulative product of the first diffusion scheduling coefficients;

[0018] S13. Gradient guidance for observation consistency: Construct the gradient of the log-conditional likelihood against the inverse mean based on the lightweight observation likelihood model. After correction, a guided backward mean with consistent observations is obtained. As shown in the following formula:

[0019] ,

[0020] in, This represents a lightweight observational likelihood model; the parameters are... Used to indicate the gradient direction; For guiding strength;

[0021] S14. Random Sampling and Stepping: At each time step d, the backward mean is obtained by applying a Gaussian distribution with variance after guidance. The next reverse state is obtained by random sampling. The enhanced speech waveform is eventually obtained through continuous reverse iteration. .

[0022] Specifically, step S2 includes the following steps:

[0023] S21. Framing and Time-Frequency Representation: Based on frame shift... Window length and Hanning Window Enhanced speech waveform The complex spectrum is obtained by dividing the frame into frames and calculating the short-time Fourier transform of each frame in the forward direction. Then according to the complex spectrum Calculate the power spectrum The frame-level acoustic vector is obtained by taking the logarithm after Mel filtering. Then, for all frames, the frame-level acoustic vectors Normalized frame-level features are obtained by mean normalization. This leads to the normalized frame-level feature sequence. ;

[0024] S22. Soft speech gating and silence suppression: Based on power spectrum Calculate the acoustic energy of each frame and construct the soft-gating coefficients. Then, the normalized frame-level feature sequence Each normalized frame-level feature in the data is gated to obtain a gated frame-level acoustic vector sequence. ; The total number of frames. ;

[0025] S23, Frame Stacking and Downsampling: Gated Frame-Level Acoustic Vector Sequences S consecutive frames are stacked, and downsampling with a step size of S is used to obtain the stacked block-level acoustic vector. and the corresponding cross-modal alignment timestamp This yields a block-level acoustic vector sequence. and timestamp sequence ; The total length of the acoustic downsampling frames;

[0026] S24, Common Dimension Projection and Position Encoding: Block-level acoustic vectors are encoded using two-layer projection. Mapping to multimodal common dimension And by introducing element-wise nonlinearity, a block-level common vector is obtained. Then the block-level common vector With position encoding Adding them together yields the speech semantic embedding. Finally, the speech semantic embedding sequence is obtained by summarizing. and voice timestamp sequence .

[0027] Specifically, step S3 includes the following steps:

[0028] S31, Spatial Patching and Local Attention Coding: Transforming the t-th video frame... Unfold the patches to obtain P patch vectors, and then perform linear embedding on these patch vectors to obtain the patch embedding. Then embed the patch The data is divided into several patch windows. After calculating the attention of each patch window, the data is stitched together and passed through a feedforward layer to obtain the frame-level spatial features of each frame. Finally, the attention outputs of all patch windows are concatenated and passed through a feedforward layer to obtain the frame-level spatial features of each frame. ;

[0029] S32, Gesture Structuring: Structuring video frames Multiple gesture frames detected in From frame-level spatial features using ROI alignment operators Extracting region vectors and aggregating them by average yields the region aggregate vector. And based on video frames Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Then, the three elements are vectorized and projected to obtain the skeleton structure fusion feature. Finally, the region aggregation vector is... Features integrated with skeletal structure Frame-level gesture tokens are obtained through fusion. ;

[0030] S33. Short-time dynamic smoothing and temporal downsampling: Frame-level spatial features and frame-level gesture tokens After exponential smoothing and attention-weighted convergence, scene-level tokens are obtained. and smooth frame-level gesture tokens Then, block-level downsampling is performed on both to obtain the block-level scene token for each frame. and block-level gesture tokens And calculate video timestamps The video timestamp sequence is obtained. ;in, For video frame rate; is the video stacking factor; k is the video block index; Reduce the number of sampling blocks for the scene; Total number of frames;

[0031] S34, Public Dimension Projection and Location Encoding: Block-level Scene Tokens and block-level gesture tokens Mapping to multimodal common dimension Then encode the same shared location. Obtain scene visual embedding and gesture visual embedding Finally, the scene visual embedding sequence is obtained. and gesture visual embedding sequence .

[0032] Specifically, step S4 includes the following steps:

[0033] S41. Unified Timeline and Index Mapping: Constructing a Time Raster Timestamp Sequence Nearest neighbor alignment is used to separate the speech timestamp sequences of the three types of modal block sequences. and video timestamp sequence Mapping to time index yields audio index mapping and video index mapping The three types of modal blocks include speech-semantic embedding sequences. Scene visual embedding sequence and gesture visual embedding sequence ;

[0034] S42. Intramodal Attention Convergence: For each time grid n, construct the modal aggregation set and apply the attention convergence operator. The three-modal intra-representation is obtained; the three-modal intra-representation includes the raster-level speech intra-representation. Grid-level scene representation and grid-level gestures represent ;

[0035] S43. Modal Soft-Gated Fusion: Obtaining the fused logic value by concatenating the internal representatives of the three modalities through an existence mask and performing a linear transformation. We assign a minimum probability to the empty mode and normalize it to obtain the weights. Obtain the grid-level fusion token ;

[0036] S44, Segment and Position Embedding, Sequence Construction: Based on Grid-Level Fusion Tokens For each time grid n, the segment and position encoding are superimposed to construct the same input. And construct a unified input sequence Z;

[0037] S45. Unified Transformer Coding: Multi-head self-attention and feedforward layer stacking coding are applied to the unified input sequence Z to obtain the fused temporal latent variable H and the global fusion vector. .

[0038] Specifically, step S5 includes the following steps:

[0039] S51. Context-aware attention aggregation: After key-value transformation of the temporal latent variable H, it is combined with the globally fused vector. Construct latent variable queries to jointly calculate latent variable attention weights The context vector c is constructed.

[0040] S52, Fusion Representation Construction and Multi-Task Header Output: Constructing the global fusion vector... The fused representation is obtained by concatenating and projecting the fused representation with the context vector c. And according to the fusion representation Construct parallel classification head and parametric regression head outputs to obtain the classification log odds z, classification probability p, and parametric regression results. ;

[0041] S53. Temporal Smoothing: The log-odds ratio z of the classification is smoothed using an exponential moving average to obtain the smoothed log-odds ratio s, which is then obtained by exponential normalization to obtain the smoothed probability distribution. ;

[0042] S54. Confidence Level Calibration and Uncertainty Measurement: Based on Smooth Probability Distribution Construct the temperature-calibrated intention confidence q and the predicted category And according to the smooth probability distribution Constructing prediction entropy confidence margin ;

[0043] S55. Structured Intent Output: Predicted Category parametric regression Intent confidence q, prediction entropy Confidence margin With time raster timestamp The director's multimodal intent result is obtained by packaging the data into n time grids and outputting each time grid. This yields the latest director's multimodal intent result. .

[0044] Specifically, in step S21, based on the frame shift... Window length and Hanning Window Enhance the speech waveform using the following formula The complex spectrum is obtained by dividing the frame into frames and calculating the short-time Fourier transform of each frame in the forward direction. :

[0045]

[0046] Where f is the frequency index. ; The number of positive frequency spectral lines; The total number of frames. ; is the total number of discrete frequency points used in the short-time Fourier transform operation; n is the index of the frame sample point; N is the number of sampling points; e is the base of the natural logarithm; j is the imaginary unit; i is the frame index of the frame, i=1,..., .

[0047] Specifically, in step S21, based on the complex spectrum... Calculate the power spectrum The frame-level acoustic vector is obtained by taking the logarithm after Mel filtering. As shown in the following formula:

[0048]

[0049] in, This is the Mel filter matrix. is the numerical stability constant; M is the Mel frequency band number.

[0050] Specifically, in step S22, based on the power spectrum... Calculate the acoustic energy of each frame and construct the soft-gating coefficients. Specifically:

[0051] First, based on the power spectrum Calculate the acoustic energy of the i-th frame Then based on acoustic energy Constructing soft gating coefficients As shown in the following formula:

[0052] ,

[0053] in, The slope; The energy reference is used; the Sigmoid function is a sigmoid activation function.

[0054] Specifically, in step S33, the frame-level spatial features are analyzed. and frame-level gesture tokens After exponential smoothing and attention-weighted convergence, scene-level tokens are obtained. and smooth frame-level gesture tokens Then, block-level downsampling is performed on both to obtain the block-level scene token for each frame. and block-level gesture tokens Specifically, it includes:

[0055] Frame-level spatial features Exponential smoothing is used to suppress frame-by-frame jitter, and attention-weighted convergence is used to obtain scene-level tokens. :

[0056] ,

[0057] in, To control the smoothing intensity; Weighted convergence of attention; The frame-level token for the previous frame;

[0058] Simultaneously, frame-level gesture tokens Perform exponential smoothing to obtain smoothed frame-level gesture tokens :

[0059] ,

[0060] in, For smooth frame-level gesture tokens from the previous frame;

[0061] Then for scene frame-level tokens and smooth frame-level gesture tokens After performing block-level downsampling, the average of all block results is used to obtain the block-level scene token for each frame. and block-level gesture tokens As shown in the following formula:

[0062] ,

[0063] in, Here, k is the video stacking factor, and k is the video block index. , Reduce the number of sampling blocks for the scene.

[0064] This invention provides a method for director intent recognition based on diffuse speech enhancement and visual gesture coding. The invention employs a pipeline approach of "reconstruction, representation, and unification": First, the acquired director commands are subjected to diffuse denoising in the time domain. Prior information is used to gradually approximate clean speech, and the sampling trajectory is corrected using observation consistency gradients, thereby recovering a highly intelligible acoustic signal under unknown and time-varying noise. Then, the enhanced speech is transformed into robust time-frequency semantic embeddings, and frame stacking and downsampling are aligned to a temporal granularity comparable to video. On the visual side, each frame is processed using patching and local attention to extract fine-grained features. Combined with ROI alignment of the hand / arm / head and skeletal key points, a structured gesture representation is constructed, and short-window smoothing and block-level downsampling are used to suppress jitter. In the fusion stage, when speech and gesture blocks are paired using a unified temporal grid, intramodal attention convergence is performed within the same grid to highlight key evidence, followed by soft gating. The invention employs adaptive weighting to generate a single multimodal token. Finally, the temporal multimodal sequence, along with its position and segment, is embedded into a unified Transformer for contextual modeling. Parallel classification and regression heads are used to output intent categories and continuous parameters. Furthermore, exponential moving averages based on log-odds and temperature scaling are used to achieve temporal stability and confidence calibration, thus providing a continuous and reliable multimodal intent result for the director in complex live streaming environments. Therefore, this invention enables collaborative understanding and reliable discrimination of the director's voice commands and gesture signals in entertainment group live streaming scenarios without relying on specific device control. It outputs more stable, comparable, and interpretable director intent results, solving the problems of large short-term fluctuations in commands and gestures, strong noise interference, and cross-modal asynchrony leading to unstable intent in complex and ever-changing entertainment group live streaming environments. This further reduces the difficulty and cost of live streaming production.

[0065] Furthermore, this embodiment, through diffusion-based speech enhancement and observation consistency guidance, significantly improves the intelligibility and detail fidelity of the director's commands by generating priors and correcting the sampling trajectory under unknown, non-stationary, and strong noise conditions, providing a stable acoustic foundation for subsequent semantic discrimination.

[0066] Furthermore, this embodiment provides structured visual gesture coding, which explicitly integrates the ROI alignment of the hand / arm / head with the skeletal key points and short-term motion volume, and combines local attention with temporal smoothing / block downsampling to form a more discriminative and shaky gesture representation for actions such as "pointing, gesturing, and nodding".

[0067] Furthermore, this embodiment provides trimodal alignment and gating fusion of a unified temporal grid, aligning speech and gesture blocks with a common time axis, performing attention convergence within the grid, and then adaptively weighting by evidence reliability using soft gating. Through unified Transformer context modeling and confidence calibration with temperature scaling, it outputs stable, comparable, and interpretable director intent results. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of a director's intent recognition method based on diffusion-based speech enhancement and visual gesture coding, provided in an embodiment of the present invention. Detailed Implementation

[0070] The present invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0071] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0072] Example 1

[0073] refer to Figure 1 This embodiment provides a method for director intent recognition based on diffusion-based speech enhancement and visual gesture coding, including the following steps:

[0074] S1. Speech Enhancement and Diffusion Denoising: An additivity observation model is set for the mixed speech waveform y. An injection noise estimate is given through a diffusion denoising network, and an inverse mean is formed as the speech prior. Then, the inverse mean is corrected by constructing the gradient of the log-conditional likelihood based on the lightweight observation likelihood model to obtain the guided inverse mean. The next reverse state is obtained by randomly sampling it using a Gaussian distribution. The enhanced speech waveform is obtained by continuously performing reverse iterations. ;

[0075] The mixed speech waveform y is obtained by directly extracting the original audio channel of the same live / recorded material from the shooting terminal / directing system while acquiring video; this audio contains target speech and environmental noise, hence the term mixed speech. It is acquired synchronously from the same source as the video acquired in step S3 and shares a unified time reference (same sampling time / timestamp).

[0076] S11. Noise Generation Hypothesis and Variable Definition: Setting the Additive Observation Model Mixed speech waveforms Restore to target clean voice It is described as a denoising problem affected by noise disturbance; among which, The noise is additive (unknown, possibly non-Gaussian / non-stationary); N is the number of sampling points.

[0077] To clarify the objectives and constraints, an additivity observation model is first presented. , will "restore The problem is described as a denoising problem under noise disturbance, and this assumption provides a unified modeling basis for the subsequent introduction of diffusion priors.

[0078] S12. Prior update of diffusion backsampling: At each time step d, the diffusion denoising network is called. Output the reverse state The injected noise is estimated, and the inverse mean is calculated based on the noise estimate. The denoising update, which serves as the diffusion prior, is shown in the following equation:

[0079]

[0080] Among them, reverse mean As a priori update result; For injecting noise estimation; The first diffusion scheduling coefficient, is the second diffusion scheduling coefficient; both are scalars. The first diffusion scheduling coefficient cumulative product (scalar); time step d is the time step of the back diffusion;

[0081] Reverse state This refers to the speech sample / intermediate result at step d in the back-diffusion process. It is not the final enhanced speech, but rather a process quantity that gradually works backward from a "noisier state" to a "cleaner state": it decreases as d=D,...,1,0. It will be continuously updated, and the enhanced speech will be obtained when d=0. D is the total number of time steps in the diffusion denoising process. It belongs to the preset hyperparameter of the diffusion schedule. Its value is generally determined according to the expected enhancement quality and real-time requirements: the larger D is, the more complete the denoising is, but the computational cost is higher; the smaller D is, the faster the speed, but the enhancement effect may decrease.

[0082] The diffusion denoising network A generative model-based denoising method adaptively removes noise through a framework guided by generative priors and observation consistency. Its input is... The output is a noise estimate. And used to calculate the reverse mean. The noise is obtained through pre-training; the diffusion denoising network tells the system "where the noise is and what it looks like" at each step, allowing the system to gradually denoise and restore the speech in the correct direction;

[0083] S13. Gradient guidance for observation consistency: To avoid "prior bias dominating" when the noise distribution is unknown and time-varying, a gradient of the log-conditional likelihood with respect to the inverse mean is constructed based on the lightweight observation likelihood model. After correction, a guided backward mean with consistent observations is obtained. As shown in the following formula:

[0084] ,

[0085] in, This represents a lightweight observation likelihood model (with parameters of...). ), which is a probability model for observing y given a signal x, with parameters as follows: It is only used to provide the gradient direction, not the complete reconstruction; For guidance strength (scalar, regulated by d), the heavier the noise (the larger d), the more dominant the prior; the lighter the noise (the smaller d), the greater the proportion of observation guidance.

[0086] The gradient of the log-conditional likelihood based on the mixed speech y is shown in the following equation:

[0087] ,

[0088] in, The gradient with respect to the variable x is expressed by the above formula. The value at x is a vector operator representing the derivative of the function with respect to the input variable x. Specifically, log-likelihood function The gradient with respect to x measures the input signal. The degree of impact on the output. In this invention, it is used to guide the optimization process of the diffusion denoising network, helping the model adjust the generated denoising process based on the current observation (i.e., the enhanced speech signal), thereby achieving the goal of restoring clear speech.

[0089] Since the on-site noise distribution is unknown and time-varying, a lightweight observation likelihood model is used to provide a direction "towards greater observational consistency" in order to avoid prior bias from dominating.

[0090] The Lightweight Observation Likelihood Model is a model for quickly evaluating the consistency between noisy samples and observed signals, specifically designed to guide the diffusion denoising process. It is a lightweight network designed according to specific application requirements, pre-trained, with noisy signals as input and consistency evaluation values ​​as output. This model guides noise removal during the generation process by calculating the likelihood function gradient of the input signal, thereby improving speech reconstruction quality.

[0091] S14. Random Sampling and Stepping: At each time step d, the backward mean is obtained by applying a Gaussian distribution with variance after guidance. The next reverse state is obtained by random sampling. The enhanced speech waveform is eventually obtained through continuous reverse iteration. ;

[0092] To balance prior diversity and reconstruction stability, the backward mean after guidance is calculated according to a Gaussian distribution with variance at each time step d. The next reverse state is obtained by random sampling. (That is, the reverse process "pushing forward one step")

[0093]

[0094] in, This represents the speech state (reverse state) at time step d. Perform Gaussian sampling; This represents the variance of the Gaussian distribution at that step, a scalar. This is a hyperparameter used to control the noise level. It is adjusted as the time step changes to ensure the stability of the sampling process. Let be the identity matrix, and let be the covariance matrix of the Gaussian distribution, which here represents the standard normal distribution.

[0095] After the reverse iteration is completed (gradually from d=D to d=0), the final result will be... As an enhancement of speech waveform Output, N is the number of sampling points;

[0096] Understandable, Represents the set of real numbers. Represents the set of complex numbers. It represents the set of natural numbers, which is common knowledge. All similar expressions used thereafter will have this meaning, and will not be elaborated here.

[0097] Since the director's commands are often given in environments with strong noise and non-stationary interference (audience noise, background music noise, walkie-talkie crosstalk, etc.), traditional spectral subtraction / thresholding methods are prone to "over-erasure" and decreased intelligibility. In order to stably recover the speech backbone under unknown noise distribution, this step adopts a backsampling framework of "diffusion prior + observation consistency guidance": first, a diffusion denoising network is used to provide a prior for generating clean speech, and then a lightweight likelihood model is used to measure the consistency gradient of the current sample with respect to the observation and correct the sampling mean, thereby suppressing over-smoothing and artifacts under complex noise and gradually reconstructing high-fidelity speech.

[0098] S2, Speech semantic coding: Enhancing speech waveforms The complex spectrum is obtained by framing and performing a short-time Fourier transform. Then, the power spectrum is calculated based on the complex spectrum, and the logarithm is obtained by Mel filtering. Normalized frame-level features are obtained by mean normalization. Then, normalized frame-level features Gating is performed to obtain the gated frame-level acoustic vector. Then, by aligning the visual temporal granularity through frame stacking and downsampling, block-level acoustic vectors are obtained. Then, it is projected onto the multimodal common dimension through two layers of projection. The speech semantic embedding sequence is obtained by superimposing positional encoding. and voice timestamp sequences ; is the total length of the acoustic downsampling frames; h is the downsampling frame index; i is the frame index of the subframe;

[0099] To preserve speech structure consistent with human auditory perception under non-stationary noise and to align it with subsequent visual / gestural inputs on a unified timeline, this step is performed sequentially:

[0100] S21. Framing and Time-Frequency Representation: Based on frame shift... Window length and Hanning Window Enhanced speech waveform The complex spectrum is obtained by dividing the frame into frames and calculating the short-time Fourier transform of each frame in the forward direction. :

[0101]

[0102] Where f is the frequency index. ; The number of positive frequency spectral lines, and ; The total number of frames. Hanning Window ; is the total number of discrete frequency points used in the Short-Time Fourier Transform (FFT) operation; n is the frame sample point index; N is the number of sampling points; e is the base of the natural logarithm; j is the imaginary unit; i is the frame index, i=1,..., ;

[0103] According to the complex spectrum Calculate the power spectrum The frame-level acoustic vector is obtained by taking the logarithm after Mel filtering. :

[0104]

[0105] in The Mel filter matrix (M is the number of Mel frequency bands, i.e., the acoustic feature dimension) ), is the numerical stability constant.

[0106] This leads to the frame-level acoustic vector sequence of all frames. Then, for all frames, the frame-level acoustic vectors Perform mean normalization (CMVN) to obtain normalized frame-level features. :

[0107] ,

[0108] in, Squaring element by element; To prevent division by zero of extremely small constants; The frame-level acoustic vector mean. ; Standard deviation, ; The dimension of the normalized frame-level features;

[0109] This leads to the normalized frame-level feature sequence. : ;

[0110] S22, Soft speech gating and silent suppression (energy-driven soft gating): based on power spectrum Calculate the acoustic energy of each frame and construct the soft-gating coefficients. Then, the normalized frame-level feature sequence Each normalized frame-level feature in the data is gated to obtain a gated frame-level acoustic vector sequence. ;

[0111] First, based on the power spectrum Calculate the acoustic energy of the i-th frame Then based on acoustic energy Constructing soft gating coefficients :

[0112] ;

[0113] in, The slope The energy baseline (which can be estimated from the silent segment); The number of positive frequency spectral lines; the Sigmoid function is a sigmoid activation function used to map the output of linear regression to the range (0,1);

[0114] This gating continuously suppresses the interference of silent / music / noise-dominant frames on subsequent semantic representations, avoiding information breaks caused by hard thresholding.

[0115] Then based on the soft gating coefficient For normalized frame-level feature sequences Normalized frame-level features Gated suppression is performed to obtain the gated frame-level acoustic vector. This leads to the gated frame-level acoustic vector sequence. ;

[0116] S23, Frame Stacking and Downsampling (Constructing Block-Level Features and Generating Aligned Timestamps): Applying gated frame-level acoustic vector sequences... S consecutive frames are stacked, and downsampling with a step size of S is used to obtain the stacked block-level acoustic vector. and the corresponding cross-modal alignment timestamp This yields a block-level acoustic vector sequence. and timestamp sequence ; The total length of the acoustic downsampling frames;

[0117] Define a stacking factor S, where S is a positive integer greater than or equal to 5; in this example, S is 5. Define the downsampling frame index. Total length of acoustic downsampling frames Then, the gated frame-level acoustic vector sequence The stacked block-level acoustic vector is obtained by stacking S consecutive frames and downsampling with a step size of S. :

[0118] ,

[0119] It also provides the speech timestamps required for cross-modal alignment. (Indicates the start time of the stacked block, in seconds):

[0120] ,

[0121] Finally, a block-level acoustic vector sequence is obtained. and voice timestamp sequences ;

[0122] Gated frame-level acoustic vector sequences Stacking consecutive S frames within a block involves sequentially concatenating / stacking the acoustic vectors of adjacent S frame-level vectors into a single block-level vector. (Enhancing local semantics and improving noise resistance); Downsampling with a stride of S selects the starting point of the stacked window between blocks with a stride of S (i.e., non-overlapping blocks with a window length of S and a stride of S), thereby achieving downsampling, shortening the sequence length, and aligning with the temporal granularity of the visual blocks. Overlapping sliding windows will only occur if the stride is less than S; the purpose of setting the stride to S here is to avoid overlap and reduce computation.

[0123] S24, Common Dimension Projection and Location Encoding (Mapping to) (and preserve the intra-block order): block-level acoustic vectors are projected through two layers. Mapping to multimodal common dimension And by introducing element-wise nonlinearity, a block-level common vector is obtained. :

[0124]

[0125] in, Indicates element-wise nonlinearity (e.g., GELU); The intermediate vector of the projection; Let be the first projection matrix. This is the first bias vector; This is the second projection matrix. This is the second bias vector; The parameters obtained through training; This is a hidden dimension in the middle. For multimodal common dimensions; This is the block-level common vector after the two layers of projection;

[0126] Then the block-level common vector With position encoding Adding them together yields the speech semantic embedding. :

[0127] ,

[0128] in, For position encoding functions;

[0129] Finally, the speech and semantic embedding sequences are obtained. and voice timestamp sequence ;

[0130] Since subsequent steps require fusing voice and video gestures on a unified timeline and extracting stable semantic rather than acoustic representations related to the director's commands, this step first transforms the temporal signal into perceptually consistent time-frequency features (facilitating noise reduction and modeling of pronunciation structure). Then, it aligns the visual temporal granularity and reduces sequence length using frame stacking and downsampling. Finally, it maps the features to a multimodal common dimension using linear-nonlinear projection and overlays positional encoding to preserve temporal information. This approach balances noise robustness and real-time performance while providing a unified input for the three-modal alignment in S4 and intent decoding in S5.

[0131] S3, Visual gesture encoding: Encoding the t-th video frame... Patch unrolling and local attention encoding are performed to obtain frame-level spatial features for each frame. For video frames The gesture bounding boxes detected in the frame-level spatial features Extract the region vector and average it with the video frame. Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Build frame-level gesture tokens Then, for frame-level spatial features... and frame-level gesture tokens Perform exponential smoothing and downsampling by block to obtain the block-level scene token for each frame. and block-level gesture tokens Finally, the block-level scene token and block-level gesture tokens Mapping to multimodal common dimension Then, shared location coding is superimposed to obtain the scene visual embedding sequence. Gesture visual embedding sequence and video timestamp sequence ;

[0132] S31, Spatial Patching and Local Attention Coding: Transforming the t-th video frame... Unfold the patches to obtain P patch vectors, and then perform linear embedding on these patch vectors to obtain the patch embedding. Then embed the patch The data is divided into several patch windows. After calculating the attention of each patch window, the data is stitched together and passed through a feedforward layer to obtain the frame-level spatial features of each frame. Finally, the attention outputs of all patch windows are concatenated and passed through a feedforward layer to obtain the frame-level spatial features of each frame. ;

[0133] Obtain video frame sequence It is a video frame obtained by sampling at time step t. The sequence formed, The total number of frames is H, the video frame height is W, the video frame width is 3, and 3 represents the RGB channels; the video frame rate is... The unit is Hz;

[0134] The video frame sequence It is the footage captured by the gesture capture camera set up at the director's seat in the live / recorded broadcast system, used to capture the director's command gestures, postures and key actions;

[0135] Video frame sequence It provides visual evidence of the director's gestures, and through S3 patch encoding and attention modeling, it obtains gesture semantic features, which serve as one of the key modalities for director intent recognition. Video frame rate. Its function is to provide a time sampling reference and timestamp conversion (mapping the frame index t to the real time), which is used for time alignment with modalities such as audio, building a common time grid and realizing multimodal synchronous fusion.

[0136] The t-th video frame Unfold the patches to obtain P patch vectors, and then perform linear embedding on these patch vectors to obtain the patch embedding. :

[0137] ,

[0138] in, For patchify operators, ; The dimension of the patch vector; For visual embedding dimensions; For patch projection matrix, This is the patch bias vector, used to assign each patch vector (dimension) to the corresponding element. Linear mapping to visual embedding dimension ;

[0139] Then embed the patch The data is divided into several patch windows. After calculating the attention of each patch window, the data is stitched together and passed through a feedforward layer to obtain the frame-level spatial features of each frame. ,

[0140] Specifically, embedding the patch Divide into several tile windows, and denote the w-th tile window. Internal query ,key ,value for:

[0141]

[0142] in, Query the matrix for the patch window; For patch window key matrix; This is the patch window value matrix; And perform attention calculations based on the patch window:

[0143]

[0144] Where M is the local mask. For each head dimension;

[0145] Finally, the attention outputs of all patch windows are concatenated and processed through a feedforward layer to obtain the frame-level spatial features of each frame. ;

[0146] It is worth noting that in this step These are the model parameters obtained through training; here This represents the attention "per head dimension," which is related to the projected hidden dimension in S2. They do not mean the same thing.

[0147] S32, Gesture Structuring: Structuring video frames Multiple gesture frames detected in From frame-level spatial features using ROI alignment operators Extracting region vectors and aggregating them by average yields the region aggregate vector. And based on video frames Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Then, the three elements are vectorized and projected to obtain the skeleton structure fusion feature. Finally, the region aggregation vector is... Features integrated with skeletal structure Frame-level gesture tokens are obtained through fusion. ;

[0148] video frames The collection of gesture boxes in The results are obtained from existing hand / human keypoint detection and target detection technologies; specifically, a deep learning-based gesture / human detection network (such as hand detection or human detection + hand region thinning) can be used to output the bounding box coordinates and confidence score of the hand target in each frame, and R valid gesture boxes are obtained through thresholding and NMS deduplication; each gesture box For the 4D bounding box parameters, the joint represents the bounding box formed by the top left / bottom right pixel coordinates, and R represents the number of gesture boxes;

[0149] For the t-th video frame R gesture boxes detected in Using ROI alignment operators to extract frame-level spatial features Extracting region vectors:

[0150]

[0151] The average aggregation of multiple regional features is then performed to obtain the regional aggregation vector. :

[0152] ,

[0153] video frames Output skeleton keypoint set using keypoint estimator With confidence vector :

[0154]

[0155] in, Here, J represents the two-dimensional coordinate components of the j-th skeleton keypoint, where J is the number of keypoints; confidence level. for The scalar corresponding to the j-th skeletal keypoint in the middle. This is the confidence vector of the key points of the entire skeleton.

[0156] The keypoint estimator is a mature existing technology, such as human / hand keypoint models based on heatmap regression or Transformer. Common implementations include OpenPose, HRNet, MediaPipe / BlazePose, ViTPose, etc. In this invention, it is only used as an external module to provide skeleton information.

[0157] And calculate the short-time displacement of key points in adjacent frames. :

[0158] ,

[0159] The skeleton structure fusion feature is obtained by vectorizing and projecting the keypoint topology (keypoint set) and motion information (short-term displacement of keypoints). :

[0160]

[0161] Where vec() is the matrix vectorization function; Here is the motion projection matrix for key points. The keypoint motion bias vector;

[0162] Finally, the appearance region (region aggregation vector) Features of integration with skeletal structure Frame-level gesture tokens are obtained through fusion. :

[0163] .

[0164] Region aggregation vector Capture local appearance and fine-grained texture of the hand / arm, and integrate skeletal structure features. The encoding topology and kinematics are then combined to obtain a structured gesture token that is more discriminative for actions such as "pointing / gesturing / shaking head".

[0165] S33. Short-time dynamic smoothing and temporal downsampling: Frame-level spatial features and frame-level gesture tokens After exponential smoothing and attention-weighted convergence, scene-level tokens are obtained. and smooth frame-level gesture tokens Then, block-level downsampling is performed on both to obtain the block-level scene token for each frame. and block-level gesture tokens ;

[0166] Frame-level spatial features Exponential smoothing is used to suppress frame-by-frame jitter, and attention-weighted convergence is used to obtain scene-level tokens. :

[0167] ,

[0168] in, To control the smoothing intensity; Weighted convergence of attention (suppressing background overlays and highlighting gesture-related areas); The frame-level token for the previous frame;

[0169] Simultaneously, frame-level gesture tokens Perform exponential smoothing to obtain smoothed frame-level gesture tokens :

[0170] ,

[0171] in, For smooth frame-level gesture tokens from the previous frame;

[0172] To maintain consistency with the temporal granularity of the audio block (S2), scene frame-level tokens are used. and smooth frame-level gesture tokens After performing block-level downsampling, the average of all block results is used to obtain the block-level scene token for each frame. and block-level gesture tokens :

[0173] ,

[0174] in, Here, k is the video stacking factor, and k is the video block index. , Reduce the number of sampling blocks for the scene:

[0175] And calculate video timestamps Used for voice timestamps output by S2 Align in S4:

[0176]

[0177] Obtain the video timestamp sequence ;

[0178] After smoothing and blockification, the length of the visual sequence is significantly shortened, reducing the impact of transient jitter on fusion and decoding, and naturally matching the audio timescale.

[0179] S34, Public Dimension Projection and Location Encoding: Block-level Scene Tokens and block-level gesture tokens Mapping to multimodal common dimension Then encode the same shared location. Obtain scene visual embedding and gesture visual embedding Finally, the scene visual embedding sequence is obtained. and gesture visual embedding sequence .

[0180] First, map the two tokens to the multimodal common dimension. Obtain block-level scene embedding and block-level gesture embedding (Consistent with S2):

[0181]

[0182] in, For block-level scene projection matrix, This is a block-level scene bias vector; For block-level gesture projection matrix, This is the block-level gesture bias vector; , ;

[0183] Embedding the same block-level scene again and block-level gesture embedding Shared location coding Obtain scene visual embedding and gesture visual embedding :

[0184]

[0185] in, For position encoding functions;

[0186] Two visual embedding sequences are obtained: scene visual embedding sequence. and gesture visual embedding sequence Output the video timestamp sequence. .

[0187] Due to the presence of strong lighting changes, occlusion, and rapid gestures in entertainment live broadcasts, this step adopts a four-stage process to stably extract visual evidence related to "director's intent" under complex conditions: "spatial encoding → gesture structuring → short-term dynamics and downsampling → common dimension projection and temporal encoding". First, spatial features of each frame are robustly extracted using patching and local attention. Then, gesture regions are explicitly structured using ROI alignment and skeleton keypoints. Subsequently, short-window temporal smoothing and downsampling are used to reduce jitter and computational burden. Finally, the representation is mapped to the multimodal common dimension and positional encoding is added to preserve the temporal structure, thereby providing a clear, robust, and length-controllable visual sequence representation for subsequent multimodal alignment and intent decoding.

[0188] S4. Construct a time raster timestamp sequence The three types of modal block sequences are aligned to the same common temporal index. Then, attention convergence is performed on each modality within each temporal grid to obtain the intra-modal representation. Subsequently, the intra-modal representations are concatenated using an existence mask, linearly transformed, and then weighted and fused into a grid-level fused token. The position and segment embeddings are then input into a unified Transformer encoder to obtain the fused temporal latent variable H and the global fusion vector. ;

[0189] S41. Unified Timeline and Index Mapping: Constructing a Time Raster Timestamp Sequence Nearest neighbor alignment is used to separate the speech timestamp sequences of the three types of modal block sequences. and video timestamp sequence Mapping to time index yields audio index mapping and video index mapping Thus, they are aligned to the same temporal grid; the three types of modal blocks include speech-semantic embedding sequences. Scene visual embedding sequence and gesture visual embedding sequence ;

[0190] Define a fixed time grid step , ,

[0191] in, This is the time step for the speech block; The time step of the video block;

[0192] and The time steps for audio and video blocks are respectively, and can be directly calculated from "sampling rate / frame rate + block stacking factor": the audio side is obtained from the frame window length / frame shift and the audio block stacking length (such as the number of frames contained in each block). The video side is determined by the video frame rate. With video stacking factor get .therefore This is used to select a finer temporal granularity as the uniform temporal grid step size.

[0193] Let the length of the alignment sequence be The center time of the nth time grid is the time grid timestamp. , Establish a unified timeline, namely a time grid timestamp sequence. n is the time raster index;

[0194] This represents the number of grid cells that can be covered on the unified timeline, determined by the total coverage duration and grid step size during fusion: Let the fusion coverage duration be... (The length of the common coverage interval of the three-modal timestamps can be taken), then

[0195]

[0196] Based on this, a time raster timestamp is constructed. ; ;

[0197] Nearest neighbor alignment is used to map the timestamps of the three types of modal block sequences to the same time index:

[0198] ,

[0199] in, A sequence of voice timestamps; A video timestamp sequence;

[0200] This assigns the k-th audio block and video block to time grid n, aligning them to the same time index; since there are a total of time grids on the unified time axis... Each grid point naturally has an index range of [number] points. Therefore, the mapping result must fall within this set (in engineering implementation, out-of-bounds values ​​can also be truncated). (To ensure legality).

[0201] S42. Intramodal Attention Convergence: For each time grid n, construct the modal aggregation set and apply the attention convergence operator. The three-modal intra-representation is obtained; the three-modal intra-representation includes the raster-level speech intra-representation. Grid-level scene representation and grid-level gestures represent ;

[0202] For each time grid n, construct the modal aggregation set:

[0203] ,

[0204] in, A collection of voice data; For scene aggregation collection; For gesture aggregation set;

[0205] Given a set S and a query vector (Can be a learnable global query), define the attention convergence operator. :

[0206] ,

[0207] Based on this, the representation within each modality is obtained:

[0208]

[0209]

[0210] in, Representation within the raster-level speech; Represents the scene at the raster level; This represents a grid-level gesture.

[0211] S43. Modal Soft-Gated Fusion: Obtaining the fused logic value by concatenating the internal representatives of the three modalities through an existence mask and performing a linear transformation. We assign a minimum probability to the empty mode and normalize it to obtain the weights. Obtain the grid-level fusion token ;

[0212] To prevent interference from empty sets, an existence mask is introduced:

[0213] ,

[0214] in, There is a mask for the speech; A mask exists for the scene; A mask is provided for the gesture;

[0215] The fused logic value is obtained by concatenating the three modal internal representations and performing a linear transformation. :

[0216]

[0217] in For modal fusion linear matrix, This is the modality fusion bias vector;

[0218] Weights are obtained by assigning minimum probabilities to empty modes and normalizing them. :

[0219]

[0220] in Let be the numerical stability constant, denoted as ;

[0221] This yields a grid-level fusion token. :

[0222] ;

[0223] Since the reliability of each modality varies in different scenarios, evidence-based soft gating is adopted to enable the system to automatically "trust" the more reliable modality when there is strong noise or occlusion. Soft gating uses learnable weighting to enable the system to automatically select the most credible combination of evidence under conditions such as environmental noise, camera occlusion, and changes in speech rate, so as to avoid the failure of a single modality dragging down the overall judgment.

[0224] S44, Segment and Position Embedding, Sequence Construction: Based on Grid-Level Fusion Tokens For each time grid n, the segment and position encoding are superimposed to construct the same input. And construct a unified input sequence Z;

[0225] For each time grid n, the segment and position codes are superimposed to construct a unified input vector. :

[0226] ,

[0227] in, For grid-level fusion tokens; Encoding the position of time grid n ; For position encoding functions; For global category tokens; Segment vector;

[0228] Global Classification Token During model initialization, it is randomly initialized as a set of trainable parameters. During training, it is optimized together with the task loss through backpropagation, and is used to gather global information of the entire sequence; segment vector. It is also initialized as a trainable parameter and updated during training;

[0229] And construct a unified input sequence ;

[0230] Segment embeddings allow the model to distinguish statistical characteristics from different sources, while positional encoding ensures that temporal relationships can be utilized in self-attention.

[0231] S45. Unified Transformer Coding: Multi-head self-attention and feedforward layer stacking coding are applied to the unified input sequence Z to obtain the fused temporal latent variable H and the global fusion vector. ;

[0232] Since intent determination requires multiple steps of context and cross-modal interaction, multi-layer self-attention encoding is used to capture long-term dependencies and the interaction relationship of "voice ↔ gesture ↔ scene".

[0233] A unified Transformer encoder is used: the unified input sequence Z is encoded by multi-head self-attention and stacked feedforward layers to obtain the fused temporal latent variable H.

[0234] ,

[0235] and then ,

[0236] in, This represents an encoder consisting of a multi-head self-attention layer and a feedforward sublayer stacked together. The attention mask (can be fully connected or set to a lower triangular shape due to timing constraints);

[0237] Then take the global fusion vector. Retain the list of fused vectors ;

[0238] It is understandable that H[0] and h0 are equivalent, just different forms of representation;

[0239] Global fusion vector The globally fused sample-level semantic vectors are used for intent classification and parameter regression in S5. For the fused temporal hidden representation, the temporal resolution is preserved for frame-level / fragment-level intent or uncertainty estimation.

[0240] Since the three modalities are inherently different in sampling rate and segmentation strategy, direct splicing would introduce temporal misalignment and redundancy, thereby weakening the complementarity of intent evidence. In order to fuse and adaptively highlight the modality with the most reliable information at the current moment on a unified time axis, this step adopts the process of "time alignment modeling → intramodal convergence → fusion gating → unified Transformer encoding": First, the speech / visual blocks are aligned to the same index using a fixed time grid. Then, attention convergence is performed on each modality within each time grid to obtain a representative token. Subsequently, the token is fused into a single multimodal token by weighting the evidence strength through soft gating. Finally, the token is input into the unified Transformer along with the position and segment embedding to obtain a context-enhanced trimodal hidden representation (fused temporal latent variable H) for intent decoding in S5.

[0241] S5. First, use the global fusion vector. The query vector is constructed and attention aggregation is performed on the temporal latent variable H to obtain the context vector c. Then, the global fusion vector is... After concatenation and projection with the context vector c, a parallel multi-task head is constructed to output the classification log odds z and parametric regression. The log-odds distribution z was smoothed in the time domain and then exponentially normalized to obtain a smoothed probability distribution. Finally, based on the smooth probability distribution Calculate the intention confidence q and the predicted category. Predicting entropy confidence margin , and parametric regression and time raster timestamp The director's multimodal intent result is generated for each time grid n, resulting in the latest director's multimodal intent result. ;

[0242] S51. Context-aware attention aggregation: After key-value transformation of the temporal latent variable H, it is combined with the globally fused vector. Construct latent variable queries to jointly calculate latent variable attention weights The context vector c is constructed.

[0243] Performing a key-value transformation on the time-series latent variable H yields the time-series latent variable key vector K and the time-series latent variable value vector V:

[0244]

[0245] in, This is the time key transformation matrix. This is the time-series value transformation matrix; The feature dimension (channels) of the temporal latent variable key vector K is the embedding dimension of the Key / Query in the attention mechanism; It is usually set as a configurable hyperparameter and is preset before training.

[0246] Then with the global fusion vector Construct latent variable query vectors and calculate latent variable attention weights :

[0247] ,

[0248] in, For latent variable lookup vectors; Time-series query matrix; softmax is a normalized exponential function;

[0249] The context vector c is constructed as follows: ,

[0250] Among them, latent variable attention weights It is by The resulting attention weight vector for the entire time series, The nth component of this vector at the nth time grid position represents the global query. The intensity of attention to the nth time-series latent variable (the larger the weight, the greater the contribution);

[0251] Time-series latent variable value vector It is by The resulting time-series value vector matrix, This represents the value vector (row n) corresponding to the nth time grid cell in the matrix.

[0252] Therefore, context vector This involves applying attention weights to all time-series value vectors. The aggregate representation obtained by weighted summation.

[0253] S52, Fusion Representation Construction and Multi-Task Header Output: Constructing the global fusion vector... The fused representation is obtained by concatenating and projecting the context vector c according to the following formula. :

[0254]

[0255] in, To fuse the projection matrix, To fuse the bias vector representation For element-wise nonlinearity (such as GELU);

[0256] And according to the fusion representation Construct parallel multi-task heads (classification head and parametric regression head) to output the classification log odds z, classification probability p, and parametric regression results. ;

[0257] Category Header:

[0258]

[0259] in, For the classification head projection matrix, This is the classifier head bias vector. Temperature (see S55 for probability calibration); C represents the number of intent categories;

[0260] Parameter regression head:

[0261]

[0262] in, For the regression head projection matrix, This is the regression head bias vector; This is the regression head dimension; ultimately, the classification log odds are obtained. Classification probability and parametric regression .

[0263] S53. Temporal Smoothing: The log-odds ratio z of the classification is smoothed using an exponential moving average to obtain the smoothed log-odds ratio s, which is then obtained by exponential normalization to obtain the smoothed probability distribution. :

[0264]

[0265] in For smoothing coefficients, The temperature after smoothing. Let be the historical logarithmic odds, representing the cumulative logarithmic odds at the previous time point (s obtained at the previous time point is used as the new logarithmic odds at the next time point). ), used for smooth temporal startup; if it is the first frame, it can be set to ; To smooth the logarithmic odds; To smooth the probability distribution, and using s as the historical log-probability of the next time step (i.e., the new log-probability) ) is used for recursion; softmax() is the normalized exponential function;

[0266] Since the log-odds ratios of classification are numerically more suitable for aggregation, the exponential moving average (EMA) can significantly reduce jitter without sacrificing response speed;

[0267] S54. Confidence Level Calibration and Uncertainty Measurement: Based on Smooth Probability Distribution Construct the temperature-calibrated intention confidence q and the predicted category :

[0268] ,

[0269] in, Indicates the category index Maximum value calculation; Intended confidence level It is the maximum posterior confidence at that moment; This represents the category index C that maximizes the probability of the target; predicting the category. This represents the intent prediction category of the nth time grid (values ​​are...). ); It is a calibrated probability after "log-odds ratio (EMA) + temperature scaling", which is a smoothed probability distribution. The c-th component (and In other words, It is a complete probability distribution. It is the probability of a single category. and These are its position (maximum item index) and size (maximum item value), respectively.

[0270] Based on the smooth probability distribution Constructing an uncertainty index: predicting entropy confidence margin ;

[0271] Predicting entropy (The smaller the value, the more certain it is):

[0272] ,

[0273] Confidence margin (Difference between the top two):

[0274] ,

[0275] in, The highest probability after sorting; The first two most likely after sorting;

[0276] Finally, the intent confidence q and predicted category are obtained after temperature calibration. Uncertainty indicators and .

[0277] S55. Structured Intent Output: Predicted Category parametric regression Intent confidence q, prediction entropy Confidence margin With time raster timestamp The director's multimodal intent result is obtained by packaging the data into n time grids and outputting each time grid. This yields the latest director's multimodal intent result. ;

[0278] ,

[0279] in, Regression of the parameters corresponding to the nth time grid; The intent confidence level for the nth time grid. The prediction entropy / uncertainty measure for the nth time grid. The confidence margin / confidence interval width of the nth time grid;

[0280] Take the result corresponding to the latest time grid index n as the latest director multimodal intent result. .

[0281] It is a structured director's intent instruction package that is output in real time according to a time grid. It is used to drive the automatic decision-making and linkage control of the directing / shooting equipment, such as triggering camera switching, zooming, panning, camera tracking, composition adjustment, and subtitle / effect / lighting cues. It solves the problems of large short-term fluctuations in commands and gestures, strong noise interference, and cross-modal asynchrony leading to unstable intents in live broadcasts.

[0282] Because live commands and gestures have short-range context dependencies and are susceptible to transient jitter, this step, in order to output stable and interpretable directorial intent, first performs "context-aware representation fusion," then "multi-task head (classification + parametric regression)," followed by "temporal smoothing" to suppress jitter, and finally uses "temperature-scaled confidence calibration" to obtain comparable probabilities and confidence levels. This approach utilizes the alignment timing information of S4 while also considering both short-term stability and uncertainty characterization at the output end.

[0283] This embodiment employs a pipeline approach of "reconstruction first, representation second, and unification third": First, the collected director's commands are subjected to diffusion denoising in the time domain. A priori generation is used to progressively approximate clean speech, and the sampling trajectory is corrected using observation consistency gradients, thereby recovering a highly intelligible acoustic signal under unknown and time-varying noise. Then, the enhanced speech is transformed into robust time-frequency semantic embeddings, and frame stacking and downsampling are used to align it to a temporal granularity comparable to the video. On the visual side, each frame is processed using patching and local attention to extract fine-grained features. Combined with ROI alignment of the hand / arm / head and skeletal key points, a structured gesture representation is constructed, and short-window smoothing and block-level downsampling are used to suppress jitter. In the fusion stage, when merging speech and gesture blocks using a unified temporal grid, intramodal attention convergence is first performed within the same grid to highlight key evidence, followed by soft gating and adaptive weighting based on reliability to generate a single multimodal representation. Finally, the temporal multimodal sequence, along with its position and segment, is embedded into a unified Transformer for contextual modeling. Parallel classification and regression heads are used to output intent categories and continuous parameters. Log-odds exponential moving averages and temperature scaling are employed to achieve temporal stability and confidence calibration, thus providing continuous and reliable multimodal intent results for directors in complex live streaming environments. Therefore, this embodiment can achieve collaborative understanding and reliable discrimination of director's voice commands and gesture signals in entertainment group live streaming scenarios without relying on specific device control. It can output more stable, comparable, and interpretable director intent results, solving the problems of large short-term fluctuations in commands and gestures, strong noise interference, and cross-modal asynchrony leading to unstable intent in complex and ever-changing entertainment group live streaming scenarios, further reducing the difficulty and cost of live streaming production.

[0284] Furthermore, this embodiment, through diffusion-based speech enhancement and observation consistency guidance, significantly improves the intelligibility and detail fidelity of the director's commands by generating priors and correcting the sampling trajectory under unknown, non-stationary, and strong noise conditions, providing a stable acoustic foundation for subsequent semantic discrimination.

[0285] Furthermore, this embodiment provides structured visual gesture coding, which explicitly integrates the ROI alignment of the hand / arm / head with the skeletal key points and short-term motion volume, and combines local attention with temporal smoothing / block downsampling to form a more discriminative and shaky gesture representation for actions such as "pointing, gesturing, and nodding".

[0286] Furthermore, this embodiment provides trimodal alignment and gating fusion of a unified temporal grid, aligning speech and gesture blocks with a common time axis, performing attention convergence within the grid, and then adaptively weighting by evidence reliability using soft gating. Through unified Transformer context modeling and confidence calibration with temperature scaling, it outputs stable, comparable, and interpretable director intent results.

[0287] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0288] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0289] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0290] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A method for recognizing director's intent based on diffusion-based speech enhancement and visual gesture coding, characterized in that, Includes the following steps: S1. Speech Enhancement and Diffusion Denoising: An additivity observation model is set for the mixed speech waveform y. An injection noise estimate is given through a diffusion denoising network, and an inverse mean is formed as the speech prior. Then, the inverse mean is corrected by constructing the gradient of the log-conditional likelihood based on the lightweight observation likelihood model to obtain the guided inverse mean. The next reverse state is obtained by randomly sampling it using a Gaussian distribution. The enhanced speech waveform is obtained by continuously performing reverse iterations. ;d represents the time step; S2, Speech semantic coding: Enhancing speech waveforms The complex spectrum is obtained by framing and performing a short-time Fourier transform. Then, the power spectrum is calculated based on the complex spectrum, and the logarithm is obtained by Mel filtering. Normalized frame-level features are obtained by mean normalization. Then, normalized frame-level features Gating is performed to obtain the gated frame-level acoustic vector. Then, by aligning the visual temporal granularity through frame stacking and downsampling, block-level acoustic vectors are obtained. Then, it is projected onto the multimodal common dimension through two layers of projection. The speech semantic embedding sequence is obtained by superimposing positional encoding. and voice timestamp sequences ; is the total length of the acoustic downsampling frames; h is the downsampling frame index; i is the frame index of the subframe; S3, Visual gesture encoding: Encoding the t-th video frame... Patch unrolling and local attention encoding are performed to obtain frame-level spatial features for each frame. For video frames The gesture bounding boxes detected in the frame-level spatial features Extract the region vector and average it with the video frame. Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Build frame-level gesture tokens Then, for frame-level spatial features... and frame-level gesture tokens Perform exponential smoothing and downsampling by block to obtain the block-level scene token for each frame. and block-level gesture tokens Finally, the block-level scene token and block-level gesture tokens Mapping to multimodal common dimension Then, shared location coding is superimposed to obtain the scene visual embedding sequence. Gesture visual embedding sequence and video timestamp sequence ; t is the video frame index; S4. Construct a time raster timestamp sequence The three types of modal block sequences are aligned to the same common temporal index. Then, attention convergence is performed on each modality within each temporal grid to obtain the intra-modal representation. Subsequently, the intra-modal representations are concatenated using an existence mask, linearly transformed, and then weighted and fused into a grid-level fused token. The position and segment embeddings are then input into a unified Transformer encoder to obtain the fused temporal latent variable H and the global fusion vector. ;in, n is the alignment sequence length; n is the time raster index; For time raster timestamps; S5. First, use the global fusion vector. The query vector is constructed and attention aggregation is performed on the temporal latent variable H to obtain the context vector c. Then, the global fusion vector is... After concatenation and projection with the context vector c, a parallel multi-task head is constructed to output the classification log odds z and parametric regression. The log-odds distribution z was smoothed in the time domain and then exponentially normalized to obtain a smoothed probability distribution. Finally, based on the smooth probability distribution Calculate the intention confidence q and the predicted category. Predicting entropy confidence margin , and parametric regression and time raster timestamp The director's multimodal intent result is generated for each time grid, resulting in the latest director's multimodal intent result. .

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Noise Generation Hypothesis and Variable Definition: Setting the Additive Observation Model Mixed speech waveforms Restore to target clean voice It is described as a denoising problem affected by noise disturbance; among which, Additive noise; N is the number of sampling points; S12. Prior update of diffusion backsampling: At each time step d, the diffusion denoising network is called. Output the reverse state The injected noise is estimated, and the inverse mean is calculated based on the noise estimate. The denoising update, which serves as the diffusion prior, is shown in the following equation: ， Among them, reverse mean As a priori update result; For injecting noise estimation; The first diffusion scheduling coefficient, This is the second diffusion scheduling coefficient; The cumulative product of the first diffusion scheduling coefficients; S13. Gradient guidance for observation consistency: Construct the gradient of the log-conditional likelihood against the inverse mean based on the lightweight observation likelihood model. After correction, a guided backward mean with consistent observations is obtained. As shown in the following formula: ， in, This represents a lightweight observational likelihood model; the parameters are... Used to indicate the gradient direction; For guiding strength; S14. Random Sampling and Stepping: At each time step d, the backward mean is obtained by applying a Gaussian distribution with variance after guidance. The next reverse state is obtained by random sampling. The enhanced speech waveform is eventually obtained through continuous reverse iteration. .

3. The method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Framing and Time-Frequency Representation: Based on frame shift... Window length and Hanning Window Enhanced speech waveform The complex spectrum is obtained by dividing the frame into frames and calculating the short-time Fourier transform of each frame in the forward direction. Then according to the complex spectrum Calculate the power spectrum The frame-level acoustic vector is obtained by taking the logarithm after Mel filtering. Then, for all frames, the frame-level acoustic vectors Normalized frame-level features are obtained by mean normalization. This leads to the normalized frame-level feature sequence. ; S22. Soft speech gating and silence suppression: Based on power spectrum Calculate the acoustic energy of each frame and construct the soft-gating coefficients. Then, the normalized frame-level feature sequence Each normalized frame-level feature in the data is gated to obtain a gated frame-level acoustic vector sequence. ; The total number of frames. N is the number of sampling points; S23, Frame Stacking and Downsampling: Gated Frame-Level Acoustic Vector Sequences S consecutive frames are stacked, and downsampling with a step size of S is used to obtain the stacked block-level acoustic vector. and the corresponding cross-modal alignment timestamp This yields a block-level acoustic vector sequence. and timestamp sequence ; The total length of the acoustic downsampling frames; S24, Common Dimension Projection and Position Encoding: Block-level acoustic vectors are encoded using two-layer projection. Mapping to multimodal common dimension And by introducing element-wise nonlinearity, a block-level common vector is obtained. Then the block-level common vector With position encoding Adding them together yields the speech semantic embedding. Finally, the speech semantic embedding sequence is obtained by summarizing. and voice timestamp sequence .

4. The method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, Spatial Patching and Local Attention Coding: Transforming the t-th video frame... Unfold the patches to obtain P patch vectors, and then perform linear embedding on these patch vectors to obtain the patch embedding. Then embed the patch The data is divided into several patch windows. After calculating the attention of each patch window, the data is stitched together and passed through a feedforward layer to obtain the frame-level spatial features of each frame. Finally, the attention outputs of all patch windows are concatenated and passed through a feedforward layer to obtain the frame-level spatial features of each frame. ; S32, Gesture Structuring: Structuring video frames Multiple gesture frames detected in From frame-level spatial features using ROI alignment operators Extracting region vectors and aggregating them by average yields the region aggregate vector. And based on video frames Set of skeleton key points detected in With confidence vector Calculate the short-time displacement of key points in adjacent frames Then, the three elements are vectorized and projected to obtain the skeleton structure fusion feature. Finally, the region aggregation vector is... Features integrated with skeletal structure Frame-level gesture tokens are obtained through fusion. ; S33. Short-time dynamic smoothing and temporal downsampling: Frame-level spatial features and frame-level gesture tokens After exponential smoothing and attention-weighted convergence, scene-level tokens are obtained. and smooth frame-level gesture tokens Then, block-level downsampling is performed on both to obtain the block-level scene token for each frame. and block-level gesture tokens And calculate video timestamps The video timestamp sequence is obtained. ;in, For video frame rate; is the video stacking factor; k is the video block index; Reduce the number of sampling blocks for the scene; Total number of frames; S34, Public Dimension Projection and Location Encoding: Block-level Scene Tokens and block-level gesture tokens Mapping to multimodal common dimension Then encode the same shared location. Obtain scene visual embedding and gesture visual embedding Finally, the scene visual embedding sequence is obtained. and gesture visual embedding sequence .

5. The method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. Unified Timeline and Index Mapping: Constructing a Time Raster Timestamp Sequence Nearest neighbor alignment is used to separate the speech timestamp sequences of the three types of modal block sequences. and video timestamp sequence Mapping to time index yields audio index mapping and video index mapping The three types of modal blocks include speech-semantic embedding sequences. Scene visual embedding sequence and gesture visual embedding sequence ; S42. Intramodal Attention Convergence: For each time grid n, construct the modal aggregation set and apply the attention convergence operator. The three-modal intra-representation is obtained; the three-modal intra-representation includes the raster-level speech intra-representation. Grid-level scene representation and grid-level gestures represent ; S43. Modal Soft-Gated Fusion: Obtaining the fused logic value by concatenating the internal representatives of the three modalities through an existence mask and performing a linear transformation. We assign a minimum probability to the empty mode and normalize it to obtain the weights. Obtain the grid-level fusion token ; S44, Segment and Position Embedding, Sequence Construction: Based on Grid-Level Fusion Tokens For each time grid n, the segment and position encoding are superimposed to construct the same input. And construct a unified input sequence Z; S45. Unified Transformer Coding: Multi-head self-attention and feedforward layer stacking coding are applied to the unified input sequence Z to obtain the fused temporal latent variable H and the global fusion vector. .

6. The method according to claim 1, characterized in that, Step S5 specifically includes the following steps: S51. Context-aware attention aggregation: After key-value transformation of the temporal latent variable H, it is combined with the globally fused vector. Construct latent variable queries to jointly calculate latent variable attention weights The context vector c is constructed. S52, Fusion Representation Construction and Multi-Task Header Output: Constructing the global fusion vector... The fused representation is obtained by concatenating and projecting the fused representation with the context vector c. And according to the fusion representation Construct parallel classification head and parametric regression head outputs to obtain the classification log odds z, classification probability p, and parametric regression results. ; S53. Temporal Smoothing: The log-odds ratio z of the classification is smoothed using an exponential moving average to obtain the smoothed log-odds ratio s, which is then obtained by exponential normalization to obtain the smoothed probability distribution. ; S54. Confidence Level Calibration and Uncertainty Measurement: Based on Smooth Probability Distribution Construct the temperature-calibrated intention confidence q and the predicted category And according to the smooth probability distribution Constructing prediction entropy confidence margin ; S55. Structured Intent Output: Predicted Category parametric regression Intent confidence q, prediction entropy Confidence margin With time raster timestamp The director's multimodal intent result is obtained by packaging the data into n time grids and outputting each time grid. This yields the latest director's multimodal intent result. .

7. The method according to claim 3, characterized in that, In step S21, according to the frame shift Window length and Hanning Window Enhance the speech waveform using the following formula The complex spectrum is obtained by dividing the frame into frames and calculating the short-time Fourier transform of each frame in the forward direction. : ， Where f is the frequency index. ; The number of positive frequency spectral lines; The total number of frames. ; is the total number of discrete frequency points used in the short-time Fourier transform operation; n is the index of the frame sample point; N is the number of sampling points; e is the base of the natural logarithm; j is the imaginary unit; i is the frame index of the frame, i=1,..., .

8. The method according to claim 7, characterized in that, In step S21, based on the complex spectrum Calculate the power spectrum The frame-level acoustic vector is obtained by taking the logarithm after Mel filtering. As shown in the following formula: ， in, This is the Mel filter matrix. is the numerical stability constant; M is the Mel frequency band number.

9. The method according to claim 8, characterized in that, In step S22, based on the power spectrum Calculate the acoustic energy of each frame and construct the soft-gating coefficients. Specifically: First, based on the power spectrum Calculate the acoustic energy of the i-th frame Then based on acoustic energy Constructing soft gating coefficients As shown in the following formula: ； in, The slope; The energy reference is used; the Sigmoid function is a sigmoid activation function.

10. The method according to claim 4, characterized in that, In step S33, frame-level spatial features are processed. and frame-level gesture tokens After exponential smoothing and attention-weighted convergence, scene-level tokens are obtained. and smooth frame-level gesture tokens Then, block-level downsampling is performed on both to obtain the block-level scene token for each frame. and block-level gesture tokens Specifically, it includes: Frame-level spatial features Exponential smoothing is used to suppress frame-by-frame jitter, and attention-weighted convergence is used to obtain scene-level tokens. : ， in, To control the smoothing intensity; Weighted convergence of attention; The frame-level token for the previous frame; Simultaneously, frame-level gesture tokens Perform exponential smoothing to obtain smoothed frame-level gesture tokens : ， in, For smooth frame-level gesture tokens from the previous frame; Then for scene frame-level tokens and smooth frame-level gesture tokens After performing block-level downsampling, the average of all block results is used to obtain the block-level scene token for each frame. and block-level gesture tokens As shown in the following formula: ， in, Here, k is the video stacking factor, and k is the video block index. , Reduce the number of sampling blocks for the scene.

Images