Wearable device video live streaming method and system based on intelligent AI large model driving

Abstract

The present application relates to the technical field of video live broadcast, in particular to a wearable device video live broadcast method and system based on intelligent AI large model driving. The emotion intensity of the user input text is compared with the multi-modal emotion intensity of the face, voice and action of the virtual anchor. When the emotion mismatch is detected, the sensitivity of the virtual anchor speech rate, fundamental frequency, energy and facial expression control channels to the emotion dimension is estimated by using the Jacobian matrix, and the PID regulation and smoothing limiting mechanism are combined to realize the dynamic adjustment of the virtual anchor emotion performance, so as to ensure that the virtual anchor can gradually approach the emotion state of the user and improve the interactivity and immersion experience in the live broadcast process.

Description

Technical Field

[0001] This invention relates to the field of video live streaming technology, specifically to a method and system for live streaming video on wearable devices driven by a large intelligent AI model. Background Technology

[0002] In existing virtual live streaming technologies, virtual anchors typically rely on preset action libraries, fixed speech synthesis parameters, or single-modal emotion recognition methods to drive the presentation of facial expressions and voices. This makes it difficult for virtual anchors to make dynamic adjustments based on real-time user feedback when interacting with users, resulting in a discrepancy between the emotional state input by the user and the emotional state expressed by the anchor, further reducing the immersion and interactivity of the live streaming process.

[0003] Meanwhile, most existing multimodal emotion recognition research is limited to single-point recognition and offline analysis, lacking a technical mechanism for dynamically matching and real-time regulating user emotions with virtual anchor emotions. Therefore, it is impossible to effectively guarantee emotion synchronization and interactive experience in live streaming scenarios. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a method and system for live video streaming of wearable devices based on intelligent AI large model, which can effectively solve the problem that virtual live streaming in the existing technology is mostly driven by preset scripts or single-modal emotions, lacks response to users' real-time emotions, and results in poor interactive experience.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention provides a method for live video streaming on wearable devices based on a large intelligent AI model, comprising the following steps:

[0007] Acquire the emotional intensity of the text and the multimodal emotional intensity of the virtual anchor;

[0008] The difference between the two values ​​determines whether the emotion regulation process is triggered.

[0009] Map the control channel parameters of the virtual anchor to the normalized domain;

[0010] The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight, and amplitude of movement;

[0011] Perturbations are applied to the input terminals of each control channel, and the response in the emotion dimension is output.

[0012] Based on the perturbation results, the sensitivity of each control channel to the emotion dimension is calculated, and the Jacobian matrix is ​​constructed.

[0013] The difference values ​​are weighted and allocated among the control channels according to their sensitivity.

[0014] The target control quantity obtained from different control channels is proportional to its sensitivity in the Jacobian matrix;

[0015] PID control is applied to the target control variables of each control channel.

[0016] Adjustment range limits are applied to the PID output results of each control channel to smooth the actual control output, and the output is then input to the virtual anchor for adjustment.

[0017] Furthermore, the method for determining the fundamental frequency is as follows:

[0018] The audio signal is divided into multiple short-time frame signals, and a fast Fourier transform is performed to output the complex spectrum of each frame.

[0019] The amplitude spectrum is calculated based on the complex spectrum, local maximum detection is performed on the amplitude spectrum, and spectral peaks, i.e. peak values, are identified in the amplitude spectrum;

[0020] Based on the gender characteristics of the speech signal, a new candidate spectral peak is output within a set frequency range;

[0021] Verification of whether candidate spectral peaks have harmonic structures includes:

[0022] Based on the frequency of the candidate spectral peak, peak detection is performed at integer multiples of the frequency. If a peak exists, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectral peak.

[0023] If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak, the candidate spectral peak passes the verification.

[0024] The candidate spectral peaks that have passed the verification are sorted, and the candidate spectral peaks corresponding to the fundamental frequency are determined based on the amplitude spectrum, and the preliminary fundamental frequency is output.

[0025] The fundamental frequency is verified based on the autocorrelation function, and the final fundamental frequency is determined.

[0026] Furthermore, the method for determining the range of motion is as follows:

[0027] Capture the 3D movements of the virtual anchor, including the three-dimensional coordinates of each joint at a point in time, to obtain the three-dimensional coordinates at the initial and end times;

[0028] Subtracting the three-dimensional coordinates at the initial time from the three-dimensional coordinates at the joint end time yields the displacement within the time window;

[0029] The displacement is calculated to obtain the displacement of each joint within the time window;

[0030] The actual distance is calculated using Euclidean distance, and corresponding weight coefficients are assigned according to the joint type.

[0031] The range of motion of the virtual anchor is calculated by weighted summation of all joint distances.

[0032] The method for determining whether the emotion regulation process has been triggered is as follows:

[0033] Facial expression weights, fundamental frequency, energy, speech rate, and range of motion are standardized.

[0034] Each emotion is mapped to its intensity using the sigmoid function;

[0035] The total emotional intensity of the virtual anchor is obtained by weighted summation;

[0036] Obtain the total emotional intensity of the text and the total emotional intensity of the virtual anchor, calculate the difference between the two emotional intensities. If the difference is less than the preset safe range, it means that the two emotions match; otherwise, they do not match.

[0037] Multiple data collection time points are set to obtain the difference values ​​at different time points. If the difference values ​​at multiple data collection time points are all judged to be mismatched, the emotion adjustment process is triggered.

[0038] Furthermore, the method for constructing the Jacobian matrix is ​​as follows:

[0039] Input the original control level and the original emotional dimension;

[0040] The original control quantity and the emotion dimension are mapped and transformed to the normalized domain;

[0041] Select a control channel, apply a step or pulse perturbation, and observe the output of the emotion dimension.

[0042] Locate the peak time and calculate the integer delay;

[0043] Input the original control quantity, emotion dimension, and integer delay of the normalized domain;

[0044] Construct incremental regression pairs;

[0045] Online recursive updates are used to construct the Jacobian matrix.

[0046] Furthermore, the method for implementing PID control on the target control quantity of each control channel is as follows:

[0047] The difference values ​​are assigned to each control channel and adjusted according to the Jacobian matrix, i.e., sensitivity, to output the target control value;

[0048] Based on the historical execution values ​​and baseline reference values ​​of the control channel, the deviation is calculated, and the channel error is output based on the target control value and the deviation calculation.

[0049] The channel error is input into the discrete PID controller, and the output control quantity is obtained by accumulating the proportional, integral, and derivative terms.

[0050] Furthermore, an exponential smoothing method is used to filter the control input to obtain a smooth-transitioning control output, wherein:

[0051] To ensure that the control quantity is within the physical boundary, the control quantity to be executed is output through inverse normalization.

[0052] A wearable device video live streaming system, applied to the aforementioned wearable device video live streaming method, includes:

[0053] The emotion intensity analysis module obtains the emotion intensity of the text and the multimodal emotion intensity of the virtual anchor, and determines whether to trigger the emotion regulation process based on the difference between the two.

[0054] The emotion parameter control module maps the control channel parameters of the virtual anchor to the normalized domain. The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight, and movement amplitude. Perturbations are applied to the input of each control channel, and the response of the emotion dimension is output. Based on the perturbation results, the sensitivity of each control channel to the emotion dimension is calculated, and a Jacobian matrix is ​​constructed. The difference values ​​are weighted and distributed among the control channels according to the sensitivity. The target control quantity obtained by different control channels is proportional to its sensitivity in the Jacobian matrix. PID control is executed on the target control quantity of each control channel, and the adjustment amplitude limit is applied to the PID output result of each control channel to smoothly output the actual executed control quantity.

[0055] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0056] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described herein.

[0057] The technical solution provided by this invention has the following advantages compared with the known prior art:

[0058] A multimodal emotion matching and dynamic control mechanism based on a large model is established. Specifically, the emotional intensity of the user's input text is compared with the multimodal emotional intensity of the virtual anchor's face, voice, and actions. When an emotional mismatch is detected, the sensitivity of the virtual anchor's speech rate, fundamental frequency, energy, and facial expressions to the emotional dimension is estimated using the Jacobi matrix. Combined with PID control and smoothing limiting mechanisms, the emotional performance of the virtual anchor is dynamically adjusted, thereby ensuring that the virtual anchor can gradually approach the user's emotional state and improve the interactivity and immersive experience during the live broadcast. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0060] Figure 1 This is a schematic diagram of the overall method of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0062] The present invention will be further described below with reference to embodiments.

[0063] Example 1 (see Figure 1 A wearable device video live streaming method driven by a large intelligent AI model includes at least the following:

[0064] Analyze the emotional intensity of the text (referring to the bullet comments entered by the user viewing terminal) and the emotional intensity of the virtual anchor (video live streaming terminal) to determine whether the virtual anchor's live streaming status and the viewing user's emotional state constitute an emotional match (a match reflects a good interactive experience between the virtual live stream and the user, while a mismatch indicates that the interaction between the virtual live stream and the user has room for improvement and optimization), including:

[0065] Collect the text (bullet comments) input by users into the virtual live stream, and preprocess the text to ensure that the text can meet the requirements of subsequent input;

[0066] The text is processed using an emotion lexicon (generated by human description or machine learning) to calculate the intensity of emotion, for example:

[0067] The text is segmented into words, each word or phrase is extracted, the sentiment value corresponding to each word is found, multiple sentiment values ​​are weighted and summed, and the final sentiment value is mapped to 0-1, outputting the total sentiment intensity of the text.

[0068] The method for identifying the emotional intensity of virtual anchors is as follows:

[0069] The system captures facial key points (eyes, lips, eyebrows, etc.) using sensors, subtracts the coordinates of neutral key points corresponding to neutral expressions (preset facial baseline expressions) from the coordinates of the facial key points to obtain key point offsets, and inputs the key point offsets into a pre-trained linear regression model or neural network model to output the corresponding facial expression weights.

[0070] The fundamental frequency is extracted from the audio signal of the virtual anchor, which reflects the basic pitch of the sound and its highness or lowness; the energy is extracted, which reflects the loudness or intensity of the sound; and the speech rate (usually obtained as the ratio of the total number of syllables to the duration of the audio signal) is extracted, which reflects the speed of speech.

[0071] The method for determining the fundamental frequency is as follows:

[0072] Preprocess the audio signal (perform noise suppression processing, remove background noise, and reduce noise interference by methods such as spectral subtraction or Wiener filtering);

[0073] The audio signal is divided into multiple short-time frame signals, each with a duration of 20ms to 40ms, and each frame has a certain overlap (usually 50%). This process can help capture the local frequency characteristics of the signal.

[0074] Perform a Fast Fourier Transform on each frame of signal to obtain the complex spectrum of each frame, which describes the amplitude and phase of each frequency component.

[0075] Based on the output complex spectrum, the amplitude spectrum is calculated to describe the energy magnitude of each frequency component, resulting in:

[0076] in, Represents the amplitude spectrum. Represents the complex spectrum at frequency The real part of the place, Represents the complex spectrum at frequency The virtual part of the location;

[0077] Furthermore, the obtained amplitude spectrum is numerically optimized to make the spectrum smoother and reduce the interference of high-frequency noise on the fundamental frequency detection.

[0078] in, This represents the amplitude of the smoothed amplitude spectrum at a given frequency. Smoothing reduces high-frequency noise, making subsequent spectral peaks more prominent. This represents half the length of the smoothing window, describing the values ​​taken from the left and right sides of the current frequency point. The average value is calculated from the neighboring points.

[0079] Local maximum detection is performed on the amplitude spectrum to find the spectral peaks of each frame, for example:

[0080] Peak values ​​in the amplitude spectrum can be identified using the local maximum detection method.

[0081] If the amplitude spectrum of a certain frequency point is greater than that of its neighboring points, it is considered to be a spectral peak, i.e., a peak value.

[0082] Significant spectral peaks in the amplitude spectrum (peaks above a threshold are identified as spectral peaks) are selected. These usually correspond to the main frequency components in the signal. By selecting spectral peaks, the amount of computation is reduced, and the strongest frequency components are focused on.

[0083] Based on the characteristics of the voice signal (the characteristics of the virtual anchor's voice correspond to the fundamental frequency range, typically 85-180Hz for males, 165-255Hz for females, and 400-1000Hz for singing), the frequency range is set, and the spectral peaks within the frequency range are related to the fundamental frequency;

[0084] Peaks located outside the frequency range are eliminated based on the frequency range to obtain new candidate spectral peaks (there may be multiple peaks).

[0085] Currently, to facilitate the determination of the fundamental frequency, a candidate spectral peak with the largest amplitude spectrum within the frequency range is identified as the candidate fundamental frequency peak. The frequency of this candidate fundamental frequency peak is used as the preliminary fundamental frequency (the fundamental frequency is usually the strongest low-frequency component, and the largest spectral peak is established as the fundamental frequency). However, due to a characteristic of speech signals, integer multiples of the fundamental frequency also have energy peaks, called harmonics. The amplitude of harmonics may be larger than the fundamental frequency itself, leading to the incorrect selection of harmonics instead of the true fundamental frequency by directly taking the largest spectral peak. Therefore:

[0086] Preserve new candidate spectral peaks within the frequency range;

[0087] Verify whether each candidate spectral peak has a harmonic structure, including:

[0088] Determine the amplitude of each candidate spectral peak;

[0089] Based on the frequency of each candidate spectral peak, peak detection is performed at harmonic positions that are integer multiples of the peak, such as 2 or 3 times the frequency. If a peak exists, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectral peak.

[0090] If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak (for a threshold of 1 / 3, if the threshold is set too high, such as having to reach 1 / 2, it will miss detection in some weak harmonic scenarios, and what is actually the fundamental frequency will be rejected because the harmonic amplitude is not strong enough; if the threshold is set too low, such as 1 / 10, some occasional small peaks may also be mistaken for harmonics, thus misjudging noise or non-harmonic peaks; therefore, 1 / 3 is usually considered an empirical threshold that can balance robustness and fault tolerance), then the candidate spectral peak is considered to have a real harmonic structure and is verified through harmonic structure.

[0091] The candidate spectral peaks that have passed the verification are sorted in descending order of amplitude, and the candidate spectral peak with the largest amplitude spectrum is taken as the preliminary fundamental frequency;

[0092] The method involves utilizing the periodicity information of the signal and verifying and optimizing the estimated fundamental frequency based on the autocorrelation function (ACF) to improve the accuracy of fundamental frequency detection. The steps are as follows:

[0093] Calculate the autocorrelation function for each frame of the time-domain signal (the short-time frame signal mentioned above);

[0094] Find the first peak value with non-zero latency;

[0095] The fundamental frequency is calculated based on the maximum peak value with non-zero delay;

[0096] Therefore, the initial frequency domain fundamental frequency can be compared with the current time domain fundamental frequency. If the difference between the two is less than a preset value, it indicates that the initial frequency domain fundamental frequency is reliable and is established as the final fundamental frequency. If the difference between the two is greater than a preset value, the two fundamental frequencies can be calculated by weighted summation to ensure the reliability of the fundamental frequency and avoid misjudgment caused by relying solely on the fundamental frequency of a single frame.

[0097] Energy includes time-domain energy and frequency-domain energy, and methods for determining it include:

[0098] For time-domain energy:

[0099] The temporal energy of each frame is calculated based on the sum of the squares of the signal amplitude of each frame; temporal energy is used to help capture changes in volume, which is an important indicator of emotional changes.

[0100] For frequency domain energy:

[0101] The frequency domain energy of each frame of signal is calculated based on the square of the amplitude spectrum. The frequency domain energy is used to reveal the frequency characteristics of emotions, such as changes in the fundamental frequency and high-frequency components in the spectrum.

[0102] The 3D movements of the virtual anchor are captured using motion capture equipment, including the three-dimensional coordinates of each joint (face joint, shoulder joint, elbow joint, wrist joint, etc.) at a given time point. For each joint, the three-dimensional coordinates at the initial and end times are obtained. The displacement within the time window is obtained by subtracting the three-dimensional coordinates at the initial time from the three-dimensional coordinates at the end time. The displacement is then calculated to obtain the displacement of each joint within the time window. Here, the displacement is a displacement vector with direction. The actual distance is calculated using Euclidean distance. Corresponding weight coefficients are assigned according to the joint type, such as 0.3 for the face joint and 0.25 for the wrist joint. The entire range of motion of the virtual anchor is obtained by weighted summation of all joint distances.

[0103] The facial expression weights, fundamental frequency, energy, speech rate, and movement amplitude of the collected virtual anchors are normalized in terms of dimensions, such as through standardization, and then mapped to emotional intensity using the sigmoid function. For example, the mapping of facial expression weights is as follows:

[0104] in, Indicates the intensity of facial emotions after mapping. Indicates the weight of facial expressions. The threshold representing the weight is calculated by substituting the corresponding data into the corresponding threshold and keeping the rest of the formula unchanged. This mapping will output the emotional intensity of the corresponding data. This will not be elaborated on here.

[0105] Then, a weighted sum is performed to obtain the total emotional intensity of the virtual anchor.

[0106] The total emotional intensity of the text and the total emotional intensity of the virtual anchor are obtained. The difference between the two emotional intensities is calculated. If the difference is less than the preset safe range, it means that the two emotions match; otherwise, they do not match.

[0107] As described above, multiple data collection time points can be set to obtain the difference values ​​at different time points. If the difference values ​​at multiple data collection time points are all judged to be mismatched, the virtual live broadcast emotion adjustment process is triggered, including:

[0108] Calculate the emotional difference between the two at the current moment (the time corresponding to the last data collection point), and gradually match the virtual anchor's emotional expression with the user's emotion by controlling features such as speech rate, fundamental frequency, energy, and facial expression weights. The specific steps are as follows:

[0109] The input virtual anchor is a normalized control quantity under different control parameters, for example:

[0110] Adjust speech rate: Speed ​​up or slow down the virtual anchor's speech output.

[0111] Adjusting the fundamental frequency: changes the pitch of the sound.

[0112] Adjusting energy: Affects the volume or tone of voice.

[0113] Adjust facial expression weights: enhance or weaken smiles, eyebrow curves, etc.

[0114] The facial key point detection method is used to extract the coordinate changes of key points such as eyebrows, corners of mouth, and eyes, and map the key point changes to standard facial action units. For example, a smile corresponds to the AU vector of the corners of the mouth (AU refers to the action unit in the facial action coding system).

[0115] Facial landmark detection methods, such as Active Shape Models (ASM), are statistical model-based methods that detect landmarks by building a statistical model of the face shape. They learn the variation rules of the face shape through a large number of samples, and then search for the shape that best matches the model in a new image to determine the location of the landmark. Alternatively, template matching methods can be used, which detect landmarks by performing a sliding window search on the target region and matching it with existing facial feature templates.

[0116] By using a pre-trained visual emotion classification model, AU vectors are combined and mapped to emotion intensities, such as happiness, surprise, or anger, to output a visual emotion feature vector for each frame. ;

[0117] Using the above method, audio features, including fundamental frequency, energy, and speech rate, are extracted from the audio signal simultaneously. These speech features are then input into a speech emotion recognition model for emotion classification, outputting a speech emotion feature vector for each frame. ;

[0118] Visual and verbal emotion feature vectors are concatenated and fused to output a fused virtual anchor emotion vector, which is then mapped to an emotion dimension using a multilayer perceptron (MLP). Arousal level indicates the degree of excitement / calmness, while polarity indicates the positive / negative nature of the emotion;

[0119] For the normalized control variables, since different control variables have different sensitivities to emotions at different times or in different contexts, the Jacobian matrix is ​​estimated by recursive least squares method. This matrix is ​​used to quantify the influence of each adjustable feature on the emotion dimension and provide a basis for online regulation. In the Jacobian matrix, rows are used to represent the emotion dimension index, such as arousal and polarity, while columns represent the control channel index, such as speech rate, fundamental frequency, and energy.

[0120] The specific method for estimating the Jacobian matrix is ​​as follows:

[0121] Input the original control quantity and the original emotion dimension, and perform linear or monotonic mapping to the normalization domain;

[0122] Apply a small step or pulse to a specific control channel of the control signal, such as speech rate:

[0123] in, Indicates the first Normalized input values ​​for each control channel Indicates the baseline value. Indicates the moment when the disturbance begins. This indicates a small disturbance, ensuring the system does not reach saturation;

[0124] Observe the output Find the peak time of the corresponding channel by observing the change curve. ;

[0125] Calculate integer delay ;

[0126] Output an integer .

[0127] The input is mapped to the original control quantity and original emotion dimension, and integer delay in the normalized domain;

[0128] Constructing incremental regression pairs:

[0129] in, This represents the normalized control increment. This represents the gradual increment of the emotion vector. Represents the model residuals;

[0130] Output the control increment for each step With observation increment ;

[0131] Input dimension information, including the total number of control channels m, the total number of emotion dimensions n, and the forgetting factor. (Typically 0-1) and the initial uncertainty scalar (usually) );

[0132] By each emotion dimension Maintain a separate set of parameters:

[0133] , Represent the identity matrix, output ;

[0134] Output the current step , And the previous step ;

[0135] right Parallel execution:

[0136] in, express The covariance matrix of each emotion dimension, Indicates the first The incremental observations in each emotional dimension Indicates the first A vector of regression parameters for each emotion dimension;

[0137] make The i-th line is ;

[0138] After each update step, the parameters of each row are concatenated to obtain the Jacobian matrix. .

[0139] Furthermore, different control channels have varying abilities to correct for emotional errors; therefore, the adjustment amount should be weighted according to sensitivity, resulting in:

[0140] Indicates the normalized weights. Indicates the first Sensitivity of each control channel Indicates time;

[0141] The emotional error, i.e., the difference value, is assigned to each control channel and adjusted according to its Jacobian matrix (sensitivity):

[0142] in, Indicates control channel The target control quantity in the normalized domain This represents the global gain, ranging from 0 to 1, and controls the update magnitude. Indicates transpose. Indicates the difference value.

[0143] The adjustment process needs to be smooth, safe, and focused on key control channels to prevent sudden changes in the virtual anchor's performance. Therefore:

[0144] Define the error per channel (normalization domain). ;

[0145] in, Indicates control channel The actual normalized control quantity at the previous moment serves as the baseline describing the current state and is used as the historical execution quantity. Indicates control channel The baseline reference value, such as neutral speech rate, is used as a benchmark for comparison to calculate the deviation of the actual output from the expected neutral value. The calculated channel error is then used to drive the PID controller for adjustment.

[0146] Perform PID tracking for each channel target:

[0147] in, Indicates proportional gain. Indicates integral gain. Represents differential gain. This represents the output control quantity of the normalized PID controller for each channel. The sampling period is used for integral and differential calculations. In the dynamic control process of emotion matching, closed-loop control is introduced to adjust each control channel (speech rate, fundamental frequency, energy, expression, etc.) so that the adjustment process can gradually eliminate deviations and maintain a smooth and natural state, avoiding abruptness and oscillation, thereby ensuring that the emotional adjustment process of the virtual anchor conforms to the natural feeling of human interaction.

[0148] In the above PID formula, anti-saturation logic can be applied using integral suppression, integral feedback, and integral limiting methods. Specifically, the integral suppression method is as follows:

[0149] when Points will not be updated if the limit is exceeded:

[0150] ;

[0151] Then there is , Indicates the first The cumulative output of the integral element of each control channel is used to accumulate historical errors and eliminate steady-state deviations.

[0152] To ensure a natural transition and prevent mutations and conflicts, the following applies:

[0153] in, This represents the control quantity after limiting each channel. This indicates the maximum adjustment range per channel per step. This indicates that the target value is limited to a certain range;

[0154] The control input is filtered using an exponential smoothing method to obtain a control output with a smooth transition.

[0155] in, This represents the control quantity for each channel after safety, smoothing, and normalization. Represents the smoothing coefficient. , Indicates the time constant for exponential smoothing;

[0156] To ensure that the control quantity remains within the physical boundaries, inverse normalization is performed:

[0157] in, This represents the control quantity issued by the physical domain. , These represent the upper and lower boundaries of the physical quantity in each channel, respectively. This indicates the current physical control quantity (such as speech rate, energy, etc.). This represents the control variables to be executed. By controlling the virtual anchor's speech rate and other characteristics through these control variables, the virtual anchor's emotions can be matched with the emotions in the text, thereby improving the interactive experience between the user and the virtual anchor.

[0158] Furthermore, if the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0160] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0161] Furthermore, in order to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features that are not relevant to the best mode of carrying out the invention as currently considered, or those features that are not relevant to implementing the invention) may be omitted.

[0162] It should be understood that numerous specific implementation decisions can be made during the development of any practical implementation, such as in any engineering or design project. Such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, the development effort will be a routine work of design, manufacturing, and production without requiring much experimentation.

[0163] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for live video streaming on wearable devices driven by a large intelligent AI model, characterized in that: Includes the following steps: Acquire the emotional intensity of the text and the multimodal emotional intensity of the virtual anchor; The difference between the two values ​​determines whether the emotion regulation process is triggered. Map the control channel parameters of the virtual anchor to the normalized domain; The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight, and amplitude of movement; Perturbations are applied to the input terminals of each control channel, and the response in the emotion dimension is output. Based on the perturbation results, the sensitivity of each control channel to the emotion dimension is calculated, and the Jacobian matrix is ​​constructed. The difference values ​​are weighted and allocated among the control channels according to their sensitivity. The target control quantity obtained from different control channels is proportional to its sensitivity in the Jacobian matrix; PID control is applied to the target control variables of each control channel. The adjustment range of the PID output results of each control channel is limited to smooth the actual control quantity, which is then input to the virtual anchor for adjustment. The method for constructing the Jacobian matrix is ​​as follows: Input the original control level and the original emotional dimension; The original control quantity and the emotion dimension are mapped and transformed to the normalized domain; Select a control channel, apply a step or pulse perturbation, and observe the output of the emotion dimension. Locate the peak time and calculate the integer delay; Input the original control quantity, emotion dimension, and integer delay of the normalized domain; Construct incremental regression pairs; Online recursive updates to construct the Jacobian matrix; The method for implementing PID control on the target control quantity of each control channel is as follows: The difference values ​​are assigned to each control channel and adjusted according to the Jacobian matrix, i.e., sensitivity, to output the target control value; Based on the historical execution values ​​and baseline reference values ​​of the control channel, the deviation is calculated, and the channel error is output based on the target control value and the deviation calculation. The channel error is input into the discrete PID controller, and the output control quantity is obtained by accumulating the proportional, integral, and derivative terms.

2. The wearable device video live streaming method according to claim 1, characterized in that, The method for determining the fundamental frequency is as follows: The audio signal is divided into multiple short-time frame signals, and a fast Fourier transform is performed to output the complex spectrum of each frame. The amplitude spectrum is calculated based on the complex spectrum, local maximum detection is performed on the amplitude spectrum, and spectral peaks, i.e. peak values, are identified in the amplitude spectrum; Based on the gender characteristics of the speech signal, a new candidate spectral peak is output within a set frequency range; Verification of whether candidate spectral peaks have harmonic structures includes: Based on the frequency of the candidate spectral peak, peak detection is performed at integer multiples of the frequency. If a peak exists, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectral peak. If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak, the candidate spectral peak passes the verification. The candidate spectral peaks that have passed the verification are sorted, and the candidate spectral peaks corresponding to the fundamental frequency are determined based on the amplitude spectrum, and the preliminary fundamental frequency is output. The fundamental frequency is verified based on the autocorrelation function, and the final fundamental frequency is determined.

3. The wearable device video live streaming method according to claim 1, characterized in that, The method for determining the range of motion is as follows: Capture the 3D movements of the virtual anchor, including the three-dimensional coordinates of each joint at a point in time, to obtain the three-dimensional coordinates at the initial and end times; Subtracting the three-dimensional coordinates at the initial time from the three-dimensional coordinates at the joint end time yields the displacement within the time window; The displacement is calculated to obtain the displacement of each joint within the time window; The actual distance is calculated using Euclidean distance, and corresponding weight coefficients are assigned according to the joint type. The range of motion of the virtual anchor is calculated by weighted summation of all joint distances.

4. The wearable device video live streaming method according to claim 1, characterized in that, The method for determining whether the emotion regulation process has been triggered is as follows: Facial expression weights, fundamental frequency, energy, speech rate, and range of motion are standardized. Each emotion is mapped to its intensity using the sigmoid function; The total emotional intensity of the virtual anchor is obtained by weighted summation; Obtain the total emotional intensity of the text and the total emotional intensity of the virtual anchor, calculate the difference between the two emotional intensities. If the difference is less than the preset safe range, it means that the two emotions match; otherwise, they do not match. Multiple data collection time points are set to obtain the difference values ​​at different time points. If the difference values ​​at multiple data collection time points are all judged to be mismatched, the emotion adjustment process is triggered.

5. The wearable device video live streaming method according to claim 1, characterized in that, The control input is filtered using an exponential smoothing method to obtain a smooth-transitioning control output, wherein: To ensure that the control quantity is within the physical boundary, the control quantity to be executed is output through inverse normalization.

6. A wearable device video live streaming system, applied to the wearable device video live streaming method according to any one of claims 1-5, characterized in that, include: The emotion intensity analysis module obtains the emotion intensity of the text and the multimodal emotion intensity of the virtual anchor, and determines whether to trigger the emotion regulation process based on the difference between the two. The emotion parameter control module maps the control channel parameters of the virtual anchor to the normalized domain. The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight, and movement amplitude. Perturbations are applied to the input of each control channel, and the response of the emotion dimension is output. Based on the perturbation results, the sensitivity of each control channel to the emotion dimension is calculated, and a Jacobian matrix is ​​constructed. The difference values ​​are weighted and distributed among the control channels according to the sensitivity. The target control quantity obtained by different control channels is proportional to its sensitivity in the Jacobian matrix. PID control is executed on the target control quantity of each control channel, and the adjustment amplitude limit is applied to the PID output result of each control channel to smoothly output the actual executed control quantity.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images