Wearable device video live broadcast method and system based on intelligent AI large model driving

Abstract

The invention relates to the technical field of video live broadcast, in particular to a wearable device video live broadcast method and system based on intelligent AI large model driving. The emotion intensity of a text input by a user is compared with the face, voice and action multi-mode emotion intensity of a virtual anchor, when it is detected that the emotion is not matched, the Jacobian matrix is used for estimating the sensitivity of control channels such as the speed, the fundamental frequency, the energy and the facial expression of the virtual anchor to the emotion dimension, and PID regulation and control and a smooth amplitude limiting mechanism are combined, so that the emotion intensity of the text input by the user is obtained. Dynamic adjustment of the emotion performance of the virtual anchor is realized, so that the virtual anchor can gradually approach to the emotion state of the user, and the interactivity and immersion experience in the live broadcast process are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video live broadcast, in particular to a wearable device video live broadcast method and system based on intelligent AI large model driving. BACKGROUND

[0002] In the existing virtual live broadcast technology, the virtual host usually relies on a preset action library, fixed voice synthesis parameters or a single mode emotion recognition method to drive the presentation of expressions and voices, which makes the virtual host often unable to make dynamic adjustments according to the real-time feedback of the user when interacting with the user, resulting in a deviation between the emotional state input by the user and the emotional state presented by the host, further reducing the immersion and interactivity of the live broadcast process.

[0003] At the same time, the existing multi-modal emotion recognition research mostly stays in single-point recognition and offline analysis, lacking a technical mechanism for dynamically matching and real-time regulating the user's emotions and the virtual host's emotions, so it cannot effectively guarantee the emotional synchronization and interactive experience in the live broadcast scene. SUMMARY

[0004] In view of the above shortcomings of the prior art, the present application provides a wearable device video live broadcast method and system based on intelligent AI large model driving, which can effectively solve the problem that the existing virtual live broadcast is mostly driven by preset scripts or single mode emotions, lacking real-time response to user emotions, resulting in poor interactive experience.

[0005] To achieve the above purpose, the present application is realized by the following technical scheme: The present application provides a wearable device video live broadcast method based on intelligent AI large model driving, comprising the following steps: Obtaining the emotional intensity of the text and the multi-modal emotional intensity of the virtual host; Determining whether to trigger the emotion regulation process according to the difference value; Mapping the control channel parameters of the virtual host to the normalized domain; The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight and action amplitude; Applying perturbation at each control channel input end and outputting the response of the emotional dimension; According to the perturbation result, the sensitivity of each control channel to the emotional dimension is calculated, and a Jacobian matrix is constructed; The difference value is weighted and distributed among the control channels according to the sensitivity; The target control amount obtained by different control channels is proportional to its sensitivity in the Jacobian matrix; Performing PID control on the target control amount of each control channel; The PID output results of each control channel are subjected to adjustment amplitude limitation, the actual execution control quantity is smoothed and output, and is input to the virtual host for adjustment.

[0006] Further, the determination method of the fundamental frequency is: The audio signal is divided into multiple short-time frame signals, fast Fourier transform is performed, and the complex spectrum of each frame is output; The amplitude spectrum is calculated according to the complex spectrum, local maximum value detection is performed on the amplitude spectrum, and the spectral peak, i.e., the peak value, in the amplitude spectrum is identified; According to the gender characteristics of the speech signal, a new candidate spectral peak is output in a frequency range; The candidate spectral peak is verified to have a harmonic structure, including: According to the frequency of the candidate spectral peak, peak value detection is performed at integer multiples thereof, and if there is a peak value, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectral peak; If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak, the candidate spectral peak passes the verification; The candidate spectral peak that passes the verification is sorted, the candidate spectral peak corresponding to the fundamental frequency is determined according to the amplitude spectrum, and a preliminary fundamental frequency is output; The fundamental frequency is verified according to the autocorrelation function, and the final fundamental frequency is determined.

[0007] Further, the determination method of the action amplitude is: The 3D action of the virtual host is captured, including the three-dimensional coordinates of each joint at a time point, to obtain the three-dimensional coordinates at the initial time and the end time; The three-dimensional coordinates of the joint at the end time are subtracted from the three-dimensional coordinates at the initial time to obtain the displacement in the time window; The displacement is calculated to obtain the displacement of each joint in the time window; The actual distance is calculated through the Euclidean distance, and the corresponding weight coefficient is assigned according to the joint type; The action amplitude of the virtual host is calculated by weighted summation of all joint distances.

[0008] The method for determining whether the emotion regulation process is triggered is: The facial expression weight, the fundamental frequency, the energy, the speech rate, and the action amplitude are subjected to standardization processing; Each is mapped to an emotional intensity through a sigmoid function; The total emotional intensity of the virtual host is obtained by weighted summation; The total emotional intensity of the text and the total emotional intensity of the virtual host are obtained, the difference value of the emotional intensity of the two is calculated, and if the difference value is less than a preset safe interval, it is indicated that the emotions of the two match, otherwise they do not match; A plurality of collection time points are set to obtain difference values at different time points. If the difference values at the plurality of collection time points are all determined to be mismatched, an emotion adjustment process is triggered.

[0009] Further, the construction method of the Jacobian matrix is as follows: The original control quantity and the original emotion dimension are inputted. The original control quantity and the emotion dimension are mapped and converted to a normalized domain. A control channel is selected, a step or pulse disturbance is applied, and the emotion dimension output is observed. The peak time is located, and the integer delay is calculated. The original control quantity, the emotion dimension and the integer delay in the normalized domain are inputted. The incremental regression pair is constructed. The Jacobian matrix is constructed by online recursive update.

[0010] Further, the method for performing PID control on the target control quantity of each control channel is as follows: The difference value is distributed to each control channel, and the target control quantity is outputted according to the sensitivity adjustment, i.e., the Jacobian matrix. The deviation is calculated according to the historical execution quantity of the control channel and the baseline reference value, and the channel error is calculated and outputted according to the target control quantity and the deviation. The channel error is inputted into the discrete PID controller, and the output control quantity is obtained by accumulating the proportional term, the integral term and the differential term.

[0011] Further, the exponential smoothing method is used to filter the control quantity to obtain a smooth transition control output, wherein: To ensure that the control quantity is within the physical boundary, the control quantity to be executed is outputted by inverse normalization.

[0012] The wearable device video live streaming system is applied to the wearable device video live streaming method and comprises: The emotion intensity analysis module acquires the emotion intensity of the text and the multi-modal emotion intensity of the virtual anchor, and determines whether to trigger the emotion regulation process according to the difference value between the two. The emotion parameter regulation module maps the control channel parameters of the virtual anchor to the normalized domain, the control channel parameters include the speech rate, the fundamental frequency, the energy, the facial expression weight and the motion amplitude, applies a disturbance at the input end of each control channel, outputs the response of the emotion dimension, calculates the sensitivity of each control channel to the emotion dimension according to the disturbance result, constructs the Jacobian matrix, and distributes the difference value to each control channel according to the sensitivity. The target control quantity obtained by different control channels is proportional to the sensitivity of the control channel in the Jacobian matrix. PID control is performed on the target control quantity of each control channel, the adjustment amplitude limit is performed on the PID output result of each control channel, and the actual execution control quantity is smoothly outputted.

[0013] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of any of the methods when executing the computer program.

[0014] A computer readable storage medium stores a computer program, and the computer program implements the steps of any of the methods when executed by a processor.

[0015] Compared with the known prior art, the technical solution provided by the present application has the following beneficial effects: A multi-modal emotion matching and dynamic regulation mechanism based on a large model is established, specifically, the emotion intensity of the user input text is compared with the multi-modal emotion intensity of the face, voice and action of the virtual anchor, when emotion mismatch is detected, the sensitivity of the virtual anchor speech rate, fundamental frequency, energy and facial expression control channels to the emotion dimension is estimated by using the Jacobian matrix, and the PID regulation and smoothing limiting mechanism are combined to realize the dynamic adjustment of the virtual anchor emotion performance, so as to ensure that the virtual anchor can gradually approach the emotion state of the user, and improve the interactivity and immersion experience in the live broadcast process. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0017] Figure 1 The overall method of the present application is shown in the figure. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] The present application will be further described below in combination with the embodiments.

[0020] Embodiment 1 (see Figure 1 ): The intelligent AI large model driven wearable device video live broadcast method at least comprises: Analyze the emotional intensity of the text (the input of the user on the viewing end) and the emotional intensity of the virtual host (the video live streaming end), and determine whether the live streaming state of the virtual host matches the user, including: Collect the text (barrage) input by the user to the virtual live streaming, and preprocess the text to ensure that the text can meet the subsequent input; Process the text through an emotional dictionary (the dictionary is generated by manual expression or machine learning), calculate the emotional intensity, for example: Carry out word segmentation on the text, extract each word or phrase, find the emotional value corresponding to the word, weight and sum multiple emotional values, and map the final emotional value to 0-1, and output the total emotional intensity of the text.

[0021] Identify the emotional intensity of the virtual host, wherein the specific method is: Capture facial key points (eyes, lips, eyebrows, etc.) through sensors, subtract the neutral key point coordinates corresponding to the neutral expression (preset facial reference expression) from the facial key point coordinates to obtain key point offsets, and input the key point offsets into a pre-trained linear regression model or neural network model to output the corresponding facial expression weight; Extract the fundamental frequency from the audio signal of the virtual host, which is used to reflect the basic pitch of the sound and reflect the high and low of the sound; extract the energy, which is used to reflect the loudness or intensity of the sound; extract the speech rate (usually obtained by the ratio of the total number of syllables to the duration of the audio signal), which is used to reflect the fast or slow degree of speaking; The determination method of the fundamental frequency is: Preprocess the audio signal (perform noise suppression processing to remove background noise, which can use spectral subtraction or Wiener filtering to reduce noise interference); Divide the audio signal into multiple short-time frame signals, and the time length of each frame is usually 20ms to 40ms, and there is a certain overlap (usually 50%) between each frame. This process can assist in capturing the local frequency characteristics of the signal; Perform fast Fourier transform on each frame signal to obtain the complex spectrum of each frame, which describes the amplitude and phase of each frequency component; According to the output complex spectrum, calculate the amplitude spectrum to describe the energy size of each frequency component, then: Wherein, represents the amplitude spectrum, represents the real part of the complex spectrum at frequency , represents the imaginary part of the complex spectrum at frequency , Further, the amplitude spectrum is numerically optimized to make the spectrum smoother and reduce the interference of high-frequency noise on the fundamental frequency detection: wherein, represents the amplitude of the smoothed amplitude spectrum at the frequency, and the subsequent spectrum peak is more prominent by reducing high-frequency noise through smoothing, represents half the length of the smoothing window, and describes the average of the left and right adjacent points of the current frequency point.

[0022] Local maximum value detection is performed on the amplitude spectrum to find the spectrum peak of each frame, for example: The peak value in the amplitude spectrum is identified by the local maximum value detection method; If the amplitude spectrum of a certain frequency point is greater than its adjacent points, it is considered to be a spectrum peak, i.e., a peak value; Significant spectrum peaks (higher than the threshold value) are selected in the amplitude spectrum, which usually correspond to the main frequency components in the signal. By screening the spectrum peaks, the calculation amount is reduced, and the strongest frequency components are focused on; According to the characteristics of the speech signal (the characteristics of the virtual anchor speaking sound correspond to the fundamental frequency range, usually 85-180Hz for men and 165-255Hz for women, and usually 400-1000Hz for singing), a frequency range is set, and the spectrum peaks within the frequency range are related to the fundamental frequency; According to the frequency range, the peak values outside the frequency range are removed to obtain new candidate spectrum peaks (there may be multiple); At present, in order to determine the fundamental frequency, a candidate spectrum peak with the maximum amplitude in the frequency range is determined as the fundamental frequency candidate spectrum peak, and the frequency of the fundamental frequency candidate spectrum peak is taken as the preliminary fundamental frequency (the fundamental frequency is usually the strongest low-frequency component, and the maximum spectrum peak is determined as the fundamental frequency). However, due to the characteristics of the speech signal, the integer multiple frequency of the fundamental frequency also has an energy peak, called a harmonic, and the amplitude of the harmonic may be greater than that of the fundamental frequency itself, resulting in the error selection of the harmonic instead of the true fundamental frequency by directly taking the maximum spectrum peak. Therefore: The new candidate spectrum peaks within the frequency range are retained; Verify whether each candidate spectrum peak has a harmonic structure, including: Determine the amplitude of each candidate spectrum peak; According to the frequency of each candidate spectrum peak, peak value detection is performed at the harmonic positions of its integer multiples, such as 2 times and 3 times, and if there is a peak value, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectrum peak; If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak (for setting a threshold of 1 / 3, if the threshold is set too high, such as must reach 1 / 2, in some weak harmonic scene, it will be missed, and the original fundamental frequency is denied because the harmonic amplitude is not strong enough; if the threshold is set too low, such as 1 / 10, some accidental small peaks may be mistaken for harmonics, resulting in false noise or non-harmonic peaks; therefore, 1 / 3 is generally considered an experienced threshold that can balance robustness and fault tolerance), it is considered that the candidate spectral peak has a real harmonic structure, and the harmonic structure is verified; The candidate spectral peaks that pass the verification are sorted in descending order of amplitude, and the candidate spectral peak with the largest amplitude spectrum is taken as the preliminary fundamental frequency; Among them, by utilizing the periodicity information of the signal, the estimated fundamental frequency is verified and optimized according to the autocorrelation function (ACF), and the accuracy of the fundamental frequency detection is improved, and the steps are as follows: Calculate the autocorrelation function for each frame of time domain signal (short time frame signal in the above); Find the maximum peak of the first non-zero delay; Calculate the fundamental frequency according to the maximum peak of the non-zero delay; Therefore, the preliminary frequency domain fundamental frequency can be compared with the current time domain fundamental frequency; if the difference between the two is less than a preset value, it means that the preliminary frequency domain fundamental frequency is reliable, and is established as the final fundamental frequency; if the difference between the two is greater than the preset value, the two fundamental frequencies can be calculated in a weighted sum to obtain the final fundamental frequency, ensuring the reliability of the fundamental frequency and avoiding false judgment caused by relying on a frame of fundamental frequency.

[0023] The energy includes time domain energy and frequency domain energy, and the determination method includes: For time domain energy: Calculate the time domain energy of each frame according to the sum of the square of the amplitude of each frame signal; the time domain energy is used to help capture the volume change, which is an important indicator of emotional change; For frequency domain energy: Calculate the frequency domain energy of each frame signal according to the square of the amplitude spectrum, and the frequency domain energy is used to reveal the frequency characteristics of the emotion, such as the change of the fundamental frequency, the high frequency component in the frequency spectrum, etc.

[0024] Capture the 3D action of the virtual host using a motion capture device, including the three-dimensional coordinates of each joint (facial joints, shoulder joints, elbow joints, wrist joints, etc.) at a time point, for each joint, get the three-dimensional coordinates at the initial time and the end time, subtract the three-dimensional coordinates at the initial time from the three-dimensional coordinates at the end time to get the displacement in the time window, then calculate the displacement to get the displacement of each joint in the time window. The displacement here is a displacement vector with direction, and the actual distance is calculated by Euclidean distance. According to the type of joint, assign a corresponding weight coefficient, such as 0.3 for facial joints and 0.25 for wrist joints. The action amplitude of the entire virtual host is calculated by weighted summation of all joint distances. The facial expression weight, fundamental frequency, energy, speech rate, and action amplitude of the collected virtual host are dimensionally normalized, such as standardized processing, and then mapped to emotional intensity through a sigmoid function, such as the mapping of facial expression weight: wherein, represents the mapped facial emotional intensity, represents the facial expression weight, represents the threshold value of the weight, that is, by substituting the corresponding data into the difference with the corresponding threshold value, the formula remains unchanged for mapping, which can output the emotional intensity of the corresponding data. Here, it is not described one by one.

[0025] Then, the weighted sum is performed to obtain the total emotional intensity of the virtual host.

[0026] The total emotional intensity of the text and the total emotional intensity of the virtual host are obtained, and the difference value of the emotional intensity of the two is calculated. If the difference value is less than the preset safe interval, it means that the emotions of the two are matched, otherwise they are not matched.

[0027] In the above, a plurality of collection time points can be set to obtain the difference value at different time points. If the difference values at multiple collection time points are judged as not matching, the virtual live emotional adjustment process is triggered, including: Calculate the emotional difference value of the two at the current time (the data corresponding to the last collection time point), and gradually match the emotional performance of the virtual host to the user's emotion by controlling the speech rate, fundamental frequency, energy, facial expression weight, etc. The specific steps are: Input the normalized control amount of the virtual host under different control parameters, for example: Adjust the speech rate: speed up or slow down the speech output of the virtual host.

[0028] Adjust the fundamental frequency: change the pitch of the sound.

[0029] Adjust the energy: affect the volume or tone of the voice.

[0030] Adjust the facial expression weight: enhance or weaken the smile, eyebrow bending, etc.

[0031] Using facial landmark detection methods, the coordinate changes of key points such as eyebrows, mouth corners, and eyes are extracted, and the key point changes are mapped to standard facial action units, such as the AU vector corresponding to the upward movement of the mouth corner (AU refers to the action unit in the facial action coding system) when smiling; The facial landmark detection method, such as Active Shape Models (ASM), is a statistical model-based method that detects key points by establishing a statistical model of face shape, learns the variation rules of face shape through a large number of samples, and then searches for the most matched shape in the new image to determine the position of the key points; template matching method can also be used to detect key points by sliding window search and matching existing facial feature templates on the target region.

[0032] The AU vector group is mapped to the emotion intensity, such as happiness, surprise, or anger, through a pre-trained visual emotion classification model, and the visual emotion feature vector of each frame is output ; Through the above method, the audio features in the audio signal are extracted synchronously, including the fundamental frequency, energy, and speech rate, and the speech features are input into the speech emotion recognition model for emotion classification, and the speech emotion feature vector of each frame is output ; The visual emotion feature vector and the speech emotion feature vector are spliced and fused, and the fused virtual anchor emotion vector is output, and the emotion dimension is mapped through a multi-layer perceptron (MLP) , the arousal degree represents the excitement / calmness, and the polarity represents the positive / negative emotion; The normalized control quantity is estimated by recursive least squares method, and the Jacobian matrix is used to quantify the influence of each adjustable feature on the emotion dimension, which provides the basis for online regulation. In the Jacobian matrix, the row represents the emotion dimension index, such as arousal and polarity, and the column represents the control channel index, such as speech rate, fundamental frequency, and energy; The specific method for estimating the Jacobian matrix is as follows: The original control quantity and the original emotion dimension are input and linearly or monotonically mapped to the normalized domain; A small step or pulse is applied on a certain control channel of the control signal, such as speech rate: Wherein, represents the normalized input value of the th control channel, represents the reference value, represents the starting time of the disturbance, represents a small disturbance, which ensures that the system does not enter saturation; Observe the output Find the peak time of the corresponding channel from the curve of change. ; Calculate integer delay ; Output an integer .

[0033] The input is mapped to the original control quantity and original emotion dimension, and integer delay in the normalized domain; Constructing incremental regression pairs: in, This represents the normalized control increment. This represents the gradual increment of the emotion vector. Represents the model residuals; Output the control increment for each step With observation increment ; Input dimension information, including the total number of control channels m, the total number of emotion dimensions n, and the forgetting factor. (Typically 0-1) and the initial uncertainty scalar (usually) ); By each emotion dimension Maintain a separate set of parameters: , Represent the identity matrix, output ; Output the current step , And the previous step ; right Parallel execution: in, express The covariance matrix of each emotion dimension, Indicates the first The incremental observations in each emotional dimension Indicates the first A vector of regression parameters for each emotion dimension; make The i-th line is ; After each update step, the parameters of each row are concatenated to obtain the Jacobian matrix. .

[0034] Furthermore, different control channels have varying abilities to correct for emotional errors; therefore, the adjustment amount should be weighted according to sensitivity, resulting in: Indicates the normalized weights. Indicates the first sensitivity of each control channel, denotes the time instant; assigns the emotion error, i.e. the difference value, to each control channel and adjusts it by its Jacobian matrix (sensitivity): wherein, denotes the control channel target control quantity in the normalized domain, denotes the global gain, ranging from 0 to 1, control update amplitude, denotes the transpose, denotes the difference value.

[0035] The adjustment process needs to be smooth, safe and focused on key control channels to prevent sudden changes in the virtual host's performance, and then: define each channel error (normalized domain) ; wherein, denotes the control channel actual normalized control quantity at the previous time instant, describing the current state of the reference, as the historical execution quantity, denotes the control channel baseline reference value, such as neutral speech rate, as a comparison reference, used to calculate the deviation of the actual output from the expected neutral value, and the calculated channel error is used to drive the PID for adjustment.

[0036] execute PID to track each channel target: wherein, denotes the proportional gain, denotes the integral gain, denotes the derivative gain, denotes the output control quantity of each channel normalized PID, denotes the sampling period, used for integral and derivative calculation, in the dynamic adjustment link of emotion matching, the adjustment of each control channel (speech rate, fundamental frequency, energy, expression, etc.) introduces closed-loop control, so that the adjustment process can gradually eliminate the deviation and maintain smoothness and naturalness, avoiding abruptness and oscillation, thereby ensuring that the emotion adjustment process of the virtual host conforms to the natural feeling of human interaction.

[0037] In the above PID formula, anti-saturation logic can be performed on the formula by integral suppression method, integral feedback method and integral limiting method. The integral suppression method is specifically: When the integral is not updated when the amplitude exceeds the limiting range: ; then , denotes the first The accumulated output of the integral link of the control channel is used to accumulate historical errors and eliminate steady-state deviation.

[0038] To ensure natural transition and prevent mutation and conflict, the following conditions are met: Wherein, The control quantity after amplitude limiting per channel is represented by, The single-step maximum adjustment amplitude per channel is represented by, The target value is limited in the range; The exponential smoothing method is used to filter the control quantity to obtain a smooth transition control output: Wherein, The normalized control quantity after safety and smoothing per channel is represented by, The smoothing coefficient is represented by, , The exponential smoothing time constant is represented by; To ensure that the control quantity is within the physical boundary, the inverse normalization is performed: Wherein, The control quantity issued by the physical domain is represented by, , The upper and lower boundaries of the physical quantity per channel are represented by, The current physical control quantity (such as speech rate, energy, etc.) is represented by, The control quantity to be executed is represented by, which controls the speech rate and other characteristics of the virtual host through the control quantity to be executed, so that the mood of the virtual host matches the mood of the text, and the interaction experience between the user and the virtual host is improved.

[0039] Furthermore, the function, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0040] The logic and / or steps represented in the flow diagrams, or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer.

[0041] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer.

[0042] Furthermore, in order to provide a concise description of exemplary embodiments, not all features of an actual implementation can be described (that is, those pertaining to the best mode for carrying out the present application, in the context of the current consideration, or those that are irrelevant to the implementation of the present application).

[0043] It is to be understood that the development of the exemplary embodiments of this application can not be limited to the precise construction that is shown in the drawings, for purposes of presentation, illustrative embodiments have been chosen that are presently preferred. Modifications of the disclosed embodiments, as well as other implementations, can occur to those skilled in the art upon the reading of this disclosure, and such modifications and implementations are intended to be within the scope of the application. It is also to be understood that not all of the features and aspects described herein are necessarily required for the practice of the present application.

[0044] The above examples are merely illustrative of the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for live streaming of a video based on an intelligent AI large model driven wearable device, characterized in that, The method comprises the following steps: Obtaining the emotional intensity of the text and the multi-modal emotional intensity of the virtual anchor; Determining whether to trigger the emotional regulation process according to the difference value; Mapping the control channel parameters of the virtual anchor to the normalized domain; The control channel parameters include speech rate, fundamental frequency, energy, facial expression weight and motion amplitude; Applying perturbation to each control channel input end and outputting the response of the emotional dimension; According to the perturbation result, the sensitivity of each control channel to the emotional dimension is calculated, and a Jacobian matrix is constructed; The difference value is weighted and distributed among the control channels according to the sensitivity; The target control amount obtained by different control channels is proportional to the sensitivity in the Jacobian matrix; Performing PID control on the target control amount of each control channel; Performing adjustment amplitude limitation on the PID output result of each control channel, smoothing the actual execution control amount, and inputting it to the virtual anchor for adjustment. 2.The live streaming method of wearable device according to claim 1, characterized in that, The determination method of the fundamental frequency is: Divide the audio signal into multiple short-time frame signals, perform fast Fourier transform, and output the complex spectrum of each frame; According to the complex spectrum, the amplitude spectrum is calculated, and the local maximum value detection is performed on the amplitude spectrum to identify the spectral peak, i.e. the peak value; According to the gender characteristics of the speech signal, set the frequency range to output new candidate spectral peaks; Verify whether the candidate spectral peak has a harmonic structure, including: According to the frequency of the candidate spectral peak, peak value detection is performed at its integer multiples, and if there is a peak value, the amplitude of the harmonic position is compared with the amplitude of the corresponding candidate spectral peak; If the amplitude of the harmonic position is greater than 1 / 3 of the amplitude of the corresponding candidate spectral peak, the candidate spectral peak passes the verification; Sort the candidate spectral peaks that pass the verification, determine the candidate spectral peak corresponding to the fundamental frequency according to the amplitude spectrum, and output the preliminary fundamental frequency; Verify the fundamental frequency according to the autocorrelation function, and determine the final fundamental frequency. 3.The wearable device live streaming method of claim 1, wherein, The determination method of the motion amplitude is: Capture the 3D motion of the virtual anchor, including the three-dimensional coordinates of each joint at the time point, to obtain the three-dimensional coordinates at the initial time and the end time; Subtract the three-dimensional coordinates at the initial time from the three-dimensional coordinates at the end time to obtain the displacement in the time window; Calculate the displacement to obtain the displacement of each joint in the time window; Calculate the actual distance by Euclidean distance, and assign corresponding weight coefficients according to the joint type; Calculate the motion amplitude of the virtual anchor by weighted summation of all joint distances.

4. The wearable device live video streaming method of claim 1, wherein, The method for determining whether to trigger the emotional regulation process is: Standardize the facial expression weight, fundamental frequency, energy, speech rate and motion amplitude; Map each one to the emotional intensity through the sigmoid function; Weighted summation obtains the total emotional intensity of the virtual anchor; Obtain the total emotional intensity of the text and the total emotional intensity of the virtual anchor, calculate the difference value of the emotional intensity of the two, and if the difference value is less than the preset safe interval, it means that the emotions of the two match, otherwise they do not match; Set multiple collection time points to obtain the difference value at different time points, and if the difference value at multiple collection time points is determined to be mismatched, the emotional adjustment process is triggered. 5.The wearable device live streaming method of claim 1, wherein, The construction method of the Jacobian matrix is: Input the original control amount and the original emotional dimension; Map the original control amount and the emotional dimension to the normalized domain; Select a control channel, apply a step or pulse disturbance, and observe the emotional dimension output; Locate the peak moment, calculate the integer delay; Input the original control quantity, emotional dimension, and integer delay in the normalized domain; Construct an incremental regression pair; Online recursive update, construct the Jacobian matrix.

6. The wearable device live streaming method of claim 1, wherein, The method for performing PID control on the target control quantity of each control channel is: Assign the difference value to each control channel and adjust the target control quantity according to the Jacobian matrix, i.e., the sensitivity; Calculate the deviation according to the historical execution quantity and the baseline reference value of the control channel, and output the channel error according to the target control quantity and the deviation; Input the channel error into the discrete PID controller, and accumulate the output control quantity according to the proportional term, the integral term, and the differential term. 7.The wearable device live streaming method of claim 1, wherein, Filter the control quantity using the exponential smoothing method to obtain a smooth transition control output, wherein: To ensure that the control quantity is within the physical boundary, output the control quantity to be executed through inverse normalization.

8. The wearable device live video streaming system, applied to the wearable device live video streaming method of any one of claims 1-7, characterized in that, It includes: An emotional intensity analysis module that obtains the emotional intensity of the text and the multi-modal emotional intensity of the virtual anchor, and determines whether to trigger the emotional regulation process according to the difference between the two; An emotional parameter regulation module that maps the control channel parameters of the virtual anchor to the normalized domain, including speech rate, fundamental frequency, energy, facial expression weight, and motion amplitude, applies a disturbance at the input end of each control channel, and outputs the response of the emotional dimension. According to the disturbance result, calculate the sensitivity of each control channel to the emotional dimension and construct the Jacobian matrix, assign the difference value to each control channel according to the sensitivity, and the target control quantity obtained by different control channels is proportional to the sensitivity in the Jacobian matrix, perform PID control on the target control quantity of each control channel, and perform adjustment amplitude limitation on the PID output result of each control channel. Smoothly output the actual execution control quantity. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 7.

Images