Voice data extraction method and system adopting artificial intelligence

Abstract

The invention relates to a voice data extraction method and system adopting artificial intelligence, and relates to the technical field of voice intelligent extraction, and the method comprises the steps: monitoring and collecting video and audio data generated by video communication, and obtaining a corresponding data sequence; carrying out speech speed and volume identification on the audio sequence, obtaining characteristic parameters, and respectively configuring mouth shape and audio weight to obtain a first group of weight and a second group of weight; performing mouth shape image proportion analysis on the video sequence, and obtaining a third group of weights and a fourth group of weights in combination with vocabulary analysis of historical voice data of the user; and fusing the four groups of mouth shapes and audio weights, and performing voice content recognition extraction on the audio and video sequence to obtain final voice data. The problems that in the prior art, dependence on voice data extraction is single, the voice data extraction quality is poor, and personalized adaptation is lacked are solved.

Description

Technical Field

[0001] This application relates to the field of voice intelligent extraction, and particularly to a method and system for extracting voice data using artificial intelligence. Background Art

[0002] With the wide application of video communication technology, voice data extraction has become a core link in information exchange and processing in fields such as intelligent customer service and online meetings. These data play a crucial role in achieving efficient human-machine communication, content analysis and storage, and understanding user needs.

[0003] Most of the existing voice data extraction technologies rely on a single voice natural language recognition method. When facing video communication scenarios, due to poor audio quality (such as background noise, too fast speech rate, too low volume, etc.), the accuracy and integrity of voice data extraction are often poor, making it difficult to meet the high-quality information extraction requirements in complex application scenarios. Summary of the Invention

[0004] To solve the above technical problems, this application provides a method and system for extracting voice data using artificial intelligence, which overcomes the defects of single voice audio recognition method and poor extraction quality in the prior art, improves the accuracy and stability of voice data extraction in complex scenarios, and realizes reliable extraction under poor conditions such as noisy environment and fast speech rate.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, the embodiments of this application provide a method for extracting voice data using artificial intelligence, the method comprising: Monitoring and collecting video data and audio data generated by video communication to obtain a video data sequence and an audio data sequence; According to the audio data sequence, performing speech rate recognition and volume recognition to obtain speech rate characteristic parameters and volume characteristic parameters, and respectively performing mouth shape weight and audio weight configuration to obtain a first mouth shape weight, a first audio weight, a second mouth shape weight, and a second audio weight; Performing mouth shape image ratio analysis on the video data sequence, processing to obtain a third mouth shape weight and a third audio weight, and performing vocabulary analysis according to the historical voice data extraction records of the target user, processing to obtain a fourth mouth shape weight and a fourth audio weight; According to the first mouth shape weight, the first audio weight, the second mouth shape weight, the second audio weight, the third mouth shape weight, the third audio weight, the fourth mouth shape weight, and the fourth audio weight, processing to obtain a fused mouth shape weight and a fused audio weight, and performing speech content recognition and extraction on the video data sequence and the audio data sequence to obtain voice data.

[0006] In a second aspect, an embodiment of the present application provides a speech data extraction system using artificial intelligence, the system comprising: A data acquisition module, configured to monitor and acquire video data and audio data generated by video communication, and obtain a video data sequence and an audio data sequence; An audio feature analysis and weight configuration module, configured to perform speech rate recognition and volume recognition according to the audio data sequence, obtain a speech rate feature parameter and a volume feature parameter, and perform mouth shape weight and audio weight configuration respectively to obtain a first mouth shape weight, a first audio weight, a second mouth shape weight, and a second audio weight; A video and vocabulary feature analysis and weight configuration module, configured to perform an analysis of the proportion of mouth shape images in the video data sequence, process to obtain a third mouth shape weight and a third audio weight, and perform vocabulary analysis according to the historical speech data extraction record of the target user, and process to obtain a fourth mouth shape weight and a fourth audio weight; A weight fusion and speech recognition module, configured to process according to the first mouth shape weight, the first audio weight, the second mouth shape weight, the second audio weight, the third mouth shape weight, the third audio weight, the fourth mouth shape weight, and the fourth audio weight to obtain a fused mouth shape weight and a fused audio weight, and perform speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data.

[0007] One or more technical solutions provided in the present application have at least the following technical effects or advantages: The present application proposes a speech data extraction method and system using artificial intelligence. By collecting audio data in video communication, comprehensively analyzing audio features, video mouth shape features, and the user's historical language habits, combining the audio-visual recognition results with dynamic fusion weights. Synchronously processing multi-form data such as video and audio, dynamically adjusting weights according to speech rate, volume, proportion of mouth shape images, and the user's historical vocabulary usage, and adapting according to the user's language habits, effectively improving the accuracy and stability of speech data extraction, achieving efficient speech recognition under poor conditions such as low-quality audio and complex speech rates, and reducing the recognition error rate. Through steps such as audio feature analysis process, mouth shape weight calculation, and user habit adaptation, multiple types of data are combined and various influencing factors are accurately evaluated, effectively avoiding the problem of extraction failure caused by relying on a single factor and environmental changes. At the same time, the dynamic weight adjustment effectively improves the environmental adaptability and application generalization of speech data extraction.

[0008] The technical solution provided in the present application solves the problems existing in the existing speech data extraction technology, such as the dependence on a single factor in speech data extraction, poor quality of speech data extraction, and lack of personalized adaptation. Description of the Drawings

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0010] Figure 1 It is a schematic flowchart of a method for extracting voice data using artificial intelligence provided by an embodiment of the present application; Figure 2 It is a schematic structural diagram of a system for extracting voice data using artificial intelligence provided by an embodiment of the present application; In the accompanying drawings, the components represented by each reference numeral are described as follows: Data acquisition module 01, audio feature analysis and weight configuration module 02, video and vocabulary feature analysis and weight configuration module 03, weight fusion and speech recognition module 04. Specific embodiments

[0011] The present application provides a method and system for extracting voice data using artificial intelligence, which are used to solve the technical problems existing in the prior art, such as the dependence on a single method for voice data extraction, poor quality of voice data extraction, and lack of personalized adaptation.

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0013] In the description of the present application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0014] In the description of the present application, the term "for example" is used to mean "serving as an example, illustration, or explanation". Any embodiment described as "for example" in this application is not necessarily to be construed as more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not elaborated in detail so as not to obscure the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but rather to be in line with the broadest scope consistent with the principles and features disclosed in this application.

[0015] Embodiment 1, as shown in the attached Figure 1 This application provides a method for extracting voice data using artificial intelligence. The method includes the following steps: S100: Monitor and collect the video data and audio data generated by video communication to obtain a video data sequence and an audio data sequence; In the embodiment of this application, during the process of voice data extraction, in order to achieve synchronous processing and reliable analysis of video information and audio information, it is necessary to obtain complete and valid original video and audio data. Then, the collected original video and audio data are converted into a data sequence form that can be used for subsequent analysis and processing, laying a data foundation for subsequent weight calculation based on audio features, voice content recognition, and fusion processing, and avoiding problems such as low processing efficiency and information omission caused by inconsistent data sequences and non-corresponding orders.

[0016] Specifically, the collection of video data is achieved by monitoring the video stream during video communication. The video capture device (such as a mobile phone camera) captures the video images during the communication process, and then uses data conversion technology to convert the captured video images into a video data sequence in a unified standard digital format, providing basic data support for subsequent voice data extraction and analysis.

[0017] Among them, the data conversion technology captures the analog video signal in the communication through the video capture device, converts it into a digital signal through analog-to-digital conversion technology, that is, performs standard compression using encodings such as H.264, and then organizes it into a video data sequence in a storage format such as MP4. After unifying parameters such as format and frame rate, the conversion from the original image to the standardized digital video is completed.

[0018] Further, the audio data generated by video communication is collected by deploying an audio collection device (such as a microphone) on the communication device, listening to the audio signal in the communication in real time, converting the analog audio into a digital audio signal, and forming an audio data sequence according to a certain sampling frequency and quantization accuracy, so as to uniformly process and analyze the video and audio data subsequently.

[0019] Among them, the collection of video data and audio data supports two modes: real-time monitoring and extraction after recording. Exemplarily, an audio-visual segment with a duration of 2 s is intercepted. In the real-time mode, the audio-visual data is synchronously captured by the audio-visual collection device deployed on the communication device; the recording mode supports extracting the corresponding segment from the stored video or call record. The collected audio-visual data will be uniformly converted into a standardized digital format sequence to ensure the consistent processing of the subsequent voice extraction and analysis process.

[0020] S200: According to the audio data sequence, perform speech rate recognition and volume recognition, obtain the speech rate feature parameter and the volume feature parameter, respectively configure the mouth shape weight and the audio weight, and obtain the first mouth shape weight, the first audio weight, the second mouth shape weight, and the second audio weight; In the embodiment of the present application, during the voice data extraction process, in order to ensure the accuracy and reliability of the voice data extraction, it is necessary to perform speech rate recognition and volume recognition processing on the real-time acquired video and audio data sequences respectively.

[0021] Specifically, through the preset recognition rules and algorithms, the audio data sequence is recognized and analyzed to obtain the speech rate feature parameter and the volume feature parameter. Further, according to the obtained speech rate feature parameter and volume feature parameter, they are input into the weight calculation module.

[0022] Further, this module analyzes and processes the input feature parameters and configures the mouth shape weight and the audio weight according to the preset weight configuration rules and algorithms, and obtains the output first mouth shape weight, the first audio weight, the second mouth shape weight, and the second audio weight.

[0023] The weight parameters obtained according to these steps can effectively optimize the voice data extraction process, ensure the accuracy of the subsequent voice data extraction, and avoid problems such as voice data extraction errors or losses caused by factors such as too fast speech rate and too low volume.

[0024] The steps in the method provided by the embodiment of the present application include: Input the audio data sequence into a speech rate recognizer to recognize and obtain the speech rate feature parameter; Extract the average volume in the audio data sequence as the volume feature parameter; According to the speech rate characteristic parameters, configure the lip shape weight and audio weight to obtain the first lip shape weight and the first audio weight. According to the volume characteristic parameters, configure the lip shape weight and audio weight to obtain the second lip shape weight and the second audio weight.

[0025] In the embodiment of the present application, the obtained audio data sequence is transmitted to a pre-constructed speech rate recognizer. Based on preset algorithms and rules, the recognizer analyzes the speech rate in the audio data sequence. By analyzing information such as the time interval and syllable distribution of the speech signal, it calculates and outputs characteristic parameters that can represent the speed of speech.

[0026] The steps of "construction of the speech rate recognizer" in the method provided by the embodiment of the present application include: Using machine learning, construct the network structure of the speech rate recognizer. Among them, the input data of the speech rate recognizer is the audio data sequence, and the output data is the speech rate characteristic parameters; According to the speech data recognition records within the historical time, collect a set of sample audio data sequences, and label the speech rates of different sample audio data sequences to obtain a set of sample speech rate characteristic parameters; Use the set of sample audio data sequences and the set of sample speech rate characteristic parameters to perform supervised training and testing on the speech rate recognizer until convergence.

[0027] In the embodiment of the present application, first, based on the machine learning method, construct the network architecture of the speech rate recognizer, and set its input as the audio data sequence and the output as the characteristic parameters representing the speech rate.

[0028] Among them, the speech rate recognizer network extracts and analyzes hierarchical features of audio data through a multi-layer neural network structure. This network design includes a feature preprocessing module, a time-frequency conversion module, a feature extraction module, and a parameter output module. Each module operates in coordination to achieve the analysis of the speech rate characteristics of the audio data sequence.

[0029] Specifically, receive the audio data sequence through the input interface. First, the audio data is processed by the feature preprocessing module for noise reduction, format unification, etc.; then, the audio signal is converted from the time domain to the frequency domain through the time-frequency conversion module; subsequently, according to the preset rules of the feature extraction module, details such as the time interval and frequency distribution of the speech signal are analyzed, and its speech rate characteristics are further calculated; finally, the quantized speech rate characteristic parameters are output through the parameter output module.

[0030] After repeated debugging and optimization of the above steps, after determining the optimal parameters and operation logic of each module, the construction of the speech rate recognizer is completed, and thus a complete audio data processing flow is formed, ensuring that the speech rate characteristics in the audio can be stably extracted and reliable speech rate characteristic parameters can be output.

[0031] Further, by collecting historical voice data recognition records, screening and constructing a set of sample audio data sequences, and manually annotating the speech rates of each sample audio data sequence, a corresponding set of sample speech rate feature parameters is formed.

[0032] Among them, the construction of the set of sample audio data sequences includes voice data in different scenarios such as formal reports and daily conversations, as well as voices of people of all ages, genders, different accents, and speaking styles. After removing invalid audio segments through audio detection, combined with a format conversion tool and parameter adjustment, the valid audio is unified into a standard coding format, and finally a set of sample audio data sequences containing various voice characteristics is formed, providing reliable data support for speech rate recognition.

[0033] Further, according to a unified standard, the speech rates of the sample audio data sequences are measured and annotated one by one to generate a corresponding set of sample speech rate feature parameters, providing a reliable basis for subsequent training optimization.

[0034] Taking the two sets of sample sets as training data, training and testing are carried out on the speech rate recognizer. By repeatedly adjusting the network parameters and optimizing the model structure through multiple iterations, and continuously verifying the training effect until the recognition result is stable.

[0035] Specifically, during the training process, the sample audio data sequences are input into the recognizer. By comparing the output speech rate feature parameters with the annotated values, after calculating the error, the processing parameters of the feature preprocessing module, time-frequency conversion module, etc. are adjusted accordingly, and the feature extraction processes such as the analysis of the time interval of the speech signal and the calculation of the frequency distribution are optimized.

[0036] Further, by repeatedly testing different module combination methods and data processing orders, the architecture of the speech rate recognizer is effectively adjusted. After multiple rounds of iterative training, when the error fluctuations of consecutive multiple trainings are controlled within a very small range, such as the error rate is lower than 0.5%, it is considered that the model reaches the convergence state, ensuring that it can accurately extract the speech rate feature parameters of various audio data.

[0037] In the embodiment of the present application, the input audio data sequence is scanned frame by frame through an audio processing algorithm, the instantaneous volume value of each frame of the audio signal is calculated, and then the volume values of all frames are subjected to an arithmetic mean operation. The resulting value is the average volume of the audio data sequence, which is output as a volume feature parameter representing the audio loudness.

[0038] Among them, the audio processing method in the above steps can use the short-time energy algorithm to process the input audio data sequence.

[0039] Specifically, first divide the collected audio data into multiple short frames, calculate the data and its energy value for each frame, and this energy value is usually related to the volume. Then sum up the energy values of each frame and divide by the total number of frames to calculate the average energy of the audio data sequence, and then convert it into the corresponding average volume value, which is used as the volume feature parameter representing the audio loudness for output.

[0040] In the method provided by the embodiment of the present application, the steps of "configuring the mouth shape weight and the audio weight according to the speech rate feature parameter to obtain the first mouth shape weight and the first audio weight, and configuring the mouth shape weight and the audio weight according to the volume feature parameter to obtain the second mouth shape weight and the second audio weight" include: Obtain the average speech rate feature parameter and the average volume feature parameter according to the speech data recognition records within the historical time; Calculate the ratio of the average speech rate feature parameter to the speech rate feature parameter, perform a correction calculation on the preset audio weight to obtain the first audio weight, and calculate the first mouth shape weight according to the first audio weight; Calculate the ratio of the volume feature parameter to the average volume feature parameter, perform a correction calculation on the preset audio weight to obtain the second audio weight, and calculate the second mouth shape weight according to the second audio weight.

[0041] In the embodiment of the present application, according to the speech data recognition records stored historically, all the speech data within the specified historical time period is screened out.

[0042] Further, for the speech rate feature parameter and the volume feature parameter in each data record, perform a summation operation respectively, and then divide the summation result by the total number of data records within this historical time period, thereby calculating the average speech rate feature parameter and the average volume feature parameter.

[0043] Further, calculate the ratio of the average speech rate feature parameter to the current speech rate feature parameter, specifically divide the average speech rate feature parameter by the current speech rate feature parameter to obtain the ratio of the two, that is: ratio = average speech rate feature parameter / current speech rate feature parameter.

[0044] Further, use this ratio to perform a correction calculation on the preset audio weight, specifically multiply the preset audio weight by this ratio to obtain the first audio weight, that is: first audio weight = preset audio weight × ratio.

[0045] Among them, the preset audio weight is a parameter preset in the speech data extraction system to measure the importance of audio information in speech content recognition. The value is usually between 0 and 1, representing the initial importance of audio information in the speech content recognition process. For example, when the preset audio weight is 0.58, it means that before adjusting by combining other factors such as speech rate and volume, the system defaults that the proportion of audio information in speech content recognition is 58%, while other information such as lip shape accounts for 42%.

[0046] Furthermore, according to the constraint relationship that the sum of the lip shape weight and the audio weight is 1, the first lip shape weight is calculated as specifically 1 minus the first audio weight, obtaining the first lip shape weight, that is: First lip shape weight = 1 - First audio weight.

[0047] Exemplarily, the preset audio weight is 0.58. If the average speech rate feature parameter is 120 words / min and the current speech rate feature parameter is 100 words / min, then the ratio of the average speech rate feature parameter to the current speech rate feature parameter is 120 / 100 = 1.2. At this time, the first audio weight is 1.2×0.58 = 0.696 (taking the approximate value 0.7), and the first lip shape weight is 1 - 0.696 = 0.304 (taking the approximate value 0.3).

[0048] Similarly, summarize the volume feature parameters in all data records within the historical time period, sum them up and divide by the total number of data records to obtain the average volume feature parameter, which reflects the average level of historical volume.

[0049] Furthermore, obtain the volume feature parameter of the current speech data, and divide it by the average volume feature parameter to obtain the ratio of the two, that is: Ratio = Average volume feature parameter / Volume feature parameter of the current speech data. This ratio reflects the degree of change of the current volume compared to the historical average volume.

[0050] Furthermore, use this ratio to correct the preset audio weight. Multiply the preset audio weight by this ratio to obtain the second audio weight. If the ratio is greater than 1, it means that the current volume is higher than the historical average level, and the second audio weight will increase accordingly; otherwise, it will decrease.

[0051] Furthermore, based on the constraint condition that the sum of the lip shape weight and the audio weight is always 1, subtract the second audio weight from 1 to calculate the second lip shape weight, so that the audio and lip shape weights are adjusted with each other as the volume changes, ensuring the reasonable distribution of the weights of the two in the speech recognition process and improving the recognition accuracy.

[0052] Exemplarily, the preset audio weight is 0.5. If the average volume feature parameter is 60 decibels and the current volume feature parameter is 60 decibels, then the ratio of the current volume feature parameter to the average volume feature parameter is 60 / 60 = 1. At this time, the second audio weight is 1×0.5 = 0.5, and the second lip weight is 1 - 0.5 = 0.5.

[0053] In the embodiments of the present application, by constructing a speech rate recognizer and through the weight calculation method of the above steps, the dynamic adjustment between the lip shape and the audio weight of the speech rate feature is realized, effectively improving the accuracy and stability of speech data extraction in complex scenarios.

[0054] Exemplarily, when the speech rate is too fast, the speech signal is likely to be blurred or overlapped, resulting in an increase in the difficulty of audio recognition and a decrease in quality. The audio weight will be automatically reduced according to the preset weight adjustment rule; when the speech volume is small, the speech signal is more vulnerable to background noise interference, resulting in poor audio quality. Similarly, the audio weight will be reduced according to the established rule. By the above steps, the proportion of video lip information in the process of speech data extraction is increased, ensuring the accuracy of speech data extraction.

[0055] S300: Perform an analysis on the proportion of lip images in the video data sequence, process to obtain the third lip weight and the third audio weight, and perform a vocabulary analysis based on the historical speech data extraction records of the target user, and process to obtain the fourth lip weight and the fourth audio weight; In the embodiments of the present application, when configuring the lip and audio weights, in order to collaboratively process the video image features and speech data features and achieve precise configuration, it is necessary to perform an analysis on the proportion of lip images in the video data sequence and perform a vocabulary analysis on the historical speech data of the target user.

[0056] Furthermore, the above analysis results are converted into corresponding weight parameters through a predefined algorithm, and subsequently, the lip and audio weights are dynamically adjusted according to the actual picture situation and the user's usage habits to achieve personalized speech interaction.

[0057] Among them, in the process of configuring the lip and audio weights, a lip proportion recognizer constructed by a convolutional neural network is used to process the video data sequence. Through the multi-layer convolution, pooling, and fully connected operations of the recognizer, the lip image proportion coefficients of each video frame are output. After averaging these coefficients and comparing with a preset reference value, the weight parameters are corrected according to a predefined proportional relationship, and then the third lip weight and the third audio weight are obtained.

[0058] Furthermore, perform a vocabulary analysis on the historical speech data of the target user, calculate to obtain the fourth lip weight and the fourth audio weight, and combine them with the third set of weights to form a final configuration scheme.

[0059] Step S300 in the method provided by the embodiments of this application includes: Input each video data in the video data sequence into the mouth shape ratio recognizer, and recognize to obtain a plurality of mouth shape image ratio coefficients, where the mouth shape ratio recognizer is constructed using a convolutional neural network and is trained using a sample video data set and a labeled sample mouth shape image ratio coefficient set; Obtain the average mouth shape image ratio coefficient; Calculate the ratio of the mean of the plurality of mouth shape image ratio coefficients to the average mouth shape image ratio coefficient, perform a correction calculation on the preset mouth shape weight, and obtain the third mouth shape weight; Calculate the third audio weight according to the third mouth shape weight.

[0060] In the embodiments of this application, when configuring the mouth shape and audio weights, in order to achieve the collaborative and accurate configuration of video image features and speech data features, a mouth shape ratio recognizer constructed based on a convolutional neural network is used to process each frame of the video data sequence.

[0061] Among them, the recognizer performs arithmetic analysis on each frame of video image through multiple convolutional, pooling, and fully connected structures, and outputs a plurality of mouth shape image ratio coefficients.

[0062] Specifically, when constructing the mouth shape ratio recognizer, using the architecture of the convolutional neural network, sliding convolution is performed on the video frame through the convolutional kernel, and the detailed features from the bottom layer to the high layer such as the edge contour, geometric shape, and dynamic changes of the mouth shape area in the video are extracted layer by layer; the feature map is further dimension-reduced through the pooling layer, while retaining the key information, reducing the computational complexity. The recognizer can, according to the established program rules, recognize the key information of the mouth shape and size changes in the video frame by frame, as well as the dynamic trend of the mouth shape in the continuous video frames, and clearly analyze the mouth shape information in the video frame.

[0063] Furthermore, to ensure the accurate operation of the recognizer, it is necessary to construct a sample video data set that includes different scenarios, characters, and mouth shape change situations (including different scenarios such as formal reports and daily conversations, as well as video data of men, women, the elderly, and children with different accents and speaking styles), and at the same time, using the method of manual annotation, calculate the ratio coefficient of the mouth shape image in each video in the picture, and collect and form a standard sample data set.

[0064] In the training stage of the recognizer, input the sample video data into the recognizer network segment by segment, and gradually adjust the internal arithmetic parameters and processing logic of the recognizer through the backpropagation algorithm. Taking the manually labeled ratio coefficient as a reference, continuously optimize the calculation process and judgment criteria of the recognizer, and reduce the error between the prediction result and the actual ratio.

[0065] Furthermore, after multiple iterations of training and parameter optimization, the mouth shape proportion recognizer can stably and efficiently identify and process the input video data sequence, accurately calculate the mouth shape image proportion coefficient of each video frame, and provide key data support for subsequent mouth shape weight configuration and mouth shape-audio collaborative processing.

[0066] After constructing and annotating the sample video data set, summarize the mouth shape image proportion coefficients of all sample videos, and obtain the average mouth shape image proportion coefficient by summing and then dividing by the total number of samples.

[0067] Among them, this coefficient, as a reference value for measuring the proportion of the mouth shape image in the video frame, can be used to compare with the analysis results of the mouth shape image proportion of the actual video data sequence in the future, and provide an important reference basis for the correction and configuration of the weight parameters.

[0068] Furthermore, summarize the multiple mouth shape image proportion coefficients output after processing the video data sequence by the mouth shape proportion recognizer, and calculate the mean value of this group of coefficients through arithmetic mean.

[0069] Furthermore, perform a ratio operation on the mean value obtained in the above step and the pre-calculated average mouth shape image proportion coefficient to obtain a correction coefficient, that is: correction coefficient = average mouth shape image proportion coefficient after recognizer processing / pre-calculated average mouth shape image proportion coefficient.

[0070] Furthermore, based on this correction coefficient, perform a multiplication operation on the preset mouth shape weight, thereby completing the adjustment and optimization of the preset mouth shape weight, and finally obtaining the third mouth shape weight applicable to the current video data characteristics.

[0071] Among them, the preset mouth shape weight is a parameter preset to measure the importance of mouth shape information in speech content recognition, representing the initial importance of mouth shape information in the speech content recognition process, and the value is usually between 0 and 1.

[0072] Furthermore, after obtaining the third mouth shape weight, based on the constraint relationship that the sum of the mouth shape weight and the audio weight is 1, the third audio weight can be obtained through subtraction operation, that is: third audio weight = 1 - third mouth shape weight.

[0073] Exemplarily, the preset mouth shape weight is 0.25. If the mouth shape image proportion coefficients output after processing the video data sequence by the mouth shape proportion recognizer are 0.1, 0.2, 0.3, and 0.2 respectively, then their mean value is (0.1 + 0.2 + 0.3 + 0.2) / 4 = 0.2. The pre-calculated average mouth shape image proportion coefficient is 0.25, then the correction coefficient is 0.2 / 0.25 = 0.8. At this time, the third mouth shape weight is 0.25 × 0.8 = 0.2, and the third audio weight is 1 - 0.2 = 0.8.

[0074] The calculation method of the above steps ensures that during the video playback, the weight distribution between lip display and audio playback always remains balanced, thus achieving the collaborative optimization of the two and providing users with a more natural and smooth audio-visual experience.

[0075] Exemplarily, when the video frame focuses on a close-up of a person and the mouth area of the person occupies a relatively large proportion of the frame, if the calculated proportion of the lip image area reaches 70%, it indicates that the lip information is clear and complete, and at this time, a higher lip weight will be assigned; if in a video frame of a long-shot, the mouth area of the person only accounts for 5% or even less of the frame, or when the mouth is completely out of the frame due to a camera switch (the proportion is 0), the corresponding lip weight will also be significantly reduced. By correlating the proportion value of the lip image area with the weight, the differential processing of lip information in different video scenarios is achieved.

[0076] The step of "extracting the historical voice data records of the target user, performing vocabulary analysis, and processing to obtain the fourth lip weight and the fourth audio weight" in the method provided by the embodiment of the present application includes: Collecting the historical extracted voice data set of the user for voice data extraction within the historical time; Counting the number of occurrences of rare words in the historical extracted voice data set to obtain the number of occurrences of rare words; Collecting the number of occurrences of rare words of multiple sample users and calculating the average value to obtain the average number of occurrences of rare words; According to the ratio of the average number of occurrences of rare words to the number of occurrences of rare words, correcting and calculating the preset lip weight to obtain the fourth lip weight; Calculating the fourth audio weight according to the fourth lip weight.

[0077] In the embodiment of the present application, multiple users with different ages, occupations, and language habits are selected as sample objects, and the voice data generated by them in scenarios such as voice chat, voice command input, and voice course learning within the historical time is captured through a built-in voice data collection device (such as a microphone, etc.).

[0078] Among them, during the collection process, according to the established data collection standards and operation procedures, the obtained voice data is processed. By uniformly converting the data storage format, for example, uniformly converting voice data from different sources into common storage formats such as MP3 and WAV, and uniformly adjusting parameters such as audio sampling rate and bit depth, ensuring that the formats of all voice data are standardized and consistent. At the same time, identifying and removing duplicate voice segments to ensure the accuracy and effectiveness of the collected data.

[0079] Further, the valid voice data that has completed format unification and duplicate elimination processing is classified according to dimensions such as data generation scenarios, usage time, and user types. For example, according to the voice interaction scenario, the voice data is divided into categories such as voice commands, voice chats, and voice learning; or classified in monthly or quarterly cycles according to the time sequence.

[0080] After the classification is completed, the data of each part is summarized and integrated, and archived and stored according to the standardized naming rules and storage paths, finally forming a complete set of historical extracted voice data. This set can provide reliable data support for further analyzing users' voice usage preferences, calculating lip shape and audio weight parameters, etc. in the future.

[0081] Further, all the voice contents in the set of historical extracted voice data are converted into text format, and compared with the pre-organized rare word dictionary one by one. During the verification process, each word in the text is carefully distinguished. If a certain word is found to be the same as the rare word in the dictionary, it is marked and counted in a special counting table or system, and the corresponding count value is increased, so as to accurately count the number of occurrences of rare words.

[0082] Further, when all the text content verification work is completed, the final cumulative value in the counting table is the number of occurrences of rare words in this set of historical extracted voice data. This value reflects the frequency of use of rare words by users during voice expression, providing an important data basis for accurately calculating weight parameters in the future.

[0083] Further, sum up the number of occurrences of rare words of all sample users, calculate the total number through summation operation, and then divide the total number by the total number of sample users to obtain the average number of occurrences of rare words. The specific calculation formula is: average number of occurrences of rare words = sum of the number of occurrences of rare words / total number of sample users. This calculation result can intuitively reflect the overall frequency level of rare word usage by the sample user group.

[0084] Further, by calculating the ratio of the number of occurrences of rare words of a single sample user to the average number of occurrences of rare words, evaluate the relative level of the rare word usage frequency of this user in the group, specifically expressed as: ratio = number of occurrences of rare words of a single sample user / average number of occurrences of rare words.

[0085] Specifically, if the ratio is greater than 1, it indicates that the user's frequency of using rare words is higher than the group average. Since there is less lip shape training data corresponding to rare words, in order to reduce the lip shape recognition deviation, the preset lip shape weight is adjusted downward according to the preset rules; if the ratio is less than 1, it means that the user's frequency of using rare words is lower than the average, and the preset lip shape weight can be appropriately increased.

[0086] Through the above steps, the preset mouth shape weight is corrected and calculated to obtain the fourth mouth shape weight that fits the user's personalized language usage habits. This weight parameter fully considers the influence of the user's rare word usage on mouth shape recognition, closely fits the user's personalized language usage habits, and can more accurately adapt to the user's speech features.

[0087] Further, based on the constraint relationship that the sum of the mouth shape weight and the audio weight is 1, after determining the fourth mouth shape weight, through subtraction, that is, 1 minus the fourth mouth shape weight, the corresponding fourth audio weight can be obtained.

[0088] Exemplarily, the preset mouth shape weight is 0.5. If there are 5 sample users, and the number of occurrences of rare words for them are 3 times, 4 times, 2 times, 5 times, and 6 times respectively, after calculation, the sum of the number of occurrences of rare words is 20 times, and the average number of occurrences of rare words is 20 / 5 = 4 times. For a target user, the number of occurrences of rare words is 2 times, and the ratio is 2 / 4 = 0.5, which is less than 1. According to the rule of "upward adjustment coefficient = 0.5×(1 - ratio)", the upward adjustment coefficient is 0.5×(1 - 0.5) = 0.25. Then the fourth mouth shape weight is 0.5×(1 + 0.25) = 0.625 (taking the approximate value of 0.6), and the fourth audio weight is 1 - 0.6 = 0.4.

[0089] Through the above calculation method, the obtained fourth audio weight and the fourth mouth shape weight are mutually adapted. When processing speech and mouth shape data subsequently, resources can be reasonably allocated, the processing efficiency of audio data can be improved, and better coordination between speech and mouth shape can be achieved.

[0090] Exemplarily, in the statistical analysis of user speech data, if the usage proportion of rare characters in the user's historical speech data reaches 30%, due to the small number of standard mouth shape samples corresponding to rare words and phrases, recognition errors are likely to occur during the matching of mouth shape and speech, and the system will assign a lower mouth shape weight; conversely, if the usage proportion of rare characters of the user is only 5%, it indicates that the user mostly uses common words and phrases, and the corresponding mouth shape training samples are rich, and the accuracy of mouth shape recognition is high, and the system will assign a higher mouth shape weight.

[0091] S400: According to the first mouth shape weight, the first audio weight, the second mouth shape weight, the second audio weight, the third mouth shape weight, the third audio weight, the fourth mouth shape weight, and the fourth audio weight, process to obtain the fused mouth shape weight and the fused audio weight, and perform speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data.

[0092] In the embodiments of the present application, during the collaborative processing of speech and video, in order to achieve the precise adaptation of the video picture and the audio content, it is necessary to integrate and optimize multiple groups of lip movement and audio weight parameters, and further analyze the video data sequence and the audio data sequence. By combining the integrated weight parameters with the extracted speech data, it provides a basis for subsequent intelligent matching of speech and video and the provision of personalized audio-visual experiences.

[0093] Among them, during the weight parameter fusion process, the first to fourth lip movement weights and audio weights are summarized, and weighted calculation and optimization processing are performed through a preset algorithm to obtain the fused lip movement weight and the fused audio weight respectively.

[0094] Furthermore, based on the obtained fused lip movement weight and fused audio weight, speech content recognition and extraction are performed on the video data sequence and the audio data sequence. By analyzing the lip movement changes of the video picture frame by frame and parsing the audio content sentence by sentence, the speech information therein is recognized and extracted. The extracted speech data is combined with the fused weights to formulate a final speech-video collaborative processing scheme to achieve the coordinated matching of speech and video.

[0095] The steps in the method provided by the embodiments of the present application for S400 include: Fusing and calculating to obtain the fused lip movement weight and the fused audio weight according to the first lip movement weight, the first audio weight, the second lip movement weight, the second audio weight, the third lip movement weight, the third audio weight, the fourth lip movement weight, and the fourth audio weight; Performing lip movement speech content recognition and extraction and audio speech content recognition and extraction on the video data sequence and the audio data sequence respectively to obtain a lip movement speech content set and an audio speech content set; According to the fused lip movement weight and the fused audio weight, performing weighted fusion processing on the lip movement speech content set and the audio speech content set to obtain speech data, where the weighted fusion processing includes weight identification for each speech content, and screening the speech content with the largest sum of total weights as the speech data.

[0096] In the embodiments of the present application, the obtained first to fourth lip movement weights and audio weights are summarized, and according to a preset weight ratio scheme, through weighted calculation and numerical optimization, the fused lip movement weight and the fused audio weight are obtained respectively, providing a core parameter basis for subsequent speech-video collaborative processing.

[0097] Among them, during the collaborative processing of speech and video, the calculation of the fused lip movement weight and the fused audio weight needs to be completed through multiple steps.

[0098] First is the data preparation stage, where four sets of lip and audio weights calculated under different algorithms such as video frame rate, speech clarity, lip proportion recognition, and rare word frequency analysis are collected. At the same time, according to the system requirements and application scenarios, a priority coefficient is assigned to each set of weights to ensure that the sum of all coefficients is 1.

[0099] Further, enter the weighted calculation stage. By multiplying the weights of the corresponding groups by the coefficients and then accumulating, the fused lip weight and the fused audio weight are calculated respectively. After completing the weighted calculation, enter the numerical optimization stage. If the sum of the fused lip weight and the fused audio weight is not 1, then perform normalization adjustment to ensure that the final weight values are within the effective range of 0 to 1.

[0100] Specifically, apply the calculated fused weights to the speech video processing work. According to the fused lip weight, the details and accuracy of the lip picture can be effectively adjusted; based on the fused audio weight, the accuracy of speech recognition can be optimized and the background noise filtering effect can be enhanced. By coordinating the ratio relationship of the two weights in real time, it is further ensured that the speech playback and the lip movement are in one-to-one correspondence.

[0101] Exemplarily, in the data preparation stage, four sets of lip and audio weights have been obtained, which are (0.3, 0.7), (0.5, 0.5), (0.2, 0.8), (0.6, 0.4), and the corresponding priority coefficients are 0.2, 0.3, 0.2, 0.3. In the weighted calculation stage, multiply each set of lip and audio weights by the coefficients and then accumulate them to obtain the initial fused lip weight of 0.43 and the fused audio weight of 0.57. Since the sum of the two is 1, no normalization adjustment is required. Finally, determine the fused lip weight of 0.43 (approximate value 0.4) and the fused audio weight of 0.57 (approximate value 0.6), which can be applied to optimize the details of the lip picture, the accuracy of speech recognition, and the synchronization adaptation of the video and audio in speech video processing.

[0102] The step of "performing lip speech content recognition extraction and audio speech content recognition extraction on the video data sequence and the audio data sequence according to the video data sequence and the audio data sequence, and obtaining a lip speech content set and an audio speech content set" in the method provided by the embodiment of the present application includes: Input the video data sequence into multiple lip speech content recognizers respectively, and recognize and output to obtain a lip speech content set. Among them, the input data of each lip speech content recognizer is the video data sequence, the output data is the lip speech content, and the training data of each lip speech content recognizer is different; Input the audio data sequence into multiple audio speech content recognizers respectively, and obtain an audio speech content set through recognition output. Among them, the input data of each audio speech content recognizer is the audio data sequence, the output data is the audio speech content, and the training data of each audio speech content recognizer is different. Each audio speech content recognizer is constructed based on a natural language recognition model and is obtained through training by collecting different sample audio data sequences and recognized and labeled sample audio speech contents.

[0103] In the embodiments of the present application, in order to more accurately recognize speech content, a multi-angle and multi-scenario analysis method is adopted, that is, the same video data sequence is sequentially input into multiple lip-sync speech content recognizers. Each recognizer uses the complete video data sequence as the analysis object, analyzes the lip movement changes in the video data, and outputs the corresponding speech content.

[0104] Among them, since each recognizer uses video samples in different scenarios during the training stage, including diverse scenarios such as daily conversations, news broadcasts, and movie dialogues, and video data with different speaking styles, accent features, and speech rates and rhythms, the output results of each recognizer are all different.

[0105] Exemplarily, when the recognizer trained with daily conversation video samples processes the video data sequence of a life-like scenario, it tends to recognize colloquial expressions and common vocabulary; when the recognizer trained with news broadcast video samples is used, it tends to recognize standard expressions and current affairs terms; while when the recognizer trained with movie dialogue video samples is used, it is more inclined to capture emotional expressions and line contents in specific contexts.

[0106] Furthermore, these different recognition results are summarized and integrated to construct a lip-sync speech content set containing multiple possibilities, further providing rich data references for subsequent speech content screening and analysis based on weight analysis.

[0107] Similarly, to achieve accurate recognition of audio data, the same audio data sequence is synchronously input into multiple audio speech content recognizers. Each audio speech content recognizer uses the complete audio data sequence as the input, and outputs the corresponding speech content recognition result by analyzing audio waveform features, voiceprint features, semantic features, etc.

[0108] Among them, since each audio speech content recognizer uses audio samples from different sources during the training stage, including diverse scenarios such as news broadcasts, conference recordings, movie originals, and phone calls, and audio data with different accents, speech rates, emotional expressions, and background noise environments, each recognizer shows significant differences in the recognition method and accuracy distribution.

[0109] Exemplarily, when using a recognizer trained with news broadcast audio samples, it tends to recognize clear and standard Mandarin, as well as current affairs vocabulary and professional terms; when using a recognizer trained with conference recording samples, it tends to recognize scenarios where multiple people speak alternately with slight background noise and can accurately distinguish the speech content of different speakers; when using a recognizer trained with movie and TV drama original sound samples, it is more sensitive to the speech recognition of lines and dialect expressions.

[0110] Furthermore, these different recognition results are integrated and summarized to form an audio speech content set containing multiple possibilities, providing a reliable data basis for subsequent speech content verification, error correction, and optimization.

[0111] Furthermore, according to the obtained fused lip movement weights and fused audio weights, weighted fusion processing is performed on the lip movement speech content set and the audio speech content set to obtain speech data.

[0112] Specifically, first, corresponding fusion weights are marked for each item in the lip movement speech content set and the audio speech content set respectively; then, the lip movement weights and audio weights of candidate speech contents with similar or identical pronunciations in the two sets are added together; finally, the content with the largest total weight is selected from all candidate speech contents and determined as the final speech data, thereby achieving accurate matching of speech and lip movement information.

[0113] Exemplarily, assume that the lip movement speech content set recognizes two "doctors" and the audio speech content set recognizes two "clothes". Since the pronunciations of "doctor" and "clothes" are similar, there is ambiguity in the recognition. At this time, the determined fused lip movement weight of 0.4 is assigned to "doctor", and the fused audio weight of 0.6 is assigned to "clothes". Through calculation, the total weight of "doctor" is its lip movement weight of 0.4, and the total weight of "clothes" is its audio weight of 0.6 plus the lip movement weight of 0.6 (because "clothes" is recognized in both sets), that is, 1.2. By comparison, the total weight of "clothes" of 1.2 is greater than the total weight of "doctor" of 0.4. Therefore, the "clothes" with the largest total weight is selected as the final speech data, effectively solving the recognition problem caused by similar pronunciations and achieving accurate matching of speech and lip movement information.

[0114] Through the above specific implementation manners, the embodiments of the present application achieve the following technical effects: A method and system for extracting speech data using artificial intelligence provided by an embodiment of the present application synchronously collect video and audio data sequences, and use multiple recognizers with different training data for analysis to construct a diverse speech content set, effectively solving the problems of traditional speech extraction relying on a single data source and one-sided information acquisition. Secondly, by collecting information in multiple aspects such as speech rate, volume, proportion of lip image occupancy, and the user's historical vocabulary usage, a dynamic weight configuration mechanism is constructed. By calculating and fusing multiple groups of weight parameters, the accuracy of speech content recognition is effectively improved, and misjudgment caused by similar pronunciations or environmental interference is avoided. Finally, by analyzing the usage frequency of rare words in the user's historical speech data, personalized weight adaptation is realized, and a large number of samples are used for training and optimization to ensure stable and accurate matching of speech and lip information in complex application scenarios with different speaking styles, different accents, video picture features, etc.

[0115] The method and system provided by the embodiment of the present application solve the problems of the existing speech data extraction technology relying on a single source, poor quality of speech data extraction, and lack of personalized adaptation, effectively avoiding recognition errors caused by environmental factors or individual differences, improving the accuracy and reliability of speech data extraction, providing support for the application of speech recognition technology in multiple scenarios, and promoting the development of speech processing technology towards intelligence and precision.

[0116] Embodiment 2, as shown in the appendix Figure 2 Based on the inventive concept of a method for extracting speech data using artificial intelligence provided in Embodiment 1, the present application further provides a system for extracting speech data using artificial intelligence, specifically including: A data acquisition module 01 for monitoring and collecting video data and audio data generated by video communication to obtain a video data sequence and an audio data sequence; An audio feature analysis and weight configuration module 02 for performing speech rate recognition and volume recognition according to the audio data sequence to obtain speech rate feature parameters and volume feature parameters, and respectively performing lip weight and audio weight configuration to obtain a first lip weight, a first audio weight, a second lip weight, and a second audio weight; A video and vocabulary feature analysis and weight configuration module 03 for analyzing the proportion of lip image occupancy of the video data sequence to obtain a third lip weight and a third audio weight, and performing vocabulary analysis according to the historical speech data extraction record of the target user to obtain a fourth lip weight and a fourth audio weight; The weight fusion and speech recognition module 04 is used to process and obtain the fused lip movement weight and the fused audio weight according to the first lip movement weight, the first audio weight, the second lip movement weight, the second audio weight, the third lip movement weight, the third audio weight, the fourth lip movement weight, and the fourth audio weight, and perform speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data.

[0117] In one embodiment, the audio feature analysis and weight configuration module 02 is further used for: Input the audio data sequence into a speech rate recognizer to recognize and obtain the speech rate feature parameters; Extract the average volume in the audio data sequence as the volume feature parameter; According to the speech rate feature parameters, configure the lip movement weight and the audio weight to obtain the first lip movement weight and the first audio weight, and according to the volume feature parameter, configure the lip movement weight and the audio weight to obtain the second lip movement weight and the second audio weight.

[0118] In one embodiment, the video and vocabulary feature analysis and weight configuration module 03 is further used for: Input each video data in the video data sequence into a lip movement proportion recognizer to recognize and obtain a plurality of lip movement image proportion coefficients, where the lip movement proportion recognizer is constructed by a convolutional neural network and trained using a sample video data set and a labeled sample lip movement image proportion coefficient set; Obtain the average lip movement image proportion coefficient; Calculate the ratio of the mean of the plurality of lip movement image proportion coefficients to the average lip movement image proportion coefficient, and perform a correction calculation on the preset lip movement weight to obtain the third lip movement weight; Calculate the third audio weight according to the third lip movement weight.

[0119] Collect the historical extracted speech data set of the user's speech data extraction within the historical time; Count the number of occurrences of rare words in the historical extracted speech data set to obtain the number of occurrences of rare words; Collect the number of occurrences of rare words of multiple sample users and calculate the mean to obtain the average number of occurrences of rare words; Perform a correction calculation on the preset lip movement weight according to the ratio of the average number of occurrences of rare words to the number of occurrences of rare words to obtain the fourth lip movement weight; Calculate the fourth audio weight according to the fourth lip movement weight.

[0120] In one embodiment, the weight fusion and speech recognition module 04 is further used for: Based on the first lip shape weight, the first audio weight, the second lip shape weight, the second audio weight, the third lip shape weight, the third audio weight, the fourth lip shape weight, and the fourth audio weight, fusion calculation is performed to obtain a fused lip shape weight and a fused audio weight; Perform lip movement speech content recognition extraction and audio speech content recognition extraction on the video data sequence and the audio data sequence respectively to obtain a lip movement speech content set and an audio speech content set; According to the fused lip shape weight and the fused audio weight, perform weighted fusion processing on the lip movement speech content set and the audio speech content set to obtain speech data, where the weighted fusion processing includes performing weight identification on each speech content and screening the speech content with the largest sum of total weights as the speech data.

[0121] It should be noted that the above sequence of embodiments of the present application is only for description and does not represent the superiority or inferiority of the embodiments. And the above describes specific embodiments of this specification. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the protection scope of the present application.

[0123] This specification and the drawings are only exemplary descriptions of the present application and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalent technologies, the present application is intended to include these changes and modifications.

Claims

1. A method for extracting voice data using artificial intelligence, characterized in that, The method includes: Monitoring and collecting video data and audio data generated by video communication to obtain a video data sequence and an audio data sequence; According to the audio data sequence, performing speech rate recognition and volume recognition to obtain speech rate characteristic parameters and volume characteristic parameters, and respectively performing mouth shape weight and audio weight configuration to obtain a first mouth shape weight, a first audio weight, a second mouth shape weight, and a second audio weight; Performing mouth shape image ratio analysis on the video data sequence, processing to obtain a third mouth shape weight and a third audio weight, and performing vocabulary analysis according to the historical speech data extraction record of the target user, and processing to obtain a fourth mouth shape weight and a fourth audio weight; According to the first mouth shape weight, the first audio weight, the second mouth shape weight, the second audio weight, the third mouth shape weight, the third audio weight, the fourth mouth shape weight, and the fourth audio weight, processing to obtain a fused mouth shape weight and a fused audio weight, and performing speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data.

2. The voice data extraction method using artificial intelligence according to claim 1, characterized in that, According to the audio data sequence, performing speech rate recognition and volume recognition to obtain speech rate characteristic parameters and volume characteristic parameters, and respectively performing mouth shape weight and audio weight configuration to obtain a first mouth shape weight, a first audio weight, a second mouth shape weight, and a second audio weight, including: Inputting the audio data sequence into a speech rate recognizer to recognize and obtain speech rate characteristic parameters; Extracting the average volume in the audio data sequence as the volume characteristic parameter; According to the speech rate characteristic parameters, performing mouth shape weight and audio weight configuration to obtain a first mouth shape weight and a first audio weight, and according to the volume characteristic parameters, performing mouth shape weight and audio weight configuration to obtain a second mouth shape weight and a second audio weight.

3. The voice data extraction method using artificial intelligence according to claim 2, characterized in that, The construction steps of the speech rate recognizer include: Using machine learning to construct the network structure of the speech rate recognizer, where the input data of the speech rate recognizer is the audio data sequence and the output data is the speech rate characteristic parameters; According to the speech data recognition record within the historical time, collecting a set of sample audio data sequences and annotating the speech rates of different sample audio data sequences to obtain a set of sample speech rate characteristic parameters; Using the set of sample audio data sequences and the set of sample speech rate characteristic parameters to perform supervised training and testing on the speech rate recognizer until convergence.

4. The voice data extraction method using artificial intelligence according to claim 2, characterized in that According to the speech rate characteristic parameters, performing mouth shape weight and audio weight configuration to obtain a first mouth shape weight and a first audio weight, and according to the volume characteristic parameters, performing mouth shape weight and audio weight configuration to obtain a second mouth shape weight and a second audio weight, including: According to the speech data recognition record within the historical time, obtaining the average speech rate characteristic parameter and the average volume characteristic parameter; Calculating the ratio of the average speech rate characteristic parameter to the speech rate characteristic parameter, performing correction calculation on the preset audio weight to obtain the first audio weight, and calculating the first mouth shape weight according to the first audio weight; Calculating the ratio of the volume characteristic parameter to the average volume characteristic parameter, performing correction calculation on the preset audio weight to obtain the second audio weight, and calculating the second mouth shape weight according to the second audio weight.

5. The voice data extraction method using artificial intelligence according to claim 1, wherein Perform lip image ratio analysis on the video data sequence, and process to obtain the third lip weight and the third audio weight, including: Input each video data in the video data sequence into a lip ratio recognizer to identify and obtain a plurality of lip image ratio coefficients. Among them, the lip ratio recognizer is constructed using a convolutional neural network and is trained using a sample video data set and a labeled sample lip image ratio coefficient set; Obtain the average lip image ratio coefficient; Calculate the ratio of the mean of the plurality of lip image ratio coefficients to the average lip image ratio coefficient, and perform a correction calculation on the preset lip weight to obtain the third lip weight; Calculate the third audio weight according to the third lip weight.

6. The method for extracting voice data using artificial intelligence according to claim 1, characterized in that, Extract and record the historical voice data of the target user, and perform vocabulary analysis to process and obtain the fourth lip weight and the fourth audio weight, including: Collect the historical extracted voice data set of the user's voice data extraction during the historical time; Count the number of occurrences of rare words in the historical extracted voice data set to obtain the number of occurrences of rare words; Collect the number of occurrences of rare words of multiple sample users and calculate the mean to obtain the average number of occurrences of rare words; Perform a correction calculation on the preset lip weight according to the ratio of the average number of occurrences of rare words to the number of occurrences of rare words to obtain the fourth lip weight; Calculate the fourth audio weight according to the fourth lip weight.

7. The voice data extraction method using artificial intelligence according to claim 1, characterized in that, According to the first lip weight, the first audio weight, the second lip weight, the second audio weight, the third lip weight, the third audio weight, the fourth lip weight, and the fourth audio weight, process to obtain the fused lip weight and the fused audio weight, and perform speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data, including: Fuse and calculate the fused lip weight and the fused audio weight according to the first lip weight, the first audio weight, the second lip weight, the second audio weight, the third lip weight, the third audio weight, the fourth lip weight, and the fourth audio weight; Perform lip speech content recognition and extraction and audio speech content recognition and extraction on the video data sequence and the audio data sequence respectively to obtain a lip speech content set and an audio speech content set; According to the fused lip weight and the fused audio weight, perform weighted fusion processing on the lip speech content set and the audio speech content set to obtain speech data. Among them, the weighted fusion processing includes marking the weight of each speech content and screening the speech content with the largest sum of total weights as the speech data.

8. The voice data extraction method using artificial intelligence according to claim 7, characterized in that, Perform lip speech content recognition and extraction and audio speech content recognition and extraction on the video data sequence and the audio data sequence respectively to obtain a lip speech content set and an audio speech content set, including: Input the video data sequence into a plurality of lip speech content recognizers respectively, and recognize and output to obtain a lip speech content set. Among them, the input data of each lip speech content recognizer is the video data sequence, the output data is the lip speech content, and the training data of each lip speech content recognizer is different; Input the audio data sequence into multiple audio speech content recognizers respectively, and obtain an audio speech content set through recognition output. Among them, the input data of each audio speech content recognizer is the audio data sequence, the output data is the audio speech content, and the training data of each audio speech content recognizer is different.

9. A voice data extraction system using artificial intelligence, characterized in that, The system is used to execute the method described in any one of claims 1-8. The system includes: A data acquisition module, which is used to monitor and acquire video data and audio data generated by video communication, and obtain a video data sequence and an audio data sequence; An audio feature analysis and weight configuration module, which is used to perform speech rate recognition and volume recognition according to the audio data sequence, obtain speech rate feature parameters and volume feature parameters, and perform mouth shape weight and audio weight configuration respectively to obtain a first mouth shape weight, a first audio weight, a second mouth shape weight, and a second audio weight; A video and vocabulary feature analysis and weight configuration module, which is used to perform an analysis of the proportion of mouth shape images in the video data sequence, process to obtain a third mouth shape weight and a third audio weight, and perform vocabulary analysis according to the historical speech data extraction record of the target user, and process to obtain a fourth mouth shape weight and a fourth audio weight; A weight fusion and speech recognition module, which is used to process and obtain a fused mouth shape weight and a fused audio weight according to the first mouth shape weight, the first audio weight, the second mouth shape weight, the second audio weight, the third mouth shape weight, the third audio weight, the fourth mouth shape weight, and the fourth audio weight, and perform speech content recognition and extraction on the video data sequence and the audio data sequence to obtain speech data.

Images