Data processing methods, apparatus, computer equipment and storage media

By performing refined word segmentation on audio data, facial expression data, and emotion data, target sample data is generated and a large language model is trained. This solves the problem that the model cannot identify the root cause of inaccurate responses and improves the model's response accuracy and fluency.

CN123117310APending Publication Date: 2026-11-17BEIJING & AIWEIWU TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511604098.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing supervised fine-tuning large language models cannot effectively identify and correct the specific root causes of inaccurate responses when generating instructional responses, resulting in the system's inability to locate and improve the failure points.

Method used

By acquiring initial samples of audio data, facial expression data, and emotion data, we perform refined word segmentation to generate target sample data for training a large language model. We then use the target sample data to train the model and optimize its response capabilities.

Benefits of technology

It significantly improves the model's ability to process audio data and the accuracy of its response, enhances the model's accuracy and fluency in the inference and response phases, and strengthens the model's overall data processing performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117310A_ABST
    Figure CN123117310A_ABST
Patent Text Reader

Abstract

This application relates to a data processing method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring initial sample data; the initial sample data includes audio data, intermediate inference data, and response data to the audio data; splitting the text data according to the frequency of occurrence of adjacent characters in the text data to obtain first word segmentation units; using each expression data and each emotion data as a second word segmentation unit; generating target sample data based on the initial sample data, the first word segmentation units, and the second word segmentation units; training a preset large language model using the target sample data to obtain a target response model; the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data. This method can effectively improve the model's ability to process multimodal data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of artificial intelligence technology in the education field, a teaching application solution based on supervised fine-tuning (SFT) of large language models has emerged. This solution trains the model by collecting dialogue data with a "student question-teacher answer" structure, enabling it to directly generate corresponding teaching responses based on student input. In traditional technical processes, the goal of model training is to learn the end-to-end mapping from question to standard answer, and its core optimization is to ensure that the model's final output is consistent with the labeled answer in terms of text. However, the current supervised fine-tuning approach has a major problem: when the model's generated response fails to accurately address the student's question, because the entire reasoning process is compressed into an unparseable whole, the system cannot pinpoint the specific root cause of the inaccurate response, making it difficult to effectively identify and correct the failed steps. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0004] Firstly, this application provides a data processing method. The method includes:

[0005] Acquire initial sample data; wherein, the initial sample data includes audio data, intermediate inference data, and response data for the audio data; the intermediate inference data includes intermediate data generated based on the audio data to guide the generation of the response data; the response data includes text data, facial expression data, and emotion data;

[0006] Based on the frequency of occurrence of each adjacent character in the text data, the text data is split to obtain a first word segmentation unit; each of the facial expression data and each of the emotion data are respectively used as a second word segmentation unit;

[0007] Based on the initial sample data, the first word segmentation unit, and the second word segmentation unit, target sample data is generated;

[0008] Using the target sample data, a preset large language model is trained to obtain a target response model; wherein, the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data.

[0009] In one embodiment, the step of splitting the text data according to the frequency of occurrence of each adjacent character in the text data to obtain a first word segmentation unit includes:

[0010] The text data is segmented to obtain the character data corresponding to the text data;

[0011] Calculate the frequency of occurrence of each adjacent character in the character data;

[0012] Based on the frequency of occurrence of each adjacent character, iteratively merge adjacent characters whose frequency of occurrence is greater than a preset threshold, and update the frequency statistics; until the frequency of occurrence of each adjacent character is less than the preset threshold, the first word segmentation unit is obtained.

[0013] In one embodiment, generating target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit includes:

[0014] Integrate the first word segmentation unit and the second word segmentation unit to construct a word segmentation vocabulary;

[0015] Based on the word segmentation vocabulary, the initial sample data is segmented and encoded to obtain the target sample data.

[0016] In one embodiment, the step of segmenting and encoding the initial sample data according to the segmentation lexicon to obtain the target sample data includes:

[0017] The audio data is identified to obtain the corresponding audio text data;

[0018] Based on the word segmentation vocabulary, the audio text data, intermediate inference data, and response data are mapped and encoded to obtain the corresponding data identification data;

[0019] Target sample data is generated based on the identification data.

[0020] In one embodiment, after obtaining the initial sample data, the method further includes:

[0021] The audio data in the initial sample data is subjected to fuzzy sound replacement to obtain adversarial audio data;

[0022] Based on the pronunciation ambiguity type corresponding to the adversarial audio data, ambiguity sound inference data is generated; and based on the ambiguity sound inference data and the initial sample data corresponding to the adversarial audio data, intermediate inference data and response data of the adversarial audio data are generated; the ambiguity sound inference data is used to guide the elimination of ambiguity sounds.

[0023] In one embodiment, training a preset large language model using the target sample data includes:

[0024] The target sample data is input into the large language model to obtain the prediction output.

[0025] The difference between the predicted output and the response data is calculated using the cross-entropy loss function to obtain the loss value;

[0026] Based on the loss value, the backpropagation algorithm is executed to determine the gradient of the model parameters;

[0027] Update the low-rank adaptation matrix parameters in the large language model according to the gradient;

[0028] The model prediction, loss calculation, backpropagation, and parameter update are performed iteratively until the training loss converges to a preset threshold, thus obtaining the target response model.

[0029] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0030] A data acquisition module is used to acquire initial sample data; wherein, the initial sample data includes audio data, intermediate inference data, and response data for the audio data; the intermediate inference data includes intermediate data generated based on the audio data, used to guide the generation of the response data; the response data includes text data, facial expression data, and emotion data;

[0031] The data segmentation module is used to split the text data according to the frequency of occurrence of each adjacent character in the text data to obtain a first segmentation unit; and to use each of the facial expression data and each of the emotion data as a second segmentation unit.

[0032] The data generation module is used to generate target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit;

[0033] The model training module is used to train a preset large language model using the target sample data to obtain a target response model; wherein, the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing method as described in any one of the embodiments of this disclosure.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the data processing method as described in any one of the embodiments of this disclosure.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any of the embodiments of this disclosure.

[0037] The aforementioned data processing methods, apparatus, computer equipment, storage media, and computer program products effectively improve the model's ability to process audio data and the accuracy of its responses by acquiring initial samples containing audio data, intermediate inference data, and response data, and performing refined word segmentation on the text, facial expressions, and emotion data in the response data to generate target sample data for training a large language model. This method not only optimizes the data sample construction process, making the sample data more comprehensive and accurate, but also significantly enhances the target response model's ability to generate high-quality intermediate inference data and response data after receiving audio data. Specifically, through detailed word segmentation of text, facial expressions, and emotion data, the model can more accurately understand and parse audio content, thus exhibiting higher accuracy and fluency in the inference and response stages, greatly improving the overall data processing and model response performance. Attached Figure Description

[0038] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0039] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0040] Figure 3 This is a structural block diagram of a data processing device in one embodiment;

[0041] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] The data processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. In this target response model training and application scheme, terminal 102 and server 104 collaborate through two core interactions. In the first interaction, terminal 102 acquires initial sample data (including audio data, intermediate inference data, and text, facial expression, and emotion-related response data) and uploads this data to server 104. After receiving the data, server 104 splits the text according to the frequency of adjacent characters to obtain the first word segmentation unit, uses facial expression and emotion data as the second word segmentation unit, combines the initial samples to generate target sample data, and then uses this data to train a preset large language model to obtain a target response model capable of processing audio-generated responses. In the second interaction, server 104 distributes the trained target response model to terminal 102. Terminal 102 can then use this model to receive new audio data, automatically generate intermediate inference data, and generate corresponding response data based on this, achieving a closed loop from sample acquisition to model application. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0044] In one embodiment, such as Figure 2 As shown, a data processing method is provided, including the following steps:

[0045] Step S200: Obtain initial sample data; wherein the initial sample data includes audio data, intermediate inference data, and response data for the audio data; the intermediate inference data includes intermediate data generated based on the audio data to guide the generation of the response data; the response data includes text data, facial expression data, and emotion data.

[0046] In acquiring initial sample data, various methods can be used for collection. For example, in educational settings, audio recordings of real conversations between teachers and students can be used as audio data. Simultaneously, data on the teacher's intermediate reasoning process when answering student questions, as well as students' textual feedback, facial expressions, and emotional reactions after hearing the answers, can be recorded as response data. This data comprehensively reflects the information flow and emotional exchange in teaching interactions, providing rich material for subsequent model training.

[0047] In one exemplary embodiment, simulated initial sample data can also be generated through a simulated dialogue system. The simulated dialogue system can automatically generate dialogue content of student questions and teacher answers based on a preset teaching scenario and question bank, and simulate the corresponding intermediate reasoning process and response data. This method can quickly generate a large amount of sample data for the initial training and validation of the model.

[0048] Step S202: Based on the frequency of occurrence of each adjacent character in the text data, the text data is split to obtain a first word segmentation unit; each of the facial expression data and each of the emotion data are respectively used as a second word segmentation unit.

[0049] When splitting text data, a statistical language model can be used to calculate the frequency of occurrence of adjacent characters. Specifically, the text data is first preprocessed, such as removing punctuation and standardizing capitalization. Then, the occurrence frequency of all adjacent character pairs in the text is counted, and their frequency is calculated accordingly. Characters with high frequency of occurrence can be merged to reduce the number of characters and obtain a more reasonable first segmentation unit. For facial expression and emotion data, they can be converted into text data, such as "<smile><happiness>". Independent facial expression or emotion data can be treated as a single word segment, and all facial expression and emotion data can be summarized to form a second segmentation unit.

[0050] In one exemplary embodiment, deep learning-based word segmentation methods can also be introduced for splitting text data. For example, recurrent neural networks or their variants, long short-term memory networks, can be used to model the text sequence. By training the network to learn character-level contextual information, meaningful word units can be identified more accurately. This method is particularly suitable for processing text data containing complex semantics and grammatical structures, and can further improve the accuracy and rationality of word segmentation.

[0051] Step S204: Generate target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit.

[0052] In generating target sample data, the first and second segmentation units can be integrated to construct a segmentation vocabulary containing all segmentation information. This vocabulary serves as the basis for subsequent data encoding, ensuring that each segment has a unique identifier. Simultaneously, the audio text data, intermediate inference data, and response data in the initial sample data can be mapped and encoded based on the vocabulary. Specifically, after converting the audio data into corresponding audio text data, it can be segmented together with the intermediate inference data and response data, and each segment can be replaced with its corresponding identifier data according to the vocabulary. Finally, these identifier data are combined in a specific format to form structured target sample data, providing high-quality input for subsequent model training.

[0053] In one exemplary embodiment, data augmentation techniques can also be introduced during the mapping and encoding process. For example, synonym replacement and sentence transformation can be performed on audio text data to generate new samples that are semantically similar to the original data but express different ideas, thereby increasing the diversity and richness of the data. Simultaneously, intermediate inference and response data can be appropriately perturbed to simulate noise and uncertainty that may exist in real-world scenarios, improving the model's adaptability to complex environments. These data augmentation methods help improve the model's generalization performance, enabling it to exhibit good performance even when faced with unseen data. Furthermore, after generating the target sample data, it can be quality-assessed and screened to remove samples with obvious errors or unreasonable information, ensuring that the data used for model training has high accuracy and reliability.

[0054] Step S206: Using the target sample data, train a preset large language model to obtain a target response model; wherein, the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data.

[0055] When training a pre-defined large language model using target sample data, various optimization algorithms can be employed to accelerate the model's convergence and improve training efficiency. For example, an adaptive moment estimation optimizer can be used, which adaptively adjusts the learning rate based on the historical gradient information of each parameter, thereby updating model parameters more effectively during training. Furthermore, an early stopping mechanism can be introduced, terminating training prematurely when the model's performance on the validation set fails to improve for several consecutive epochs (batches) to prevent overfitting. During training, training strategies can be dynamically adjusted by monitoring metrics such as the model's loss and accuracy on the validation set, such as adjusting the learning rate or increasing / decreasing training epochs, to achieve better training results. Ultimately, the fully trained target response model will possess powerful audio understanding and response generation capabilities, providing users with accurate, natural, and emotionally resonant interactive experiences in practical applications.

[0056] In one exemplary embodiment, a multi-task learning strategy can be introduced during training. Specifically, in addition to the main task of training the model to generate intermediate inference and response data, auxiliary tasks such as sentiment classification and intent recognition can be designed. These auxiliary tasks can help the model better capture emotional and intent information in audio data, thereby improving the accuracy and relevance of the model when generating response data. For example, in the sentiment classification task, the model needs to determine the emotional tendency (such as positive, negative, or neutral) in the audio data and incorporate this information into the generated response data, making the response more consistent with the user's emotional state. Through multi-task learning, the model can learn multiple related tasks simultaneously, thereby improving overall performance and generalization ability. Furthermore, after training, the target response model can be further optimized and adjusted. For example, user feedback mechanisms can be used to collect user satisfaction ratings for the model's generated responses, and the model can be fine-tuned based on these ratings to continuously improve the user experience. Simultaneously, the model's training data can be updated periodically, introducing new samples and scenarios, enabling the model to adapt to constantly changing application environments and user needs.

[0057] The aforementioned data processing method acquires initial samples containing audio data, intermediate inference data, and response data. It then performs refined word segmentation on the text, facial expressions, and emotion data within the response data to generate target sample data for training the large language model. This effectively improves the model's ability to process audio data and the accuracy of its responses. This method not only optimizes the data sample construction process, making the sample data more comprehensive and accurate, but also significantly enhances the target response model's ability to generate high-quality intermediate inference and response data after receiving audio data. Specifically, through detailed word segmentation of text, facial expressions, and emotion data, the model can more accurately understand and parse audio content, thus exhibiting higher accuracy and fluency in the inference and response stages, greatly improving the overall data processing and model response performance.

[0058] In one embodiment, the step of splitting the text data according to the frequency of occurrence of each character in the text data to obtain a first word segmentation unit includes:

[0059] The text data is segmented to obtain the corresponding character data.

[0060] Statistically analyze the frequency of occurrence of each adjacent character in the character data.

[0061] Based on the frequency of occurrence of each adjacent character, iteratively merge adjacent characters whose frequency of occurrence is greater than a preset threshold, and update the frequency statistics; until the frequency of occurrence of each adjacent character is less than the preset threshold, the first word segmentation unit is obtained.

[0062] During word segmentation, preprocessing of the text data can be performed to ensure accuracy and consistency. Preprocessing steps may include removing special characters, standardizing character encoding (such as UTF-8), and converting case, thus providing a clean and standardized data foundation for subsequent word segmentation statistics. In counting the frequency of adjacent characters, a sliding window approach can be used to traverse the entire text data, recording the occurrence count of adjacent character pairs within each window and calculating their frequency. This process helps capture local linguistic features in the text, providing a strong basis for word segmentation.

[0063] In one exemplary embodiment, when iteratively merging adjacent characters whose frequency exceeds a preset threshold, the frequency statistics can be dynamically updated to ensure that the occurrence of remaining character pairs is accurately reflected after each merge. By continuously adjusting the preset threshold, the granularity of the segmentation units can be controlled, thereby meeting the needs of different application scenarios. Ultimately, when the frequency of each adjacent character is less than the preset threshold, it indicates that a relatively ideal segmentation state has been achieved, and the first segmentation unit obtained at this time will have high accuracy and usability. Natural language processing techniques, such as deep learning-based segmentation models, can also be introduced. These models can capture complex linguistic phenomena in text by learning from a large amount of labeled data, thus exhibiting stronger robustness and adaptability in the segmentation process. By combining statistical methods with deep learning techniques, a more efficient and accurate segmentation system can be constructed, providing strong support for subsequent data processing and model training.

[0064] In this embodiment, by performing detailed word segmentation on the text data and iteratively merging adjacent characters based on their frequency of occurrence, semantic units in the text can be captured more accurately, thereby improving the accuracy and effectiveness of subsequent model training. This method splits the text data into basic character data and counts the frequency of occurrence of each adjacent character in this data. Based on this frequency information, adjacent characters with a frequency higher than a preset threshold are iteratively merged, and the frequency statistics are updated. This process continues until the frequency of all adjacent characters is lower than the preset threshold, resulting in the first word segmentation unit. These word segmentation units not only retain the original semantic information of the text data but also provide richer and more accurate input data for subsequent target response model training.

[0065] In one embodiment, generating target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit includes:

[0066] Integrate the first word segmentation unit and the second word segmentation unit to construct a word segmentation vocabulary.

[0067] Based on the word segmentation vocabulary, the initial sample data is segmented and encoded to obtain the target sample data.

[0068] In constructing the word segmentation vocabulary, it's ensured that the vocabulary comprehensively covers all words in the first and second word segmentation units, while avoiding duplication and redundancy. Specifically, the first and second word segmentation units are first merged, duplicate segmentation items are removed, and then arranged in a certain order (such as alphabetical order or frequency of occurrence) to form a complete word segmentation vocabulary. This vocabulary provides a unified lexical foundation for subsequent data encoding, ensuring that each segment has a unique identifier.

[0069] In one exemplary embodiment, when segmenting and encoding the initial sample data according to a word segmentation lexicon, a one-to-one mapping method can be used, replacing each word in the initial sample data with its corresponding identifier in the word segmentation lexicon. For example, for each first word segmentation unit in the text data, and each second word segmentation unit in the facial expression data and emotion data, its corresponding identifier can be found in the word segmentation lexicon, and these identifiers are combined according to a certain format to form structured target sample data. This encoding method helps to convert the original text data, facial expression data, and emotion data into a numerical form that can be processed by a computer, providing high-quality input data for subsequent model training.

[0070] In one exemplary embodiment, additional processing steps can be introduced during word segmentation encoding. For example, the word segmentation vocabulary can be dynamically updated, continuously adding new segmentation terms or deleting infrequently used terms as needed for model training. Simultaneously, more detailed preprocessing can be performed on the initial sample data, such as removing noise and correcting spelling errors, to improve data accuracy and reliability. Furthermore, data encoding techniques, such as word embedding or contextual encoding, can be introduced to better capture the semantic relationships and contextual information between words, thereby enhancing the expressive power of the target sample data and the effectiveness of model training.

[0071] In this embodiment, by acquiring various types of information from the initial sample data and accurately segmenting the text based on the frequency of adjacent characters, the accuracy and rationality of word segmentation can be effectively improved. Facial expression data and emotion data are used as second word segmentation units, combined with the first word segmentation unit to generate target sample data, providing a rich and targeted data foundation for the subsequent training of the large language model. Using this target sample data to train the preset large language model, the resulting target response model can more efficiently and accurately receive audio data, sequentially generating intermediate inference data and response data, thus better meeting the needs of data processing and response generation in practical applications.

[0072] In one embodiment, the step of segmenting and encoding the initial sample data according to the segmentation lexicon to obtain the target sample data includes:

[0073] The audio data is identified to obtain the corresponding audio text data.

[0074] Based on the word segmentation vocabulary, the audio text data, intermediate inference data, and response data are mapped and encoded to obtain the corresponding data identification data.

[0075] Target sample data is generated based on the identification data.

[0076] When obtaining audio text data through audio data recognition, speech recognition technology can be used, such as end-to-end speech recognition models based on deep learning. These models can directly convert audio signals into text sequences, effectively reducing errors that may be introduced in traditional speech recognition processes. Simultaneously, preprocessing of the audio data before recognition, such as noise reduction and gain control, can improve audio quality.

[0077] In one exemplary embodiment, during the process of mapping and encoding audio text data, intermediate inference data, and response data into identifier data, attention can be paid to ensuring the completeness and accuracy of the word segmentation vocabulary. The word segmentation vocabulary can include all possible word segments and their unique identifiers, so that the corresponding identifiers can be accurately found during encoding. Furthermore, the encoding process can follow certain rules and formats to ensure that the resulting target sample data has consistency and readability. After generating the target sample data, it can also be subjected to quality checks and verification. Manual sampling or automatic verification can be used to check for encoding errors, omissions, or inconsistencies in the target sample data.

[0078] In this embodiment, by combining techniques such as audio data recognition, word segmentation, and vocabulary mapping encoding, efficient processing and transformation of the initial sample data are achieved. The generated target sample data not only contains all the key information from the original data but is also converted into a numerical form that can be processed by a computer through encoding, providing strong data support for the subsequent training of the large language model. Simultaneously, quality checks and verification of the target sample data further ensure the accuracy and reliability of the data, laying a solid foundation for the model's training effectiveness.

[0079] In one embodiment, after obtaining the initial sample data, the method further includes:

[0080] The audio data in the initial sample data is subjected to fuzzy sound replacement to obtain adversarial audio data.

[0081] Based on the pronunciation ambiguity type corresponding to the adversarial audio data, ambiguity sound inference data is generated; and based on the ambiguity sound inference data and the initial sample data corresponding to the adversarial audio data, intermediate inference data and response data of the adversarial audio data are generated; the ambiguity sound inference data is used to guide the elimination of ambiguity sounds.

[0082] When performing ambiguous sound replacement on audio data, common pronunciation ambiguity scenarios can be predefined, such as including the confusion between "z / c / s" and "zh / ch / sh" in the replacement rule base. After generating adversarial audio data by simulating potential ambiguity in actual user pronunciation, the type of pronunciation ambiguity (e.g., confusion between retroflex and alveolar consonants, confusion between front and back nasal consonants) must be recorded simultaneously. During the generation of ambiguity sound inference data, acoustic and language models can be combined to analyze the pronunciation characteristics of ambiguities in the adversarial audio data and infer their possible clear pronunciation versions. For example, when confusion between "si" and "shi" is detected, the ambiguity sound inference data must explicitly label the set of candidate clear pronunciations corresponding to the ambiguity sound. When generating intermediate inference data based on the ambiguity sound inference data and initial sample data, a mapping relationship from ambiguity pronunciation to standard pronunciation must be constructed, and this mapping information must be embedded in the intermediate inference data structure. The generation of response data must consider the ambiguity sound type and contextual information to ensure that the generated text response accurately reflects the user's intent while naturally eliminating ambiguity caused by pronunciation ambiguity. It is also possible to construct test sets containing various types of pronunciation ambiguity to evaluate the accuracy and naturalness of the target response model under ambiguous audio input.

[0083] In this embodiment, by performing fuzzy sound replacement processing on the initial sample data, the model's adaptability to pronunciation ambiguity can be enhanced. Potentially fuzzy pronunciation segments in the audio data are identified, and these segments are replaced with other syllables or words with similar pronunciations but different semantics, thereby generating adversarial audio data. Based on the pronunciation ambiguity type of the adversarial audio data, fuzzy sound inference data is further generated. This data can simulate a reasonable intermediate inference process and corresponding response data under pronunciation ambiguity. Utilizing this adversarial audio data and its corresponding intermediate inference and response data enriches the diversity of the initial sample data and improves the accuracy and robustness of the target response model when processing fuzzy pronunciation audio.

[0084] In one embodiment, training a preset large language model using the target sample data includes:

[0085] The target sample data is input into the large language model to obtain the prediction output.

[0086] The difference between the predicted output and the response data is calculated using the cross-entropy loss function to obtain the loss value.

[0087] Based on the loss value, the backpropagation algorithm is executed to determine the gradient of the model parameters.

[0088] Update the low-rank adaptation matrix parameters in the large language model based on the gradient.

[0089] The model prediction, loss calculation, backpropagation, and parameter update are performed iteratively until the training loss converges to a preset threshold, thus obtaining the target response model.

[0090] In this process, when inputting target sample data into a large language model for prediction, the format of the input data can be matched with the model requirements, and the predicted output can serve as the model's response to the current input. The cross-entropy loss function quantifies the difference between the predicted output and the actual response data; this difference is expressed as a loss value, providing a clear direction for model optimization.

[0091] In one exemplary embodiment, when executing the backpropagation algorithm, the gradients of the model parameters can be calculated layer by layer, starting from the output layer. During this process, the gradient reflects the degree to which parameter adjustments affect the change in loss value and is a key basis for optimizing the model. Based on the calculated gradients, the parameters of the low-rank adaptation matrix in the large language model can be updated, adjusting the internal parameters of the model. In each iteration, the model can fine-tune its parameters based on the previous loss value and gradient information, gradually approaching the optimal solution. When the training loss converges to a preset threshold, it indicates that the model has achieved good predictive performance, and the resulting model can be considered the target response model.

[0092] In one exemplary embodiment, after obtaining the first word segmentation unit and the second word segmentation unit, the method may further include:

[0093] Word embedding transformation: Through the word embedding layer, the first and second word segmentation units are mapped to high-dimensional semantic vectors (batch size, sequence length, feature dimension).

[0094] Rotation position encoding: The complex frequency tensor is pre-computed and the queries and keys of the attention layer are rotated to inject relative position information.

[0095] Root mean square normalization: Perform root mean square normalization on the input tensor, calculate the normalization result according to the formula and multiply it by the learnable weights to stabilize the training gradient.

[0096] Grouped Query Attention Calculation: Query, key, and value are generated through query projection, key projection, and value projection layers. The keys and values ​​are expanded to match the dimensions of the query, attention scores are calculated, and a weighted sum is performed.

[0097] Attention residual connection: The output of the grouped query attention is added to the input of the current layer to preserve the original information and alleviate the gradient vanishing problem.

[0098] Feedforward neural network computation: The hidden layer tensor is generated by the first and third feedforward networks, activated by the SwiGLU activation function, and then the dimension is restored by the second feedforward network.

[0099] Feedforward residual connection: The output of the feedforward neural network is added to the result of the previous root mean square normalization, thereby completing the computation of a single Transformer module.

[0100] Model pre-training: Input general text data, generate logistic values ​​through forward propagation, calculate the loss using the cross-entropy loss function, and update all model parameters using the AdamW optimizer.

[0101] Low-rank adaptive fine-tuning: Freeze the pre-trained model weights and add low-rank adaptive branches only next to the query projection and value projection layers. Train only these low-rank matrix parameters to adapt to the task in the vertical domain.

[0102] Inference key-value caching: During text generation, the key and value tensors of all historical time steps are cached, and only the query tensor of the current term is calculated, thereby improving inference efficiency.

[0103] Dynamic generation strategy: Kernel sampling and temperature coefficient are used to adjust the selection probability of lexical units, and a repetition penalty mechanism is combined to reduce the repetition rate of the generated text.

[0104] Output: Generate text autoregressively until the maximum generated length is reached or a terminator is encountered, and finally decode the word sequence to obtain the final text.

[0105] Model file saving: Upload the model files generated by training with the PyTorch framework to the cloud server for storage.

[0106] In this embodiment, by acquiring initial sample data containing multimodal information and performing refined word segmentation, data features can be captured more comprehensively. Training a large language model using the segmented target sample data effectively improves the model's ability to understand audio data and the quality of its response generation. By introducing fuzzy sound replacement and adversarial training mechanisms, the model's robustness in scenarios with ambiguous pronunciation is enhanced, enabling it to generate intermediate inference data and response data more accurately. The resulting target response model, when receiving audio data, can efficiently generate context-appropriate intermediate inference data and produce accurate and rich responses based on this data, including multi-dimensional information such as text, facial expressions, and emotions.

[0107] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0108] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0109] In one embodiment, such as Figure 3 As shown, a data processing device 300 is provided, including: a data acquisition module 301, a data segmentation module 303, a data generation module 305, and a model training module 307, wherein:

[0110] A data acquisition module is used to acquire initial sample data; wherein, the initial sample data includes audio data, intermediate inference data, and response data for the audio data; the intermediate inference data includes intermediate data generated based on the audio data, used to guide the generation of the response data; the response data includes text data, facial expression data, and emotion data;

[0111] The data segmentation module is used to split the text data according to the frequency of occurrence of each adjacent character in the text data to obtain a first segmentation unit; and to use each of the facial expression data and each of the emotion data as a second segmentation unit.

[0112] The data generation module is used to generate target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit;

[0113] The model training module is used to train a preset large language model using the target sample data to obtain a target response model; wherein, the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data.

[0114] In one embodiment, the data segmentation module is further configured to:

[0115] The text data is segmented to obtain the character data corresponding to the text data;

[0116] Calculate the frequency of occurrence of each adjacent character in the character data;

[0117] Based on the frequency of occurrence of each adjacent character, iteratively merge adjacent characters whose frequency of occurrence is greater than a preset threshold, and update the frequency statistics; until the frequency of occurrence of each adjacent character is less than the preset threshold, the first word segmentation unit is obtained.

[0118] In one embodiment, the data generation module is further configured to:

[0119] Integrate the first word segmentation unit and the second word segmentation unit to construct a word segmentation vocabulary;

[0120] Based on the word segmentation vocabulary, the initial sample data is segmented and encoded to obtain the target sample data.

[0121] In one embodiment, the data generation module is further configured to:

[0122] The audio data is identified to obtain the corresponding audio text data;

[0123] Based on the word segmentation vocabulary, the audio text data, intermediate inference data, and response data are mapped and encoded to obtain the corresponding data identification data;

[0124] Target sample data is generated based on the identification data.

[0125] In one embodiment, the data acquisition module is further configured to:

[0126] The audio data in the initial sample data is subjected to fuzzy sound replacement to obtain adversarial audio data;

[0127] Based on the pronunciation ambiguity type corresponding to the adversarial audio data, ambiguity sound inference data is generated; and based on the ambiguity sound inference data and the initial sample data corresponding to the adversarial audio data, intermediate inference data and response data of the adversarial audio data are generated; the ambiguity sound inference data is used to guide the elimination of ambiguity sounds.

[0128] In one embodiment, the model training module is further configured to:

[0129] The target sample data is input into the large language model to obtain the prediction output.

[0130] The difference between the predicted output and the response data is calculated using the cross-entropy loss function to obtain the loss value;

[0131] Based on the loss value, the backpropagation algorithm is executed to determine the gradient of the model parameters;

[0132] Update the low-rank adaptation matrix parameters in the large language model according to the gradient;

[0133] The model prediction, loss calculation, backpropagation, and parameter update are performed iteratively until the training loss converges to a preset threshold, thus obtaining the target response model.

[0134] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0135] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0136] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0138] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized by, The method includes: Acquire initial sample data; wherein, the initial sample data includes audio data, intermediate inference data, and response data for the audio data; the intermediate inference data includes intermediate data generated based on the audio data to guide the generation of the response data; the response data includes text data, facial expression data, and emotion data; Based on the frequency of occurrence of each adjacent character in the text data, the text data is split to obtain a first word segmentation unit; each of the facial expression data and each of the emotion data are respectively used as a second word segmentation unit; Based on the initial sample data, the first word segmentation unit, and the second word segmentation unit, target sample data is generated; Using the target sample data, a preset large language model is trained to obtain a target response model; wherein, the target response model is used to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data.

2. The method of claim 1, wherein, The step of splitting the text data according to the frequency of occurrence of each adjacent character in the text data to obtain the first word segmentation unit includes: The text data is segmented to obtain the character data corresponding to the text data; Calculate the frequency of occurrence of each adjacent character in the character data; Based on the frequency of occurrence of each adjacent character, iteratively merge adjacent characters whose frequency of occurrence is greater than a preset threshold, and update the frequency statistics; until the frequency of occurrence of each adjacent character is less than the preset threshold, the first word segmentation unit is obtained.

3. The method of claim 1, wherein, The step of generating target sample data based on the initial sample data, the first word segmentation unit, and the second word segmentation unit includes: Integrate the first word segmentation unit and the second word segmentation unit to construct a word segmentation vocabulary; Based on the word segmentation vocabulary, the initial sample data is segmented and encoded to obtain the target sample data.

4. The method of claim 3, wherein, The step of segmenting and encoding the initial sample data according to the segmentation vocabulary to obtain the target sample data includes: The audio data is identified to obtain the corresponding audio text data; Based on the word segmentation vocabulary, the audio text data, intermediate inference data, and response data are mapped and encoded to obtain the corresponding data identification data; Target sample data is generated based on the identification data.

5. The method of claim 1, wherein, After obtaining the initial sample data, the process also includes: The audio data in the initial sample data is subjected to fuzzy sound replacement to obtain adversarial audio data; Based on the pronunciation ambiguity type corresponding to the adversarial audio data, ambiguity sound inference data is generated; and based on the ambiguity sound inference data and the initial sample data corresponding to the adversarial audio data, intermediate inference data and response data of the adversarial audio data are generated; the ambiguity sound inference data is used to guide the elimination of ambiguity sounds.

6. The method of claim 1, wherein, The step of training a pre-defined large language model using the target sample data includes: The target sample data is input into the large language model to obtain the prediction output. The difference between the predicted output and the response data is calculated using the cross-entropy loss function to obtain the loss value; Based on the loss value, the backpropagation algorithm is executed to determine the gradient of the model parameters; Update the low-rank adaptation matrix parameters in the large language model according to the gradient; The model prediction, loss calculation, back propagation and parameter updating are iteratively performed until the training loss converges to a preset threshold, and the target response model is obtained.

7. A data processing apparatus, characterized by The device comprises: The data acquisition module is configured to acquire initial sample data, wherein the initial sample data comprises audio data, intermediate inference data, and response data corresponding to the audio data; the intermediate inference data comprises intermediate data generated based on the audio data and used to guide generation of the response data; and the response data comprises text data, expression data, and emotion data. The data tokenization module is configured to split the text data according to the frequency of occurrence of each adjacent character in the text data to obtain a first tokenization unit; and each expression data and each emotion data is taken as a second tokenization unit. The data generation module is configured to generate target sample data based on the initial sample data, the first tokenization unit, and the second tokenization unit. The model training module is configured to train a preset large language model based on the target sample data to obtain a target response model; wherein the target response model is configured to receive audio data, generate intermediate inference data, and generate response data based on the intermediate inference data. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.