Guangdong-Chinese-English hybrid recognition technology based on large model

Abstract

The invention relates to the technical field of speech recognition, and particularly discloses a Guangdong Chinese-English hybrid recognition technology based on a large model. And the user binds the equipment through the applet, creates dialogue configuration and call-out configuration, and pushes the configuration to the equipment side. And the equipment end makes a call to the customer according to the set time, plays a preset question and records voice answered by the customer. The voice recognition model converts mixed voice contents of Cantonese, Mandarin and English into texts and uploads the texts to the cloud. The large model carries out consistency evaluation on the question answering process and evaluation of customer answers, and a detailed analysis report is generated. The technology is widely applied to the fields of intelligent voice customer service, voice translation, multi-language communication and the like, and complex Guangdong, Chinese and English mixed voice input can be efficiently processed.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and specifically discloses a mixed Cantonese-Chinese-English speech recognition technology based on a large model. Background Technology

[0002] With the development of deep learning and artificial intelligence technologies, speech recognition and natural language processing technologies have made significant progress. These advancements have enabled machines to understand and respond to human language more accurately, laying the foundation for the development of intelligent voice systems. Currently, speech recognition technology on the market mainly focuses on single-language recognition, while mixed-language recognition remains a technical challenge, especially achieving efficient speech recognition in multilingual environments. Furthermore, users have increasingly higher demands for the real-time response and accuracy of intelligent voice systems, which places higher demands on data processing and computing power. To address these issues, there is an urgent need for a technical solution that can support multilingual recognition and combine large models for intelligent analysis. Summary of the Invention

[0003] This invention aims to develop a large-scale model-based Cantonese-Chinese-English mixed speech recognition technology to improve the accuracy of mixed-language speech recognition and the efficiency of intelligent analysis. Users bind their devices to an application or mini-program by scanning a QR code. Users can create dialogue nodes and configurations, and push them to the device. The device then retrieves the speech content of the dialogue nodes based on the received configurations, including speech in Cantonese, Mandarin, and English.

[0004] Users can also set up outbound call tasks and create outbound call configurations, including call time, phone number, and language preferences, and push these configurations to the device. Upon receiving the configuration, the device will automatically call the customer at the set time, play a preset question, and record the customer's voice response. The device utilizes built-in large model technology to convert the customer's voice content into text, with particular support for mixed language input and recognition.

[0005] After the Q&A session concludes, the device uploads the collected voice data to the cloud. The large-scale model then performs a consistency assessment of the questions and answers throughout the process, deeply analyzes the customer's responses, generates a detailed feedback report, and finally delivers the results to the customer. This invention utilizes advanced large-scale model technology, combined with mixed Cantonese-Chinese-English recognition capabilities, and is widely applicable in fields such as intelligent voice customer service, voice translation, and multilingual communication, providing an efficient and accurate solution.

[0006] This system not only has significant advantages in handling complex mixed speech input, but also performs excellently in intelligent analysis and feedback, effectively improving work efficiency and meeting the needs of multiple language environments.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a Cantonese-Chinese-English hybrid recognition technology based on a large model, the method comprising the following steps:

[0008] Step 1: Users bind their devices using the mini-program and create conversation and call-out configurations.

[0009] Users can scan a QR code through the mini-program to bind their devices and then directly create dialogue and call configurations within the mini-program.

[0010] Step 2: The cloud sends the dialogue configuration and outbound call configuration to the device. After receiving the data, the device schedules a timed call task.

[0011] After the user saves the conversation configuration and call configuration, the cloud will push them to the device, and the device will automatically create a scheduled call task upon receiving them.

[0012] Step 3: The device makes an outgoing call to the customer, engages in a question-and-answer session, and uses a model that supports mixed Cantonese, Chinese, and English speech recognition to convert the speech into text.

[0013] When the device reaches the scheduled time, it will automatically dial a preset phone number to initiate communication with the customer, interacting using pre-defined language and dialogue configurations. During the call, the device will input the collected voice content into a model that supports mixed Cantonese, Chinese, and English recognition for processing.

[0014] Step 4: After the question-and-answer session, use a large model to evaluate the consistency of the question-and-answer process.

[0015] After the user answers the last question, the call data will be uploaded to the cloud. The cloud will use a large model to evaluate the consistency of the questions and answers. If any inconsistencies are found, the customer will be asked to provide further explanation.

[0016] Step 5: Input the text of the question-and-answer process into the large model to generate a detailed analysis report.

[0017] After the user completes the supplementary instructions, the device will collect and upload data containing the user's supplementary information to the cloud. The cloud will then use advanced big data models to conduct in-depth analysis of this data and generate a detailed analysis report.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] (1) The speech recognition model of the present invention can efficiently process mixed speech inputs of Cantonese, Mandarin and English and recognize them as text.

[0020] (2) This invention trains the model using a carefully constructed mixed Cantonese-Chinese-English speech dataset, enabling the model to recognize mixed Cantonese-Chinese-English speech.

[0021] (3) This invention innovatively combines large model technology, which significantly improves the accuracy and robustness of multilingual recognition. By integrating the language features of Cantonese, Mandarin and English into the large model, the system can maintain efficient and accurate speech recognition and translation functions in multilingual interactive environments, and is particularly suitable for complex scenarios where multiple languages ​​are used in combination.

[0022] (4) By applying advanced large model technology and precisely designed prompt words, this system evaluates the consistency of questions and answers during the question-and-answer process and generates a detailed analysis report for the entire call process, thereby improving the intelligence level and response accuracy of the system. Attached Figure Description

[0023] Figure 1 This is the flowchart for this method.

[0024] Figure 2 Flowchart for large model analysis Detailed Implementation

[0025] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While these embodiments may be implemented in various specific ways, they should not be construed as limiting the invention to the embodiments described herein. Rather, these exemplary embodiments are provided to make the content of the invention more comprehensive and to facilitate a full understanding of the concept and application of the invention by those skilled in the art.

[0026] Subject to the inventive concept, the structure, performance, effect or other features of a particular embodiment can be flexibly combined with one or more other embodiments to adapt to different application needs and scenarios.

[0027] The description of specific embodiments details the structure, performance, effects, or other features to enable those skilled in the art to fully understand the specific implementation of these embodiments. However, this does not preclude those skilled in the art from employing technical solutions different from the described structure, performance, effects, or features to achieve the core content of the present invention under specific circumstances.

[0028] As attached Figure 1 As shown, Figure 1 This is a flowchart of a large-model-based mixed recognition technology for Cantonese, Chinese and English languages. This embodiment includes the following five steps.

[0029] Users bind their devices using the mini-program and create conversation and call configurations.

[0030] Users scan a QR code using the mini-program to bind their device, then create dialogue and call configurations within the mini-program. Users can set dialogue endpoints and end points, configure dialogue content and call parameters (including dialing time, phone number, and language). All settings will be synchronized to the device, which will automatically acquire the voice and create a scheduled task.

[0031] The cloud sends the outbound call settings to the device, and the device then schedules a timed call.

[0032] After the user saves the outbound call configuration, the cloud will synchronize and update it to the device. Upon receiving the configuration information, the device will automatically create a scheduled task based on the set outbound call time.

[0033] The device makes a call to the customer, engages in a question-and-answer session, and uses a model that supports mixed Cantonese, Chinese, and English speech recognition to convert the speech into text.

[0034] When the scheduled time arrives, the device will dial the phone numbers configured in the call log sequentially. During each call, the device will play the corresponding audio message for the specified node text, using the language set in the call log. After hearing the audio, the customer must answer a question. If the customer does not answer, the device will repeat the question once; if the customer still does not answer, the device will continue playing the audio message for the next node until the end node is reached.

[0035] After the responses are completed, a large model is used to evaluate the consistency of the question-and-answer process:

[0036] After the user answers the last question, the generated text data from the call will be uploaded to the cloud. The cloud will then use carefully crafted prompts to input the text data into a large model for consistency evaluation. If any inconsistencies are found during the question-and-answer process, the customer will be asked to provide further explanation.

[0037] Here is an example of prompts combined with question-and-answer text used for consistency assessment:

[0038] {

[0039] A project manager reviewed the day's task tracking in their end-of-day summary. The following is the project manager's response:

[0040] Q: How many project tasks did you track today?

[0041] A: Five tasks

[0042] Q: How many tasks have been completed?

[0043] A: Three tasks

[0044] Q: How many hours did it take to complete the task today?

[0045] A: It took about fifteen hours.

[0046] Q: How many tasks are at risk and require follow-up today?

[0047] A: Two tasks

[0048] Q: Please list the details of each task, including the task name, the person in charge, and the completion status.

[0049] A: The first task is the development module A, led by Engineer Zhang, and has been completed; the second task is the testing module B, led by Engineer Li, and there are risks involved.

[0050] Please check if the number of task traces mentioned in Question 1 matches the number of task details listed in Question 5. If the numbers match, return QA_RELEVANT; if the numbers do not match, please provide a prompt and request further explanation.

[0051] }

[0052] Input the text of the question-and-answer process into the large model to generate a detailed analysis report:

[0053] After the customer provides additional information, the device will upload the call transcript, including the supplementary information, to the cloud. The cloud will then combine this data with carefully designed prompts and input them into a large-scale model to generate a detailed analysis report of the call process.

[0054] Below is an example of prompts combined with question-and-answer text including supplementary information used to generate results feedback:

[0055] {

[0056] A project manager reviewed the day's task tracking in their end-of-day summary. The following is the project manager's response:

[0057] Q: How many project tasks did you track today?

[0058] A: Five tasks

[0059] Q: How many tasks have been completed?

[0060] A: Three tasks

[0061] Q: How many hours did it take to complete the task today?

[0062] A: It took about fifteen hours.

[0063] Q: How many tasks are at risk and require follow-up today?

[0064] A: Two tasks

[0065] Q: Please list the details of each task, including the task name, the person in charge, and the completion status.

[0066] A: The first task is the development module A, led by Engineer Zhang, and has been completed; the second task is the testing module B, led by Engineer Li, and there are risks involved.

[0067] Q: You mentioned that you tracked five project tasks today, but only provided information on two of them. Please provide information on the other three tasks.

[0068] A: The third task is designing module C, led by Engineer Zhao. It's progressing smoothly and is expected to be completed tomorrow. The fourth task is document organization, led by Engineer Sun. Some issues have arisen and require additional time to resolve. The fifth task is customer feedback analysis, led by Engineer Wu. It's completed, and the report is currently being compiled.

[0069] According to the company's daily project management data, each project manager tracks an average of 5 tasks per day, completes 2 tasks, works an average of 30 hours per task, and has an average of 3 tasks with risks.

[0070] Now you need to provide a specific evaluation of the situation summarized by the project manager above. Be concise and avoid generalities, so that the sales team feels your sincerity. At the end of your evaluation, please thank the sales team for their efforts and say goodbye.

[0071] }

Claims

1. A large-scale model-based Cantonese-Mandarin-English hybrid speech recognition technology has the following characteristics: A parallel corpus containing 100,000 hours of Cantonese, Mandarin, and English speech data is constructed using self-recorded and publicly available open-source speech datasets; the underlying model employs a finely tuned Whisper Large architecture to enhance multilingual hybrid recognition capabilities. After the speech content is recognized as text, it is input into the large-scale model for processing, the consistency of the question-and-answer process is evaluated, and the entire call process is analyzed in depth, ultimately generating a detailed analysis report.

2. The method according to claim 1, characterized in that, The corpus was constructed by including not only speech data recorded from subjects from different regions and backgrounds in the Greater Bay Area, but also extensive speech data collected from multiple open-source platforms, ensuring coverage of various speech environments and accents to improve the robustness and applicability of the model in practical applications.

3. The method according to claim 1, characterized in that, The underlying model is Whisper Large, which is fine-tuned in a multi-GPU environment to adapt to the mixed recognition of Cantonese, Mandarin, and English. An adaptive learning rate and periodic evaluation mechanism are employed during the fine-tuning process to ensure optimal learning performance at different training stages.

4. The method according to claim 1, characterized in that, By optimizing the model through a multi-task learning strategy, and using a specific loss function to simultaneously handle speech recognition tasks in three languages, the overall recognition accuracy is improved. This strategy enhances the model's versatility and flexibility by refining the processing logic for each language and utilizing shared layers to capture common characteristics across different languages.

5. The method according to claim 1, characterized in that, During training, dynamic learning rate adjustments are implemented, and the model is periodically saved and validated to ensure optimal performance at each training stage. This process includes automatically adjusting the learning rate based on the performance on the validation set within specific training cycles, and avoiding overfitting through early stopping strategies.

6. The method according to claim 1, characterized in that, After training, the model underwent comprehensive testing on independent test sets to verify its effectiveness in processing Cantonese, Mandarin, and English. The testing process included evaluating the model's performance in various real-world scenarios, such as speech recognition accuracy in noisy environments and its ability to handle fast-paced speech.

7. The method according to claim 1, characterized in that, After the customer answers the last question, the device uploads the question-and-answer text to the cloud. The cloud then combines the prompt template with the text, inputs it into a large model, and performs a consistency assessment of the question-and-answer content. If inconsistencies are found, the system will request the customer to provide further clarification once, and up to one more time if necessary.

8. The method according to claim 1, characterized in that, After the large model completes its consistency assessment, if the customer needs to supplement information, the device will upload the supplemented response text to the cloud. The cloud will then combine the new prompt word template with the new text and input it back into the large model for analysis, ultimately generating a detailed analysis report.

Images