A speech recognition method based on fuzzy matching mechanism

Abstract

The application relates to the technical field of speech recognition, in particular to a speech recognition method based on a fuzzy matching mechanism, which comprises the following steps: step one, collecting an audio stream to obtain audio data; step two, identifying the audio data based on a speech recognition algorithm to obtain initial text data; and step three, updating the initial text data based on the fuzzy matching mechanism, which comprises the following steps: establishing a self-defined error correction table, and corresponding preset error instruction text and preset instruction text through the self-defined error correction table; after obtaining the initial text data, updating the preset error instruction text in the initial text data into the corresponding preset instruction text through the self-defined error correction table. The speech recognition method in the application can be applied to a man-machine interaction scene, can improve the accuracy of robot instruction analysis, and does not require the user to repeatedly adjust the language and repeat the instruction.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more specifically to a speech recognition method based on a fuzzy matching mechanism. Background Technology

[0002] With the continuous development of robotics technology, robots have been widely applied to various production and daily life scenarios, such as factory patrols, poultry farming, and home care. To achieve natural and convenient interaction between robots and users, voice interaction has become the mainstream interaction method for robots. Through voice interaction, functions such as command transmission and information feedback can be realized, greatly improving the convenience and efficiency of human-robot interaction.

[0003] The complete process of robot voice interaction mainly includes four core stages: voice acquisition, speech-to-text conversion, semantic parsing, command execution, and feedback. The voice acquisition stage obtains the user's voice signal, forming a continuous audio stream. The speech-to-text conversion stage is the core preliminary step for the robot to read user commands. It processes the acquired audio stream using Automatic Speech Recognition (ASR) technology, converting the analog voice signal into digital text data that the robot can parse. The semantic parsing stage performs intent recognition and command parsing on the converted text data. Finally, the command execution stage completes the corresponding operation based on the parsing results, and the feedback stage uses speech synthesis technology to convert the execution result into a voice signal and feed it back to the user.

[0004] Currently, existing robot speech-to-text conversion processes suffer from significant technical deficiencies, making them ill-suited for the complex speech environments of real-world applications. Specifically, current speech recognition technologies are mostly trained on standard Mandarin, exhibiting poor adaptability to regional dialects and accented Mandarin, and lacking the ability to identify and distinguish homophones and synonyms in speech signals. In practical use, users' regional dialects and accents, as well as homophones and synonyms in spoken expression, can cause deviations in the speech recognition technology's conversion of the audio stream, leading to the generation of erroneous text data and consequently affecting the accuracy of subsequent semantic analysis.

[0005] The aforementioned technical flaws in the speech-to-text process can easily lead to robot command parsing failures. Specifically, erroneous text data prevents the robot from accurately understanding the user's true instructions. Existing technologies largely rely on precise matching of fixed keywords, offering little ability to handle incorrectly converted text data or homophones and synonyms. This forces users to repeatedly recite standard instructions, reducing human-computer interaction efficiency and limiting the widespread application of robots in diverse scenarios such as factories, farms, and homes.

[0006] In view of this, how to overcome the shortcomings of the existing technology has become the subject of study and solution of this invention. Summary of the Invention

[0007] The purpose of this invention is to provide a speech recognition method based on a fuzzy matching mechanism.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A speech recognition method based on fuzzy matching mechanism includes:

[0010] Step 1: Acquire the audio stream to obtain audio data;

[0011] Step 2: Recognize the audio data using a speech recognition algorithm to obtain the initial text data;

[0012] Step 3: Update the initial text data based on the fuzzy matching mechanism, including:

[0013] Create a custom error correction table, and use this custom error correction table to map preset error command texts to preset command texts;

[0014] After obtaining the initial text data, the preset error instruction text in the initial text data is updated to the corresponding preset instruction text using a custom error correction table. It is understood that if the initial text data does not contain preset error instruction text, no update is needed, and the custom error correction table is established before acquiring the audio stream.

[0015] This application is primarily described using movement instructions.

[0016] This application illustrates the application of speech recognition methods in robot interaction scenarios, using a robot equipped with an audio acquisition and preprocessing module, a voice activity detection (VAD) module, a speech recognition module, a two-layer intent classification module (including a pinyin fuzzy matching layer and a large language model semantic understanding layer), a motion control module, a robot state acquisition module, and a text-to-speech (TTS) module. When using an offline speech recognition algorithm, the speech recognition module is an offline speech recognition module (ASR). The motion control module can be integrated into the ROS2 robot operating system, acquiring the robot's mileage and pose information by subscribing to odometry and IMU topics, and issuing motion control commands to the robot chassis controller based on the parsed motion parameters. The robot state acquisition module can collect the robot's internal operating status in real time, including battery level and real-time position coordinates. The speech synthesis module can convert system-generated text information into speech signals and output them through a speaker, enabling the robot to respond to user commands.

[0017] In step one, an audio acquisition and preprocessing module is used to acquire the audio stream in real time. Specifically, the audio stream can be acquired in real time through a microphone. This audio acquisition and preprocessing module is also used to perform the following preprocessing steps.

[0018] In step two, we will use offline speech recognition as an example. By using an offline speech recognition algorithm based on the ONNX format, we can recognize and process the speech signals collected by the microphone and output the initial text sequence data, which provides a data foundation for the robot's semantic understanding and corresponding instruction execution.

[0019] In step three, after obtaining the initial text data, a custom error correction table is used to update the preset error command text in the initial text data with the corresponding preset command text. This improves the accuracy of the text data, enabling the robot to correctly execute commands and perform voice interaction by completing semantic parsing. For example, the preset error command text includes erroneous command text caused by dialects, accents, homophones, and synonyms in the speech-to-text stage. This avoids interference from erroneous command text in the semantic parsing stage from the source, compensating for the shortcomings of existing speech recognition technology in its poor adaptability to non-standard speech.

[0020] In summary, by applying the speech recognition method described in this application to human-computer interaction scenarios, the accuracy of robot command parsing can be improved, eliminating the need for users to repeatedly adjust their speech and reiterate commands. Simultaneously, it enables the robot to adapt to non-standard Mandarin-speaking users and diverse application scenarios, lowering the barrier to entry and facilitating the widespread adoption of robots in factories, farms, homes, and other settings. Furthermore, it eliminates the need for large-scale reconstruction of existing speech recognition models; the accuracy of robot command parsing can be continuously improved simply by optimizing a custom error correction table. This approach has low implementation costs and strong practicality and scalability.

[0021] It should be noted that semantic parsing, instruction execution, and feedback, two routine aspects of robots, are not covered in this application. They are only briefly explained to aid understanding and will not be elaborated upon here.

[0022] In a further technical solution, after obtaining the audio data in step one, the audio data is preprocessed. This preprocessing step includes resampling, noise reduction, and audio feature extraction, thereby improving the accuracy of the initial text data obtained in step two.

[0023] It is understandable that if there is a large deviation between the initial text data and the actual text data, when the speech recognition method in this application is applied to robot interaction scenarios, even if the initial text data is updated using a fuzzy matching mechanism, the robot's recognition accuracy of user commands is likely to be lower than expected.

[0024] It should be noted that resampling, denoising, and audio feature extraction of audio data are routine preprocessing procedures in the field of audio signal processing: resampling involves uniformly adjusting parameters such as the audio sampling rate to adapt to subsequent processing requirements; denoising uses signal processing methods to remove environmental noise and other interference signals to improve audio quality; and feature extraction extracts numerical features such as MFCC that can characterize the essence of audio from the audio waveform, facilitating algorithm recognition and analysis. The above operations are all common knowledge and general technical means familiar to those skilled in the field.

[0025] In a further technical solution, in step one, audio streams are selectively acquired based on a speech segment determination mechanism. This speech segment determination mechanism includes: performing frame-by-frame detection on the audio stream; if the audio energy of consecutive A frames is detected to be ≥ a first threshold, then the starting frame of the consecutive frames is marked as the start endpoint of the speech segment; subsequently, if the audio energy of consecutive B frames is detected to be < a first threshold, then the starting frame of the consecutive frames is marked as the end endpoint of the corresponding speech segment, and then the speech segment is used as audio data; A and B are both positive integers.

[0026] Understandably, A and B can adjust their specific values ​​according to requirements.

[0027] In summary, the speech segment determination mechanism executed by the speech activity detection module (VAD) can monitor the start and end points of the audio stream in real time based on an end-to-end neural network model, autonomously determine whether the speech has ended, and prevent silent segments from entering the recognition process.

[0028] As the raw audio signal, the audio stream needs to be processed by framing (e.g., the frame length is set to 10-15ms) to ensure that the energy changes of each audio segment can be accurately captured, so that the audio energy of each audio frame can be calculated frame by frame. Audio energy is used to characterize the strength of a single frame of audio. The higher the energy, the clearer and louder the sound.

[0029] The first threshold can be adjusted as needed. If the audio energy of consecutive A frames (e.g., 3-6 frames) is greater than or equal to the first threshold, it indicates that what is happening is continuous and effective speech, rather than momentary noise or a silent phase, thus avoiding misjudging the start and end points of the speech segment.

[0030] If the audio energy of a certain frame is less than the first threshold, it indicates the presence of a silent segment (such as when the user takes a breath). Considering some special cases (such as when the user speaks intermittently), the speech segment is only judged to have ended when the audio energy of consecutive B frames is less than the first threshold, in order to avoid the robot being unable to accurately recognize user commands due to premature termination of the speech segment.

[0031] By selectively acquiring audio streams based on a speech segment determination mechanism, speech segments containing target instructions can be accurately obtained, avoiding blank audio (which is noise or jargon that does not contain the user's actual instructions) from entering the recognition process in step two, thereby improving speech recognition efficiency and thus improving the efficiency and effectiveness of human-computer interaction.

[0032] Here's an example to help you understand: The user gives the command "move forward 2 meters". The interval between the input of "move forward" and "2 meters" in the command is 2 seconds. Taking seconds as an example, set the acquisition stop time threshold in seconds and set the threshold to 3 seconds. At this time, the user command can be fully acquired, and the audio stream will only be acquired within 3 seconds after the "2 meters" input is completed. The audio stream will not be continuously acquired.

[0033] Further technical solutions, including fuzzy matching mechanisms, include:

[0034] Establish an instruction mapping table, which includes preset instruction text and corresponding pinyin strings;

[0035] Before or after using a custom error correction table, selectively update the current text data based on a prefix matching strategy, which includes:

[0036] At least the pinyin string of the beginning of the current text data of a preset length is obtained, and the pinyin string is matched with the pinyin string in the instruction mapping table. If the similarity between the two is greater than the second threshold, the current text data is updated based on the corresponding preset instruction text.

[0037] Understandably, the specific value of the second threshold can be adjusted according to actual needs.

[0038] Preset command texts, such as forward, backward, left turn, right turn, and stop, can be customized and updated. The command mapping table is particularly suitable for short text strings like motion command texts (using pinyin). For action command texts, the command mapping table can include corresponding motion control action identifiers, allowing the robot to execute motion commands based on these identifiers. This application can be described using preset action command texts (or simply action command texts) to represent preset command texts.

[0039] For example, the instruction mapping table contains motion instruction texts "forward", "backward", and "stop", with corresponding pinyin strings "qianjin", "houtui", and "tingzhi" respectively, and corresponding motion control action identifiers "MOVE_FORWARD", "MOVE_BACKWARD", and "STOP" respectively.

[0040] For example, the current text data contains "前景" (qianjing in pinyin), and the corresponding preset instruction text contains "前进" (qianjin in pinyin). The similarity between the two pinyin strings is 0.857 (greater than the second threshold of 0.6), which belongs to homophone / near-homophone errors. The "前景" in the current text data needs to be updated to "前进". Based on this, the custom error correction table may not include this unconventional instruction variant of "前景". Another example is that the current text data contains "停职" (tingzhi in pinyin), and the corresponding preset instruction text contains "停止" (tingzhi in pinyin). The two pinyin strings are the same, which is judged as homographs with different characters. The "停职" in the current text data needs to be updated to "停止".

[0041] It should be noted that compared with the following similarity matching strategy, the prefix matching strategy has higher adaptability for some instruction texts (such as action instruction texts), and has the advantages of high computational efficiency, low probability of incorrect matching, and maintaining semantic integrity, as follows:

[0042] When using prefix matching, only the pinyin string of the beginning preset length part of the current text data needs to be compared, with low time complexity and high matching efficiency;

[0043] Considering that the word order of users first saying action words (such as "前进", "左转", etc.) and then parameters is very common in Chinese instructions, at this time, the prefix matching strategy can directly extract the action intention and will not be affected by similar syllables in the subsequent parameters;

[0044] Users' instructions usually consist of an action + a parameter (that is, the proportion of action instructions is high). The prefix matching strategy can directly identify the action word and leave the parameter part for subsequent parsing; if direct similarity matching is done, it is easy to compare the entire pinyin string of the instruction (including the parameter) with the standard pinyin string as a whole, reducing the matching accuracy; for example, the overall pinyin of "前景2米" and the pinyin of "前进" will decrease due to the presence of "2米", resulting in the similarity being lower than the preset value and unable to correctly update "前景".

[0045] In summary, the error correction ability of the custom error correction table is limited. With the cooperation of the instruction mapping table, the accuracy of the final text data can be improved. Specifically, the instruction mapping table can supplement the custom error correction table, so that the custom error correction table does not need to enumerate all instruction variants, and only needs to list the conventional variants of the instruction text. Both the custom error correction table and the instruction mapping table can be flexibly configured, facilitating the addition of new instruction types and error correction rules according to the actual usage scenario, and having high practicality and expandability.

[0046] Furthermore, for common issues in speech recognition such as homophones and near-homophone errors that are difficult to identify accurately, existing methods generally use word vector semantic similarity. However, the fuzzy matching mechanism (or fuzzy matching algorithm) used in this application can directly utilize the speech features of Chinese Pinyin for matching, which has the characteristics of low computational cost and high accuracy, and is especially suitable for robot motion control scenarios.

[0047] In some implementations, the prefix matching strategy includes: obtaining all the pinyin strings of the current text data, and matching the pinyin strings of the first preset length portion of the current text data with the pinyin strings in the instruction mapping table. If they are completely consistent, the current text data is updated based on the corresponding preset instruction text.

[0048] In some implementations, after using a custom error correction table, the current text data is selectively updated based on a prefix matching strategy. Specifically, if the current text data can be fully recognized, the prefix matching strategy is abandoned.

[0049] Further technical solutions, including fuzzy matching mechanisms, include:

[0050] After adopting a custom error correction table, the current text data is selectively updated based on a similarity matching strategy. This similarity matching strategy includes: matching all the pinyin strings of the current text data with the pinyin strings in the instruction mapping table to calculate the similarity between the two. If the similarity is greater than the third threshold, the current text data is updated based on the corresponding preset instruction text.

[0051] Understandably, the specific value of the third threshold can be adjusted according to actual needs.

[0052] Similarity matching strategies can assist prefix matching strategies or custom error correction tables to further ensure the accuracy of the final text data.

[0053] A further technical solution involves first parsing the initial text data after acquisition. If the initial text data cannot be completely parsed, it is updated using a custom error correction table to obtain the first updated text data, which is then parsed. If the first updated text data cannot be completely parsed, it is updated using a prefix matching strategy to obtain the second updated text data, which is then parsed. If the second updated text data cannot be completely parsed, it is updated using a similarity matching strategy to obtain the third updated text data, which is then parsed. If the third updated text data cannot be completely parsed, it is updated based on a large language model to obtain the fourth updated text data, which is then parsed.

[0054] In summary, each time text data is updated, it is first parsed. If some text data still cannot be parsed, then subsequent update operations are performed. If all text data can be parsed, then the update of the current text data is abandoned.

[0055] Compared to using a large language model, using a custom error correction table, prefix matching strategy, or similarity matching strategy has lower requirements for cost, computing resources, and transmission speed. Therefore, the large language model is used as the last update method.

[0056] Compared to prefix matching or similarity matching strategies, using a custom error correction table does not require pinyin string comparison and has lower requirements for cost, computing resources, and transmission speed. Therefore, the custom error correction table is used as the initial update method.

[0057] For example, if the first updated text data can be completely parsed and determined to be for precise motion control, the motion parameter parsing subprocess can be initiated. This subprocess extracts distance or angle values ​​(such as "5 meters," "30 degrees," "2 meters," etc.) from the text, generates structured motion commands, and hands them over to the motion control module for processing. After processing, the speech synthesis module provides voice feedback, completing the human-computer interaction loop.

[0058] A further technical solution involves calculating the length of the pinyin string of the initial text data before using the custom error correction table. If the length is less than or equal to the fourth threshold, the custom error correction table is called; if the length is greater than the fourth threshold, the large language model is called directly.

[0059] To illustrate, the overall length of the pinyin strings for motion commands is generally relatively short, and the number of their common semantic / spelling variations (such as pinyin input errors and synonymous command expressions) is relatively fixed. Therefore, by constructing a custom error correction table, these variations can be efficiently stored and quickly matched, ensuring the parsing efficiency and accuracy of motion commands. However, for non-motion-related complex interactive commands (such as commands from users asking about the robot's current working status or historical task execution), the semantic complexity is high and the variations are diverse. Custom error correction tables are difficult to achieve accurate adaptation. Therefore, a pre-set large language model module can be directly called to perform semantic completion, logical sorting, and parsing rule optimization on the initial text data, thereby achieving accurate understanding and response to complex interactive commands.

[0060] Generally speaking, in combination with the following offline speech recognition algorithm, the offline speech recognition algorithm and the pinyin fuzzy matching algorithm (corresponding to the fuzzy matching mechanism) are responsible for the rapid parsing of precise motion instructions. At the same time, a lightweight large language model is deployed locally, which is responsible for natural language understanding and dialogue generation, and realizes multi-task understanding and scheduling through double-layer intention classification. Motion control instructions have clear pattern characteristics (short instructions, concentrated keywords), which are suitable for rapid processing with lightweight pinyin fuzzy matching. Natural language dialogue, status query and complex instructions are more suitable for leveraging the deep semantic understanding ability of the large language model.

[0061] Taking the fuzzy matching mechanism and the large language model as an example (illustrated with motion control instructions), the former constructs a pinyin fuzzy matching layer, and the latter constructs a semantic understanding layer of the large language model. The cooperation method is as follows:

[0062] The initial text data is first sent to the pinyin fuzzy matching layer. If the pinyin fuzzy matching layer fails to match (that is, it is not a robot motion control instruction), the initial text data is sent to the semantic understanding layer of the large language model for in-depth semantic parsing. The large language model judges the user's intention to belong to one of the following categories based on the semantics and context of the user input and generates a corresponding reply:

[0063] (a) Status query: Such as "How much power is left?" and "Where are you?" The large language model calls the status query module to output the current state of the robot through speech synthesis;

[0064] (b) Daily chat: Such as "What do you think of the current development of robots?" and "Tell a joke." The large language model generates appropriate chat replies;

[0065] [[ID=!15]] (c) Semantic correction: Correct homophones and typos caused by accents and recognition errors (such as "qianjin 5 meters") to correctly understand the instruction as "go forward five meters", and then generate a motion instruction; Through the collaborative correction of the large language model and the fuzzy matching mechanism, the fault tolerance rate of voice instructions can be further improved. In particular, it can automatically correct homophonic and near-sound errors. Even in the case of unclear pronunciation or accents of users, as well as in the case of complex instructions output by users, the instruction recognition success rate is guaranteed;

[0066] (d) Complex instructions: Such as "Go to the kitchen to see what's there." The large language model decomposes the task and generates a sequence of motion instructions.

[0067] It should be noted that in the translation of item , there seems to be an exclamation mark (!) in the original text which is not in a standard format. I have translated it as it is while keeping the punctuation mark for your reference. If this is an error in the original text, you may want to correct it before final use.The pinyin fuzzy matching layer and the large language model semantic understanding layer constitute a two-layer architecture. This architecture allows most motion control commands (the most prevalent and frequently invoked commands) to pass through the lightweight pinyin fuzzy matching channel, resulting in low computational overhead. The large language model only handles tasks requiring deep semantic understanding, avoiding invoking the large language model for every voice interaction, thus reducing computational resource investment and latency. Furthermore, for example, by constructing this two-layer architecture, the system can automatically identify the type of user input intent and route motion control, status query, and casual conversation tasks to the optimal processing channel respectively. This allows users to complete operations, obtain information, and engage in conversations using everyday language without needing to learn specific command formats.

[0068] It should be noted that the response text generated by the large language model is converted into speech output through a speech synthesis module, forming a complete dialogue interaction. This is a common method for achieving dialogue interaction in robots. The large language model is any existing conventional AI large model, and there are no specific restrictions, as long as it meets the above requirements.

[0069] To facilitate understanding of how large language models are used, further details are provided below:

[0070] By dynamically injecting the robot's real-time status information into the system prompts of the large language model, the large language model can accurately answer various natural language queries from users about the robot's own status.

[0071] For robots, their status information, such as battery level and location coordinates, is collected in real time and then published through the Ros2 topic.

[0072] When speech-to-text recognition (i.e., the text data mentioned above) triggers the large language model, such as when a user inquires about the robot's status, the algorithm is invoked, and the robot's status information is subscribed to. The system can also construct contextual prompts (the system prompts can be adjusted according to the robot's role and functional scope). For example, if a user asks, "Xiao Zhi, how much battery do you have left?", "Xiao Zhi" serves as the prompt, triggering the robot's understanding process to convert the information into text data. At this point, the large language model can be directly invoked. The robot's status can be queried using natural language; this is a standard setting. For instance, through a prompting engineering method that dynamically injects status information into the large language model, users can directly query the robot's real-time status using natural language, without needing to view it through an app or webpage, thus lowering the barrier to human-computer interaction.

[0073] Understandably, using a large language model as an example, if a large language model receives a complex instruction but cannot fully parse it, thus preventing the robot from executing the instruction, it is still possible to synthesize a response voice. This voice can then be used to allow the user to issue the instruction again, for example, generating a response voice like "I didn't hear you clearly, please say it again."

[0074] For a further technical solution, in step two, the audio data is recognized based on an offline speech recognition algorithm to obtain a recognized text, and then the recognized text is converted based on a text conversion mechanism to obtain initial text data; the text conversion mechanism includes:

[0075] Convert the recognized Chinese capital numbers into Arabic numerals;

[0076] Convert the recognized Chinese numbers with decimal points into Arabic numerals with decimal points;

[0077] Convert the recognized "two" into 2.

[0078] The text conversion mechanism can be implemented relying on existing parsing algorithms. This algorithm uses regular expression matching and a recursive Chinese number parsing function, which can parse integers and one decimal place within the range of 0 to 999, and can meet the accuracy requirements in the robot motion control scenario.

[0079] Illustrations are as follows:

[0080] The recognized text includes "five" and "one hundred and eighty" (which are Chinese capital numbers), and they are automatically converted into 5 and 180 (which are Arabic numerals) based on the text conversion mechanism;

[0081] The recognized text includes "one point five" and "three point five" (which are Chinese numbers with decimal points), and they are automatically converted into 1.5 and 3.5 (which are Arabic numerals with decimal points) based on the text conversion mechanism.

[0082] It should be noted that using an offline speech recognition algorithm can greatly reduce the device's dependence on the network, and voice conversion can be completed without being connected to the network. It can still operate stably in scenarios without network or with weak network; at the same time, the voice data is processed locally throughout the process, which can effectively avoid the privacy and security risks brought by cloud transmission, and can also reduce the network interaction delay and improve the instruction response efficiency. However, there are also certain problems. Specifically, the core function of offline speech recognition is to convert human speech into text, rather than text recognition. Due to the limitations of the embedded end model volume and computing power, most lightweight offline solutions have trimmed the post-processing module for digital normalization in order to achieve lightweight deployment. At this time, if the user says "advance fifty point two centimeters", the algorithm can only output the Chinese characters "advance fifty point two centimeters" and cannot automatically convert them into the Arabic numerals "advance 50.2 centimeters", resulting in the robot being difficult to parse the accurate numerical value. The text conversion mechanism can effectively solve this problem, realize the automatic conversion of Chinese numbers and decimal expressions into standard Arabic numerals, and output a normalized numerical value that can be directly parsed by the robot, ensuring the accurate recognition and smooth execution of quantitative instructions.

[0083] The offline speech recognition model and the local large language model are combined to form a hierarchical architecture, enabling the robot to still execute precise motion instructions and provide voice feedback in a network-free environment, meeting the requirements of the robot working in a network-free environment in reality.

[0084] In some embodiments, for the final text data (already updated), an offline speech synthesis algorithm is used to convert it into voice data, facilitating the robot to reply to the user and improving the degree of human-computer interaction. The offline speech synthesis algorithm can also get rid of the network signal and bandwidth limitations, and can normally complete the operation of text-to-speech output in a network-free scenario, adapting to the usage requirements of various mobile operations and closed scenarios; at the same time, the voice content is synthesized locally, avoiding data外传 (it seems there is a wrong word here, maybe "transmission"), improving the security of interaction information, and at the same time conforming to the low-power and lightweight deployment characteristics of embedded terminals.

[0085] It should be noted that the offline speech synthesis algorithm has been lightweighted and streamlined to adapt to hardware performance limitations, and generally lacks the text digital preprocessing logic. When processing Arabic numerals and decimal points, its conversion ability is relatively poor, and there may even be problems where the decimal point cannot be recognized. For example, when synthesizing "3.5 meters", the user will hear "Okay, go forward three five meters", resulting in a rigid pronunciation and incorrect semantic expression, affecting the interaction experience and instruction understanding. Based on this, before using the offline speech synthesis algorithm, the final text data is adjusted, including:

[0086] Judge whether the moving distance is ≥ 1 meter. If so, convert the corresponding Arabic numerals to Chinese capital numerals, and convert the decimal point in them to the Chinese numeral "point". For example, convert "1 meter" to "one meter", and when synthesizing speech, it can directly synthesize "Okay, have advanced one meter". Another example is to convert "1.5 meters" to "one point five meters", and when synthesizing speech, it can directly synthesize "Okay, have advanced one point five meters"; if not, convert the corresponding moving unit to centimeters, and convert the corresponding Arabic numerals to Chinese capital numerals. For example, convert "0.5 meters" to "50 centimeters", and when synthesizing speech, it can directly synthesize "Okay, have advanced fifty centimeters". Through the coordinated cooperation of the above text conversion mechanism and the conversion mechanism here, the user can say instructions in the most natural Chinese way (such as "advance one point five meters"), and at the same time the robot can also give feedback in the same natural Chinese (such as "Okay, advance one point five meters"), greatly improving the naturalness of voice interaction and the user experience.

[0087] In summary, when the speech recognition method in this application is applied to the robot interaction scenario, the robot can not only accurately respond to motion control instructions, but also conduct natural language chat conversations and status inquiries, and at the same time can significantly reduce the impact of speech recognition errors on the instruction parsing success rate, and can still achieve intelligent scheduling of multiple types of tasks in an offline environment.

[0088] Due to the application of the above-mentioned solution, the technical solution of this application has the following advantages and effects compared with the prior art:

[0089] The speech recognition method in this application includes the following steps:

[0090] Step 1: Acquire the audio stream to obtain audio data;

[0091] Step 2: Recognize the audio data using a speech recognition algorithm to obtain the initial text data;

[0092] Step 3: Update the initial text data based on the fuzzy matching mechanism, including:

[0093] Create a custom error correction table, and use this custom error correction table to map preset error command texts to preset command texts;

[0094] After obtaining the initial text data, the preset error instruction text in the initial text data is updated to the corresponding preset instruction text through a custom error correction table.

[0095] This application describes the application of speech recognition methods in robot interaction scenarios, and illustrates the robot's equipment with an audio acquisition and preprocessing module, a voice activity detection (VAD) module, a speech recognition module, a two-layer intent classification module (including the pinyin fuzzy matching layer and the large language model semantic understanding layer described below), a motion control module, a robot state acquisition module, and a text-to-speech (TTS) module.

[0096] In step one, the audio stream is acquired in real time using an audio acquisition and preprocessing module. Specifically, the audio stream can be acquired in real time using a microphone.

[0097] In step two, we will use offline speech recognition as an example. By using an offline speech recognition algorithm based on the ONNX format, we can recognize and process the speech signals collected by the microphone and output the initial text sequence data, which provides a data foundation for the robot's semantic understanding and corresponding instruction execution.

[0098] In step three, after obtaining the initial text data, a custom error correction table is used to update the preset error command text in the initial text data with the corresponding preset command text. This improves the accuracy of the text data, enabling the robot to correctly execute commands and perform voice interaction by completing semantic parsing. For example, the preset error command text includes erroneous command text caused by dialects, accents, homophones, and synonyms in the speech-to-text stage. This avoids interference from erroneous command text in the semantic parsing stage from the source, compensating for the shortcomings of existing speech recognition technology in its poor adaptability to non-standard speech.

[0099] In summary, by applying the speech recognition method described in this application to human-computer interaction scenarios, the accuracy of robot command parsing can be improved, eliminating the need for users to repeatedly adjust their speech and reiterate commands. Simultaneously, it enables the robot to adapt to non-standard Mandarin-speaking users and diverse application scenarios, lowering the barrier to entry and facilitating the widespread adoption of robots in factories, farms, homes, and other settings. Furthermore, it eliminates the need for large-scale reconstruction of existing speech recognition models; the accuracy of robot command parsing can be continuously improved simply by optimizing a custom error correction table. This approach has low implementation costs and strong practicality and scalability. Attached Figure Description

[0100] Figure 1 This is a flowchart of the speech recognition method according to an embodiment of the present invention. Detailed Implementation

[0101] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0102] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence, nor are they intended to limit this case; they are merely used to distinguish components or operations described using the same technical terms.

[0103] The terms "connection" or "positioning" as used in this article can refer to two or more components or devices making direct physical contact with each other, or making indirect physical contact with each other, or to two or more components or devices operating or moving with each other.

[0104] The terms “include,” “including,” and “have” used in this article are all open-ended, meaning they include but are not limited to.

[0105] Unless otherwise specified, the terms used herein generally have their ordinary meaning in the context of the art, the subject matter, and the specific context. Certain terms used to describe this case will be discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing this case.

[0106] The terms “front,” “back,” “up,” “down,” “left,” and “right” used in this article are directional terms. In this case, they are only used to describe the positional relationship between the structures and are not intended to limit the specific direction of the protection scheme or its actual implementation.

[0107] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the scope of this work. Singular forms such as “a,” “this,” “this,” “the,” and “the” as used herein also include plural forms.

[0108] See Figure 1 A speech recognition method based on fuzzy matching mechanism includes:

[0109] Step 1: Acquire the audio stream to obtain audio data;

[0110] Step 2: Recognize the audio data using a speech recognition algorithm to obtain the initial text data;

[0111] Step 3: Update the initial text data based on the fuzzy matching mechanism, including:

[0112] Create a custom error correction table, and use this custom error correction table to map preset error command texts to preset command texts;

[0113] After obtaining the initial text data, the preset error instruction text in the initial text data is updated to the corresponding preset instruction text using a custom error correction table. It is understood that if the initial text data does not contain preset error instruction text, no update is needed, and the custom error correction table is established before acquiring the audio stream.

[0114] This embodiment is mainly described using motion commands.

[0115] This embodiment illustrates the application of speech recognition methods in a robot interaction scenario, using a robot equipped with an audio acquisition and preprocessing module, a voice activity detection (VAD) module, a speech recognition module, a two-layer intent classification module (including a pinyin fuzzy matching layer and a large language model semantic understanding layer), a motion control module, a robot state acquisition module, and a text-to-speech (TTS) module. When using an offline speech recognition algorithm, the speech recognition module is an offline speech recognition module (ASR). The motion control module can be integrated into the ROS2 robot operating system, acquiring the robot's mileage and pose information by subscribing to odometry and IMU topics, and issuing motion control commands to the robot chassis controller based on the parsed motion parameters. The robot state acquisition module can collect the robot's internal operating status in real time, including battery level and real-time position coordinates. The speech synthesis module can convert system-generated text information into speech signals and output them through a speaker, enabling the robot to respond to user commands.

[0116] In step one, an audio acquisition and preprocessing module is used to acquire the audio stream in real time. Specifically, the audio stream can be acquired in real time through a microphone. This audio acquisition and preprocessing module is also used to perform the following preprocessing steps.

[0117] In step two, we will use offline speech recognition as an example. By using an offline speech recognition algorithm based on the ONNX format, we can recognize and process the speech signals collected by the microphone and output the initial text sequence data, which provides a data foundation for the robot's semantic understanding and corresponding instruction execution.

[0118] In step three, after obtaining the initial text data, a custom error correction table is used to update the preset error command text in the initial text data with the corresponding preset command text. This improves the accuracy of the text data, enabling the robot to correctly execute commands and perform voice interaction by completing semantic parsing. For example, the preset error command text includes erroneous command text caused by dialects, accents, homophones, and synonyms in the speech-to-text stage. This avoids interference from erroneous command text in the semantic parsing stage from the source, compensating for the shortcomings of existing speech recognition technology in its poor adaptability to non-standard speech.

[0119] In summary, by applying the speech recognition method described in this embodiment to human-computer interaction scenarios, the accuracy of robot command parsing can be improved, eliminating the need for users to repeatedly adjust their speech and repeat commands. Simultaneously, it enables the robot to adapt to non-standard Mandarin-speaking users and diverse application scenarios, lowering the barrier to entry and facilitating the widespread adoption of robots in factories, farms, and homes. Furthermore, it eliminates the need for large-scale reconstruction of existing speech recognition models; the accuracy of robot command parsing can be continuously improved simply by optimizing a custom error correction table. This results in low implementation costs and strong practicality and scalability.

[0120] It should be noted that semantic parsing, instruction execution, and feedback, two routine aspects of a robot, are not covered in this embodiment. They are only briefly explained to aid understanding and will not be elaborated upon here.

[0121] In one embodiment of this application, in step one, after obtaining the audio data, the audio data is preprocessed. The preprocessing steps include resampling, noise reduction, and audio feature extraction, thereby improving the accuracy of the initial text data obtained in step two.

[0122] It is understandable that if there is a large deviation between the initial text data and the actual text data, when the speech recognition method in this embodiment is applied to a robot interaction scenario, even if the initial text data is updated using a fuzzy matching mechanism, the robot's recognition accuracy of user commands is likely to be lower than expected.

[0123] It should be noted that resampling, denoising, and audio feature extraction of audio data are routine preprocessing procedures in the field of audio signal processing: resampling involves uniformly adjusting parameters such as the audio sampling rate to adapt to subsequent processing requirements; denoising uses signal processing methods to remove environmental noise and other interference signals to improve audio quality; and feature extraction extracts numerical features such as MFCC that can characterize the essence of audio from the audio waveform, facilitating algorithm recognition and analysis. The above operations are all common knowledge and general technical means familiar to those skilled in the field.

[0124] In one embodiment of this application, in step one, audio streams are selectively acquired based on a speech segment determination mechanism. This speech segment determination mechanism includes: performing frame-by-frame detection on the audio stream; if the audio energy of consecutive A frames is detected to be ≥ a first threshold, then the starting frame of the consecutive frames is marked as the start endpoint of the speech segment; subsequently, if the audio energy of consecutive B frames is detected to be < the first threshold, then the starting frame of the consecutive frames is marked as the end endpoint of the corresponding speech segment, and then the speech segment is used as audio data; A and B are both positive integers.

[0125] Understandably, A and B can adjust their specific values ​​according to requirements.

[0126] In summary, the speech segment determination mechanism executed by the speech activity detection module (VAD) can monitor the start and end points of the audio stream in real time based on an end-to-end neural network model, autonomously determine whether the speech has ended, and prevent silent segments from entering the recognition process.

[0127] As the raw audio signal, the audio stream needs to be processed by framing (e.g., the frame length is set to 10-15ms) to ensure that the energy changes of each audio segment can be accurately captured, so that the audio energy of each audio frame can be calculated frame by frame. Audio energy is used to characterize the strength of a single frame of audio. The higher the energy, the clearer and louder the sound.

[0128] The first threshold can be adjusted as needed. If the audio energy of consecutive A frames (e.g., 3-6 frames) is greater than or equal to the first threshold, it indicates that what is happening is continuous and effective speech, rather than momentary noise or a silent phase, thus avoiding misjudging the start and end points of the speech segment.

[0129] If the audio energy of a certain frame is less than the first threshold, it indicates the presence of a silent segment (such as when the user takes a breath). Considering some special cases (such as when the user speaks intermittently), the speech segment is only judged to have ended when the audio energy of consecutive B frames is less than the first threshold, in order to avoid the robot being unable to accurately recognize user commands due to premature termination of the speech segment.

[0130] By selectively acquiring audio streams based on a speech segment determination mechanism, speech segments containing target instructions can be accurately obtained, avoiding blank audio (which is noise or jargon that does not contain the user's actual instructions) from entering the recognition process in step two, thereby improving speech recognition efficiency and thus improving the efficiency and effectiveness of human-computer interaction.

[0131] Here's an example to help you understand: The user gives the command "move forward 2 meters". The interval between the input of "move forward" and "2 meters" in the command is 2 seconds. Taking seconds as an example, set the acquisition stop time threshold in seconds and set the threshold to 3 seconds. At this time, the user command can be fully acquired, and the audio stream will only be acquired within 3 seconds after the "2 meters" input is completed. The audio stream will not be continuously acquired.

[0132] In one embodiment of this application, the fuzzy matching mechanism further includes:

[0133] Establish an instruction mapping table, which includes preset instruction text and corresponding pinyin strings;

[0134] Before or after using a custom error correction table, selectively update the current text data based on a prefix matching strategy, which includes:

[0135] At least the pinyin string of the beginning of the current text data of a preset length is obtained, and the pinyin string is matched with the pinyin string in the instruction mapping table. If the similarity between the two is greater than the second threshold, the current text data is updated based on the corresponding preset instruction text.

[0136] The preset command text in the command mapping table and the preset command text in the custom error correction table can be exactly the same or different, depending on the specific requirements.

[0137] Understandably, the specific value of the second threshold can be adjusted according to actual needs.

[0138] Preset command texts, such as forward, backward, left turn, right turn, and stop, can be customized and updated. The command mapping table is particularly suitable for short texts like motion command texts (characters with pinyin). For action command texts, the command mapping table can include corresponding motion control action identifiers, allowing the robot to execute motion commands based on these identifiers. This embodiment can be described using preset action command texts (or simply action command texts) to represent preset command texts.

[0139] For example, the instruction mapping table contains motion instruction texts "Forward", "Backward", "Stop", the corresponding pinyin strings are "qianjin", "houtui", "tingzhi" respectively, and the corresponding motion control action identifiers are "MOVE_FORWARD", "MOVE_BACKWARD", "STOP".

[0140] For example, the current text data contains "前景", whose corresponding pinyin string is "qianjing", while the preset instruction text contains "前进", whose corresponding pinyin string is "qianjin". The similarity between the two pinyin strings is 0.857 (greater than the second threshold 0.6), which belongs to homophone / near-homophone error. It is necessary to update "前景" in the current text data to "前进". Based on this, the custom error correction table may not include this unconventional instruction variant "前景". For another example, the current text data contains "停职", whose corresponding pinyin string is "tingzhi", while the preset instruction text contains "停止", whose corresponding pinyin string is "tingzhi". The two pinyin strings are the same, judged as homographs with different characters, and it is necessary to update "停职" in the current text data to "停止".

[0141] It should be noted that compared with the following similarity matching strategy, the prefix matching strategy has higher adaptability for some instruction texts (such as action instruction texts), and has the advantages of high computational efficiency, low probability of mis-matching, and maintaining semantic integrity, as follows:

[0142] When performing prefix matching, only the pinyin string of the beginning preset length part of the current text data needs to be compared, with low time complexity and high matching efficiency;

[0143] Considering that the word order of the user first saying the action word (such as "Forward", "Turn left", etc.) and then the parameter is very common in Chinese instructions, at this time the prefix matching strategy can directly extract the action intention and will not be affected by similar syllables in the subsequent parameters;

[0144] The user's instructions are usually composed of action + parameter (that is, the proportion of action instructions is high). The prefix matching strategy can directly identify the action word and leave the parameter part for subsequent parsing; if direct similarity matching is done, it is easy to compare the entire pinyin string of the instruction (including the parameter) with the standard pinyin string as a whole, reducing the matching accuracy; for example, the overall pinyin of "前景2米" and the pinyin of "前进" will decrease due to the presence of "2米", resulting in the similarity being lower than the preset value and unable to correctly update "前景".

[0145] In summary, the error correction capabilities of a custom error correction table are limited. However, by using an instruction mapping table in conjunction with the custom error correction table, the accuracy of the final text data can be improved. Specifically, the instruction mapping table can supplement the custom error correction table, eliminating the need to enumerate all instruction variations in the custom table; only the common variations of the instruction text need to be listed. Both the custom error correction table and the instruction mapping table are flexibly configurable, allowing for the addition of new instruction types and error correction rules based on actual usage scenarios, thus demonstrating high practicality and scalability.

[0146] Furthermore, for common issues in speech recognition such as homophones and near-homophone errors that are difficult to identify accurately, existing methods generally use word vector semantic similarity. However, the fuzzy matching mechanism (or fuzzy matching algorithm) used in this embodiment can directly utilize the speech features of Chinese Pinyin for matching, which has the characteristics of low computational cost and high accuracy, and is especially suitable for robot motion control scenarios.

[0147] In some embodiments, the prefix matching strategy includes: obtaining all the pinyin strings of the current text data, and matching the pinyin strings of the first preset length portion of the current text data with the pinyin strings in the instruction mapping table. If they are completely consistent, the current text data is updated based on the corresponding preset instruction text.

[0148] In some embodiments, after adopting a custom error correction table, the current text data is selectively updated based on a prefix matching strategy. Specifically, if the current text data can be fully recognized, the prefix matching strategy is abandoned.

[0149] In one embodiment of this application, the fuzzy matching mechanism further includes:

[0150] After adopting a custom error correction table, the current text data is selectively updated based on a similarity matching strategy. This similarity matching strategy includes: matching all the pinyin strings of the current text data with the pinyin strings in the instruction mapping table to calculate the similarity between the two. If the similarity is greater than the third threshold, the current text data is updated based on the corresponding preset instruction text.

[0151] Understandably, the specific value of the third threshold can be adjusted according to actual needs.

[0152] Similarity matching strategies can assist prefix matching strategies or custom error correction tables to further ensure the accuracy of the final text data.

[0153] In one embodiment of this application, after obtaining the initial text data, it is first parsed; if the initial text data cannot be completely parsed, the initial text data is updated through a custom error correction table to obtain the first updated text data, and then parsed; if the first updated text data cannot be completely parsed, the first updated text data is updated through a prefix matching strategy to obtain the second updated text data, and then parsed; if the second updated text data cannot be completely parsed, the second updated text data is updated through a similarity matching strategy to obtain the third updated text data, and then parsed; if the third updated text data cannot be completely parsed, the third updated text data is updated based on a large language model to obtain the fourth updated text data, and then parsed.

[0154] In summary, each time text data is updated, it is first parsed. If some text data still cannot be parsed, then subsequent update operations are performed. If all text data can be parsed, then the update of the current text data is abandoned.

[0155] Compared to using a large language model, using a custom error correction table, prefix matching strategy, or similarity matching strategy has lower requirements for cost, computing resources, and transmission speed. Therefore, the large language model is used as the last update method.

[0156] Compared to prefix matching or similarity matching strategies, using a custom error correction table does not require pinyin string comparison and has lower requirements for cost, computing resources, and transmission speed. Therefore, the custom error correction table is used as the initial update method.

[0157] For example, if the first updated text data can be completely parsed and determined to be for precise motion control, the motion parameter parsing subprocess can be initiated. This subprocess extracts distance or angle values ​​(such as "5 meters," "30 degrees," "2 meters," etc.) from the text, generates structured motion commands, and hands them over to the motion control module for processing. After processing, the speech synthesis module provides voice feedback, completing the human-computer interaction loop.

[0158] In one embodiment of this application, before using a custom error correction table, the length of the pinyin string of the initial text data is calculated. If the length is less than or equal to the fourth threshold, the custom error correction table is called. If the length is greater than the fourth threshold, the large language model is called directly.

[0159] To illustrate, the overall length of the pinyin strings for motion commands is generally relatively short, and the number of their common semantic / spelling variations (such as pinyin input errors and synonymous command expressions) is relatively fixed. Therefore, by constructing a custom error correction table, these variations can be efficiently stored and quickly matched, ensuring the parsing efficiency and accuracy of motion commands. However, for non-motion-related complex interactive commands (such as commands from users asking about the robot's current working status or historical task execution), the semantic complexity is high and the variations are diverse. Custom error correction tables are difficult to achieve accurate adaptation. Therefore, a pre-set large language model module can be directly called to perform semantic completion, logical sorting, and parsing rule optimization on the initial text data, thereby achieving accurate understanding and response to complex interactive commands.

[0160] In summary, the offline speech recognition algorithm and the pinyin fuzzy matching algorithm (corresponding to the fuzzy matching mechanism) are responsible for the rapid parsing of precise motion commands. Simultaneously, a lightweight large language model is deployed locally, responsible for natural language understanding and dialogue generation. Multi-task understanding and scheduling are achieved through two-layer intent classification. Motion control commands have clear pattern characteristics (short commands, concentrated keywords), making them suitable for rapid processing using lightweight pinyin fuzzy matching. Natural language dialogues, state queries, and complex commands are better suited to leveraging the deep semantic understanding capabilities of the large language model.

[0161] Using fuzzy matching mechanisms and large language models as examples (using motion control commands as an example), the former constructs a fuzzy matching layer for pinyin, while the latter constructs a semantic understanding layer for the large language model. The combination is as follows:

[0162] The initial text data is first fed into the pinyin fuzzy matching layer. If the pinyin fuzzy matching layer does not match (i.e., it is not a robot motion control command), the initial text data is then fed into the semantic understanding layer of the large language model for deep semantic parsing. Based on the semantics and context of the user input, the large language model determines that the user's intent belongs to one of the following categories and generates the corresponding response:

[0163] (a) Status query: such as "How much power is left?" or "Where are you?" The large language model calls the status query module to output the robot's current status through speech synthesis;

[0164] (b) Casual conversation: such as "What do you think of the current development of robots?" or "Tell me a joke", the large language model generates appropriate casual conversation responses;

[0165] (c)Semantic error correction: Correct homophones and wrongly written or misspelled words (such as "qianjin 5 mi") misrecognized due to accents. Correctly understand the instruction as "go forward five meters", and then generate a motion instruction. Through the collaborative error correction of the large language model and the fuzzy matching mechanism, the error tolerance rate of voice instructions can be further improved. In particular, it can automatically correct homophonic and near-sound errors. Even when the user's pronunciation is unclear or has an accent, and when the user outputs complex instructions, the instruction recognition success rate is guaranteed.

[0166] (d)Complex instructions: Such as "go to the kitchen to see what's there", the large language model decomposes the task and generates a sequence of motion instructions.

[0167] The pinyin fuzzy matching layer and the semantic understanding layer of the large language model form a two-layer architecture. This two-layer architecture enables most motion control instructions (with the highest proportion and most frequent calls) to go through the lightweight pinyin fuzzy matching channel, with low computational overhead. The large language model only processes tasks that require deep semantic understanding, avoiding calling the large language model every time there is a voice interaction, reducing computational resource investment and latency. In addition, for example, by constructing this two-layer architecture, the system can automatically identify the type of intention of the user input, route the three types of tasks of motion control, status query, and daily chat to the optimal processing channels respectively, so that the user does not need to learn a specific instruction format and can complete operations, obtain information, and have conversations using daily language.

[0168] It should be noted that the response text generated by the large language model is converted into voice output through the speech synthesis module to form a complete dialogue interaction. This is the conventional way to implement dialogue interaction in the robot. The large language model is an existing conventional AI large model, and there is no specific limitation. As long as it meets the above requirements.

[0169] To facilitate understanding of the specific usage method of the large language model, the following is continued to be supplemented:

[0170] By dynamically injecting the real-time status information of the robot into the system prompt words of the large language model, the large language model can accurately answer various natural language queries of the user regarding the robot's own status.

[0171] For the robot, its status information such as battery power and position coordinates is collected in real time, and then it is published through the Ros2 topic.

[0172] When speech-to-text recognition (i.e., the text data mentioned above) triggers the large language model, such as when a user inquires about the robot's status, the algorithm is invoked, and the robot's status information is subscribed to. The system can also construct contextual prompts (the system prompts can be adjusted according to the robot's role and functional scope). For example, if a user asks, "Xiao Zhi, how much battery do you have left?", "Xiao Zhi" serves as the prompt, triggering the robot's understanding process to convert the information into text data. At this point, the large language model can be directly invoked. The robot's status can be queried using natural language; this is a standard setting. For instance, through a prompting engineering method that dynamically injects status information into the large language model, users can directly query the robot's real-time status using natural language, without needing to view it through an app or webpage, thus lowering the barrier to human-computer interaction.

[0173] Understandably, using a large language model as an example, if a large language model receives a complex instruction but cannot fully parse it, thus preventing the robot from executing the instruction, it is still possible to synthesize a response voice. This voice can then be used to allow the user to issue the instruction again, for example, generating a response voice like "I didn't hear you clearly, please say it again."

[0174] In one embodiment of this application, in step two, audio data is recognized based on an offline speech recognition algorithm to obtain recognized text, and then the recognized text is converted based on a text conversion mechanism to obtain initial text data; the text conversion mechanism includes:

[0175] Convert the recognized Chinese uppercase numerals into Arabic numerals;

[0176] Convert the recognized Chinese numerals with decimal points into Arabic numerals with decimal points;

[0177] Convert the identified two or two into 2.

[0178] The text conversion mechanism can be implemented using existing parsing algorithms. These algorithms employ regular expression matching and recursive Chinese numeric parsing functions, capable of parsing integers and decimals with one decimal place within the range of 0 to 999, meeting the accuracy requirements of robot motion control scenarios. While parsing algorithms can also achieve text conversion using methods such as for loops, the approach using regular expressions and recursive Chinese numeric parsing functions is more efficient.

[0179] Examples are given below:

[0180] The system recognizes the numbers five and one hundred and eighty (which are Chinese capital numerals) in the text and automatically converts them into 5 and 180 (which are Arabic numerals) based on the text conversion mechanism.

[0181] Identify that the text includes 一点五 and 三点五 (Chinese numbers with decimal points), and automatically convert them to 1.5 and 3.5 (Arabic numbers with decimal points) based on the text conversion mechanism.

[0182] It should be noted that the use of the offline speech recognition algorithm can significantly reduce the device's dependence on the network, and the speech conversion can be completed without being connected to the network. It can still run stably in scenarios with no network or weak network. At the same time, the speech data is processed locally throughout the process, which can effectively avoid the privacy and security risks brought by cloud transmission, and can also reduce the network interaction delay and improve the instruction response efficiency. However, there are also certain problems. Specifically, the core function of offline speech recognition is to convert human speech into text, rather than text recognition. Limited by the model volume and computing power of the embedded terminal, most lightweight offline solutions have trimmed the post-processing module for digital normalization in order to achieve lightweight deployment. At this time, if the user says "前进五十点二厘米", the algorithm can only output the Chinese characters "前进五十点二厘米" and cannot automatically convert them into the Arabic number "前进50.2厘米", resulting in the robot being difficult to parse the exact value. The text conversion mechanism can effectively solve this problem and achieve the automatic conversion of Chinese numbers and decimal expressions into standard Arabic numbers, outputting a normalized value that can be directly parsed by the robot, ensuring the accurate recognition and smooth execution of quantitative instructions.

[0183] The offline speech recognition model is combined with the local large language model to form a hierarchical architecture, enabling the robot to still complete the execution of precise motion instructions and voice feedback in a network-free environment, meeting the requirements of the robot working in a network-free environment in reality.

[0184] In some embodiments, for the final text data (which has been updated), the offline speech synthesis algorithm is used to convert it into speech data to facilitate the robot's response to the user and improve the degree of human-computer interaction. The offline speech synthesis algorithm can also get rid of the network signal and bandwidth limitations and can normally complete the operation of text-to-speech output in a network-free scenario, adapting to the usage requirements of various mobile operations and closed scenarios. At the same time, the speech content is synthesized locally, avoiding data transmission outside and improving the security of interactive information, while conforming to the low-power and lightweight deployment characteristics of the embedded terminal.

[0185] It should be noted that the offline speech synthesis algorithm has been lightweight and streamlined to adapt to the hardware performance limitations, and generally lacks the text digital preprocessing logic. Its conversion ability is poor when dealing with Arabic numbers and decimal points, and there may even be problems where the decimal point cannot be recognized. For example, when synthesizing "3.5米", the user will hear "好的,前进三五米", resulting in a rigid pronunciation and incorrect semantic expression, affecting the interaction experience and instruction understanding. Based on this, before using the offline speech synthesis algorithm, the final text data is adjusted, including:

[0186] Determine whether the moving distance is ≥ 1 meter. If so, convert the corresponding Arabic numeral to a Chinese capital numeral, and convert the decimal point therein to the Chinese numeral "point". For example, convert "1 meter" to "one meter", and when performing voice synthesis, it can directly synthesize "Okay, it has advanced one meter". Another example is to convert "1.5 meters" to "one point five meters", and when performing voice synthesis, it can directly synthesize "Okay, it has advanced one point five meters". If not, convert the corresponding moving unit to centimeters and convert the corresponding Arabic numeral to a Chinese capital numeral. For example, convert "0.5 meters" to "50 centimeters", and when performing voice synthesis, it can directly synthesize "Okay, it has advanced fifty centimeters". Through the coordinated cooperation of the above text conversion mechanism and the conversion mechanism here, users can issue instructions in the most natural Chinese way (such as "advance one point five meters"), and at the same time, the robot can also give feedback in the same natural Chinese (such as "Okay, advance one point five meters"), greatly improving the naturalness of voice interaction and the user experience.

[0187] In summary, when the voice recognition method in this embodiment is applied to the robot interaction scenario, the robot can not only accurately respond to motion control instructions, but also conduct natural language chat conversations and status inquiries. At the same time, it can significantly reduce the impact of voice recognition errors on the instruction parsing success rate, and can still achieve intelligent scheduling of multiple types of tasks in an offline environment.

[0188] The following further describes the voice recognition method in this embodiment in detail in combination with a specific application embodiment (for action instructions).

[0189] Implementation environment: An intelligent inspection robot equipped with an NVIDIA Jetson Orin NX embedded platform, with an offline lightweight large language model DeepSeekR1 7b locally deployed, running on the Linux system, and equipped with the ROS2 Humble system.

[0190] Subscribe to odometer information, with the topic name / odom; subscribe to imu information, with the topic name / imu; subscribe to robot status information, with the topic name / robot_status.

[0191] Load the offline voice recognition algorithm Sherpa-onnx and the offline voice synthesis algorithm piper-tts.

[0192] The user issues an instruction, the microphone collects the audio stream in real time, and the VAD algorithm determines in real time whether the user's instruction has ended.

[0193] The Sherpa-onnx algorithm recognizes the user's instruction and converts it into Chinese text for the following processing:

[0194] (1) Send the recognized Chinese text into the pinyin fuzzy matching algorithm. The algorithm first performs an exact search in the custom error correction table. If a match is found, directly obtain the corresponding Chinese instruction text;

[0195] (2) If no match is found, convert the instruction text into a pinyin string through the pypinyin library and match it with the standard pinyin strings of each preset instruction;

[0196] (3) The matching algorithm first checks whether the instruction pinyin starts with a certain standard pinyin string. If so, the match is successful; if not, calculate the sequence similarity, and when the similarity ≥ 0.6, the match is determined to be successful;

[0197] (4) If the match is successful, obtain the corresponding action identifier, and the system determines that the current is an exact motion control task;

[0198] (5) Enter the motion parameter parsing sub - process: Extract the distance or angle value from the instruction text; For distance extraction, first match the regular expression "([零一二三四五六七八九十百千万两]+)点([零一二三四五六七八九])\s*米" (Chinese decimal), then match "([零一二三四五六七八九十百千万两]+)\s*米" (Chinese integer), and finally match "(\d+(?:\.\d+)?)\s*米" (Arabic numerals); Angle extraction adopts a similar strategy;

[0199] (6) If no numerical parameter is extracted, use the default values (default distance is 0.5 meters, default angle is 90 degrees) combined with the action identifier to generate motion target parameters, set the timeout time and tolerance range, start the motion control task, and give a confirmation feedback through the speech synthesis module; If no numerical parameter is extracted, the user can also be asked to repeat the instruction;

[0200] (7) If the pinyin fuzzy matching layer fails to match, trigger the large - language model call process;

[0201] (8) The system constructs a system prompt word containing the real - time state information of the robot. The state information is obtained by the state acquisition module from the ROS2 node, including battery power, current position, motion state, etc.;

[0202] (9) Use the user instruction text as the user message and send it to the large - language model together with the system prompt word;

[0203] (10) The large - language model generates a reply based on the user input and system context. The system analyzes the reply content. If it contains a motion control intention (identified by keywords or structured output markers), further parse the motion parameters and hand them over to the motion control execution module for processing;

[0204] (11) The response text (pure dialogue content) generated by the large language model is fed back through the speech synthesis module; after the system generates the feedback text, it is converted into speech output through piper-tts, and before the output, the system converts the numerical values ​​in the feedback text into colloquial Chinese.

[0205] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A speech recognition method based on fuzzy matching mechanism, characterized in that: include: Step 1: Acquire the audio stream to obtain audio data; Step 2: Recognize the audio data using a speech recognition algorithm to obtain the initial text data; Step 3: Update the initial text data based on the fuzzy matching mechanism, including: Create a custom error correction table, and use this custom error correction table to map preset error command texts to preset command texts; After obtaining the initial text data, the preset error instruction text in the initial text data is updated to the corresponding preset instruction text through a custom error correction table.

2. The speech recognition method based on fuzzy matching mechanism according to claim 1, characterized in that: In step one, after obtaining the audio data, the audio data is preprocessed. The preprocessing steps include resampling, noise reduction, and audio feature extraction.

3. The speech recognition method based on fuzzy matching mechanism according to claim 1, characterized in that: In step one, audio streams are selectively acquired based on a speech segment determination mechanism. This speech segment determination mechanism includes: performing frame-by-frame detection on the audio stream; if the audio energy of consecutive A frames is detected to be ≥ a first threshold, then the starting frame of the consecutive frames is marked as the start endpoint of the speech segment; subsequently, if the audio energy of consecutive B frames is detected to be < a first threshold, then the starting frame of the consecutive frames is marked as the end endpoint of the corresponding speech segment, and then the speech segment is used as audio data; A and B are both positive integers.

4. The speech recognition method based on fuzzy matching mechanism according to claim 1, characterized in that: Fuzzy matching mechanisms also include: Establish an instruction mapping table, which includes preset instruction text and corresponding pinyin strings; Before or after using a custom error correction table, selectively update the current text data based on a prefix matching strategy, which includes: At least the pinyin string of the beginning of the current text data of a preset length is obtained, and the pinyin string is matched with the pinyin string in the instruction mapping table. If the similarity between the two is greater than the second threshold, the current text data is updated based on the corresponding preset instruction text.

5. The speech recognition method based on fuzzy matching mechanism according to claim 4, characterized in that: Fuzzy matching mechanisms also include: After adopting a custom error correction table, the current text data is selectively updated based on a similarity matching strategy. This similarity matching strategy includes: matching all the pinyin strings of the current text data with the pinyin strings in the instruction mapping table to calculate the similarity between the two. If the similarity is greater than the third threshold, the current text data is updated based on the corresponding preset instruction text.

6. The speech recognition method based on fuzzy matching mechanism according to claim 5, characterized in that: After obtaining the initial text data, it is first parsed. If the initial text data cannot be completely parsed, it is updated using a custom error correction table to obtain the first updated text data, which is then parsed. If the first updated text data cannot be completely parsed, it is updated using a prefix matching strategy to obtain the second updated text data, which is then parsed. If the second updated text data cannot be completely parsed, it is updated using a similarity matching strategy to obtain the third updated text data, which is then parsed. If the third updated text data cannot be completely parsed, it is updated based on a large language model to obtain the fourth updated text data, which is then parsed.

7. The speech recognition method based on fuzzy matching mechanism according to claim 6, characterized in that: Before using the custom error correction table, the length of the pinyin string of the initial text data is calculated. If the length is less than or equal to the fourth threshold, the custom error correction table is called. If the length is greater than the fourth threshold, the large language model is called directly.

8. The speech recognition method based on fuzzy matching mechanism according to claim 1, characterized in that: In step two, the audio data is recognized using an offline speech recognition algorithm to obtain the recognized text. Then, the recognized text is converted using a text conversion mechanism to obtain the initial text data. The text conversion mechanism includes: Convert the recognized Chinese uppercase numerals into Arabic numerals; Convert the recognized Chinese numerals with decimal points into Arabic numerals with decimal points; Convert the identified two or two into 2.

Images