A Multimodal Fusion-Based Enterprise Management Voice Interaction Method and System

Abstract

This application proposes a multimodal fusion-based enterprise management voice interaction method and system, relating to the field of voice interaction technology. The method includes receiving an operator's instruction containing voice signals and auxiliary visual information, parsing the core purpose, key operational information, and recognition confidence level of the instruction, and establishing an instruction context record; updating the corresponding instruction state in the instruction context record to a state to be corrected, and sending an operation pause instruction to the target device; receiving the instruction to be corrected corresponding to the state to be corrected to obtain the operator's subsequent voice input, parsing the subsequent voice input into supplementary correction information; obtaining the specific operational information to be corrected based on the supplementary correction information and the key operational information of the instruction to be corrected; updating the corresponding key operational information in the instruction context record based on the specific operational information; and submitting the updated instruction to the target device for execution. This application can improve error correction efficiency and recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, and more specifically, to a voice interaction method and system for enterprise management based on multimodal fusion. Background Technology

[0002] In modern enterprise management, voice interaction technology is widely used for interaction between employees and business systems such as office, production scheduling, and customer management to improve operational efficiency and convenience. However, traditional voice recognition technology faces challenges in complex enterprise environments, such as environmental noise, multilingual mixing, and complex cross-system operations, which can easily lead to misinterpretation of instructions, affecting system reliability and user experience. To address this, the industry is exploring multimodal fusion technology that combines voice and vision to improve the accuracy of instruction recognition in complex environments. However, in industrial production workshops with high timeliness and accuracy requirements, when encountering sudden, instantaneous, high-intensity, high-frequency noise interference, even when combining voice and visual information, the system's accuracy in recognizing key parameters such as numbers whose pronunciation is easily interfered with will still decrease significantly, leading to instruction execution errors. For example, when a workshop supervisor issues an instruction to adjust the cutting depth of a machine tool to 1.5 mm, high-frequency noise from the welding equipment interferes with the pronunciation of the numbers. Although the system confirms the operation purpose through lip movements, it mistakenly recognizes "1.5" as "7.5". More significantly, when operators use short, colloquial phrases to correct errors, existing systems lack the ability to remember dialogue context and perform correlation analysis, failing to understand the intention to correct the error and recognizing it as an invalid instruction, forcing the operator to repeat the complete instruction. This inefficient error correction method not only reduces work efficiency, but may also cause production accidents due to delays. Existing technologies urgently need targeted improvements. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a multimodal fusion-based enterprise management voice interaction method and system, which aims to solve the technical problems of existing enterprise management voice interaction systems, such as significantly reduced accuracy in identifying key parameters in complex environments, especially when facing instantaneous high-intensity high-frequency noise interference, and lack of dialogue context memory and correlation analysis capabilities for previous commands, resulting in low error correction efficiency and even potentially causing production accidents.

[0004] In a first aspect, embodiments of this application provide an enterprise management voice interaction method based on multimodal fusion, including: The system receives instructions from the operator that include voice signals and auxiliary visual information, and combines the voice signals and auxiliary visual information to parse out the core purpose and key operational information of the instructions, as well as to obtain the recognition confidence of the key operational information based on the voice signals. The auxiliary visual information includes the operator's lip movements and gestures. Establish an instruction context record, which includes the core purpose of the instruction, the key operation information, the identification confidence level of the key operation information, and the instruction status. When the confidence level of the identification of the key operation information is lower than the preset dynamic threshold, or in response to the operator's explicit correction expression, the corresponding instruction status in the instruction context record is updated to the state to be corrected, and an operation pause instruction is sent to the target device, which is the device controlled by the operator through the instruction; In the state to be corrected, the corresponding correction instruction is received to obtain the operator's subsequent voice input, and the subsequent voice input is parsed into supplementary correction information, which includes correction type and correction value; Based on the supplementary correction information and combined with the key operation information of the instruction to be corrected, the specific operation information that needs to be corrected is obtained; Update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and then submit the updated instruction to the target device for execution.

[0005] According to some embodiments of this application, the step of combining the voice signal and the auxiliary visual information to parse the core purpose and key operational information of the instruction, and obtaining the recognition confidence level of the key operational information based on the voice signal, includes: The voice signal is input into a preset edge computing unit, which includes a preprocessing module and a preset voice recognition module. The preprocessing module removes known background noise components from the speech signal and converts the speech signal after removing the background noise components into an acoustic feature sequence. The acoustic feature sequence is converted into text commands based on a preset speech recognition module. Based on the text instructions and the auxiliary visual information, the core purpose and key operational information of the instructions are parsed out, and the recognition confidence of the key operational information is obtained based on the text instructions.

[0006] According to some embodiments of this application, establishing the instruction context record includes: Obtain the operator's instruction information for a preset number of times, the instruction information including the most recently processed instruction information and the instruction information currently being processed; Based on the instruction information of the preset number of times, multiple instruction data packets are obtained. Each instruction data packet includes a unique identifier, the core purpose of the instruction, the identified key operation information, the identification confidence level of the key operation information, and the instruction status. An instruction context record is established based on multiple instruction data packets.

[0007] According to some embodiments of this application, obtaining the identification confidence level of the key operation information based on the text instruction includes: The confidence score of each word or phrase in the text instruction is obtained based on the text instruction. The average confidence score is obtained by averaging the confidence scores of each word or phrase. The identification confidence level of the key operation information is obtained based on the average confidence score.

[0008] According to some embodiments of this application, before the step of combining the voice signal and the auxiliary visual information to parse the core purpose and key operational information of the instruction, the method further includes: The image quality of the operator's mouth area video stream is obtained based on the auxiliary visual information, and the image quality includes brightness, contrast, sharpness, and whether there is occlusion. When the image quality is below a preset threshold or there is partial occlusion, the speech signal is enhanced.

[0009] According to some embodiments of this application, parsing the subsequent voice input into supplementary correction information includes: Based on the information in the subsequent speech input analyzed by the preset natural language understanding module, the correction value is obtained and the correction type of the subsequent speech input is determined. The natural language understanding module is used to perform colloquial recognition and processing on the information in the subsequent speech input. Supplementary correction information is obtained based on the correction value and the correction type.

[0010] According to some embodiments of this application, the step of obtaining specific operational information requiring correction based on the supplementary correction information and the key operational information of the instruction to be corrected includes: Based on the supplementary correction information, the key operation information of the instruction to be corrected is traversed to obtain the positioning instruction with the same correction type as the supplementary correction information, and the key parameter list of the positioning instruction is extracted. The supplementary correction information is matched with the list of key parameters to determine the target parameters to be corrected in order to obtain the specific operational information that needs to be corrected.

[0011] According to some embodiments of this application, the step of matching the supplementary correction information with the key parameter list to determine the target parameter to be corrected in order to obtain the specific operational information that needs to be corrected includes: When the key parameter list contains multiple numerical parameters, the parameter slots in the key parameter list with numerical parameter types are searched first. Based on the parameter slots and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. Furthermore, the semantic relevance is matched between the correction type of the supplementary correction information and the key parameter list to obtain a semantic relevance score. Based on the semantic relevance score and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. When the list of key parameters contains only one numerical parameter, the target parameter to be corrected is determined based on the numerical parameter and the correction value of the supplementary correction information to obtain the specific operation information that needs to be corrected.

[0012] According to some embodiments of this application, the step of updating the corresponding key operation information in the instruction context record based on the specific operation information to obtain the updated instruction includes: Extract key operation fields from the specific operation information; The operation key field is matched and replaced with the corresponding key operation information in the instruction context record to obtain the updated instruction.

[0013] Secondly, this application also discloses an enterprise management voice interaction system based on multimodal fusion, comprising: The receiving and parsing module is used to receive the operator's instructions containing voice signals and auxiliary visual information, and to parse the core purpose and key operation information of the instructions by combining the voice signals and the auxiliary visual information, and to obtain the recognition confidence of the key operation information based on the voice signals, wherein the auxiliary visual information includes the operator's lip movements and gestures; The context recording module is used to establish an instruction context record, which includes the core purpose of the instruction, the key operation information, the identification confidence level of the key operation information, and the instruction status. The status update module is used to update the corresponding instruction status in the instruction context record to a state to be corrected when the recognition confidence of the key operation information is lower than a preset dynamic threshold, or in response to the operator's explicit correction expression, and to send an operation pause instruction to the target device, wherein the target device is the device controlled by the operator through the instruction; The correction information parsing module is used to receive the correction instruction corresponding to the correction state in the correction state to obtain the operator's subsequent voice input, and parse the subsequent voice input into supplementary correction information, wherein the supplementary correction information includes correction type and correction value; The specific operation information determination module is used to obtain the specific operation information that needs to be corrected based on the supplementary correction information and the key operation information of the instruction to be corrected. The update and verification module is used to update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and submit the updated instruction to the target device for execution.

[0014] The technical solution according to the embodiments of this application has at least the following beneficial effects: The enterprise management voice interaction method based on multimodal fusion disclosed in this application receives instructions from operators containing voice signals and auxiliary visual information, and combines the two to parse the core purpose and key operation information of the instructions. Simultaneously, it obtains the recognition confidence level of the key operation information based on the voice signal, effectively solving the problem of low accuracy in key parameter recognition of traditional voice recognition technology in complex enterprise environments, such as under high-frequency noise interference. This method further establishes an instruction context record, including the core purpose of the instruction, key operation information, recognition confidence level, and instruction status. Based on this, when the recognition confidence level is lower than a preset dynamic threshold or the operator explicitly corrects it, the instruction status is updated to a state of pending correction, and the operation of the target device is paused. Subsequently, in the state of pending correction, the system receives subsequent voice input from the operator as supplementary correction information, and obtains the specific operation information that needs to be corrected based on this information and the key operation information of the instruction to be corrected. Finally, the system updates the key operation information in the instruction context record according to the specific operation information, obtains the updated instruction, verifies it, and submits it to the target device for execution. Through the above technical solution, this application overcomes the problem of low error correction efficiency caused by the lack of dialogue context memory and correlation analysis capabilities in existing technologies. Specifically, this method can intelligently understand the operator's colloquial and brief corrective expressions that omit subjects, verbs, and objects, avoiding the tedious process of the operator repeating the complete instruction, and significantly improving the convenience and efficiency of error correction. This application not only greatly improves the accuracy of instruction recognition and error correction efficiency of enterprise management voice interaction systems in complex environments, but also effectively reduces the risk of production accidents that may be caused by error correction delays, significantly improving user experience and system reliability.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0017] Figure 1A flowchart illustrating an enterprise management voice interaction method based on multimodal fusion provided in one embodiment of this application; Figure 2 This is a schematic diagram of an enterprise management voice interaction system based on multimodal fusion, provided as an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical methods, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that the meaning of "multiple" (or "more than") in the description of the embodiments of this application refers to two or more, and "greater than," "less than," "exceeding," etc. are understood to exclude the number itself, while "above," "below," "within," etc. are understood to include the number itself. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0020] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: the existence of a alone, the existence of b alone, the existence of c alone, the simultaneous existence of a and b, the simultaneous existence of a and c, the simultaneous existence of b and c, or the simultaneous existence of a, b, and c, where a, b, and c can be single or multiple.

[0021] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0022] The enterprise management voice interaction method based on multimodal fusion provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the enterprise management voice interaction method based on multimodal fusion, etc., but is not limited to the above forms.

[0023] This application can be applied to numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices. It should be noted that in various specific embodiments of this invention, when processing is required based on data related to the characteristics of an object (e.g., user attributes or sets of attribute information), permission or consent from the corresponding object is obtained first, and the collection, use, and processing of this data comply with relevant laws and standards. Furthermore, when the embodiments of the present invention need to obtain the attribute information of an object, they will obtain the separate permission or separate consent of the corresponding object through pop-up windows or redirection to a confirmation page. After obtaining the separate permission or separate consent of the corresponding object, they will then obtain the relevant data of the object necessary for the embodiments of the present invention to operate normally.

[0024] See Figure 1 , Figure 1This is a flowchart illustrating an enterprise management voice interaction method based on multimodal fusion, provided in one embodiment of this application. The enterprise management voice interaction method based on multimodal fusion provided in this embodiment includes, but is not limited to, steps S110 to S160, which are described in detail below.

[0025] Step S110: Receive the operator's instruction containing voice signals and auxiliary visual information, and combine the voice signals and auxiliary visual information to parse out the core purpose and key operation information of the instruction, as well as obtain the recognition confidence of the key operation information based on the voice signals. The auxiliary visual information includes the operator's lip movements and gestures. Step S120: Establish an instruction context record. The instruction context record includes the core purpose of the instruction, key operation information, the identification confidence level of the key operation information, and the instruction status. Step S130: When the confidence level of the key operation information identification is lower than the preset dynamic threshold, or in response to the operator's explicit correction expression, the corresponding instruction status in the instruction context record is updated to the state to be corrected, and an operation pause instruction is sent to the target device, which is the device controlled by the operator through the instruction. Step S140: In the state to be corrected, receive the correction instruction corresponding to the state to be corrected to obtain the operator's subsequent voice input, and parse the subsequent voice input into supplementary correction information, which includes correction type and correction value. Step S150: Based on the supplementary correction information and the key operation information of the instruction to be corrected, obtain the specific operation information that needs to be corrected. Step S160: Update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and then submit the updated instruction to the target device for execution.

[0026] It should be noted that: voice signal refers to the acoustic information containing instructions issued by the operator; auxiliary visual information refers to visual data acquired synchronously with the operator's voice instructions, such as the operator's lip movements and gestures, which helps assist in voice recognition and understanding; instruction context record refers to a data structure maintained by the system for each instruction, which includes the core purpose of the instruction, key operational information, recognition confidence level, and current processing status, used to track and manage the lifecycle of the instruction; recognition confidence level refers to the system's evaluation score of the accuracy of the recognized key operational information; preset dynamic threshold refers to a confidence level threshold that can be dynamically adjusted according to the environment or task; if it is lower than this threshold, the recognition result is considered unreliable; target device refers to the actual device or system controlled by the operator through voice instructions; supplementary correction information refers to the information provided by the operator to correct errors after instruction recognition errors, including the type and specific value of the correction. The implementation environment of this application is typically an enterprise management scenario, such as an industrial production workshop, a smart office environment, or a customer service center, where operators interact with various business systems through voice.

[0027] In one embodiment, the system first receives an instruction from the operator that includes voice signals and auxiliary visual information. For example, the operator might issue a voice instruction to adjust the cutting depth of machine tool number 5 to 1.5 mm, while the system simultaneously captures the operator's lip movements and gestures via a camera. The system combines these voice signals and auxiliary visual information to parse the instruction, identifying its core purpose (adjusting the cutting depth) and key operational information (e.g., "machine tool number 5," "1.5 mm"). Simultaneously, the system also assesses the recognition confidence level of the key operational information (especially the number 1.5) based on the voice signal. Auxiliary visual information, such as lip movements and gestures, can supplement voice recognition; for example, when the voice signal is subject to noise interference, lip movement information can help the system more accurately determine pronunciation. Next, the system establishes an instruction context record. This record is a dynamically updated data structure used to store relevant information for current and recent instructions. For example, for the aforementioned instruction to adjust the cutting depth, the instruction context record would include its core purpose, the identified key operational information (machine tool number 5, 1.5 mm), the recognition confidence levels of this information, and the current state of the instruction (pending execution). The establishment of an instruction context record allows the system to "remember" previous instruction content and status. Subsequently, the system determines whether correction is needed based on the recognition confidence level of key operational information or the operator's explicit correction expression. For example, if the system's recognition confidence level for 1.5 mm is lower than a preset dynamic threshold, or if the operator directly expresses an explicit correction such as an incorrect instruction, the system updates the corresponding instruction status in the instruction context record to "pending correction." Simultaneously, the system sends an operation pause command to the target device (e.g., machine tool number 5) to prevent the execution of erroneous instructions, thereby avoiding potential production accidents or operational errors. While the instruction is in the pending correction state, the system receives subsequent voice input from the operator as supplementary correction information for the current instruction. For example, the operator might say, "It's 1.5, not 7.5." The system parses this subsequent voice input into supplementary correction information, including the correction type (e.g., "numerical correction") and the correction value (e.g., "1.5"). This conversational correction method allows operators to correct errors in a more natural and efficient way. Then, based on the received supplementary correction information and the key operational information of the instruction to be corrected, the system determines the specific operational information that needs correction. For example, based on the supplementary correction information of "numerical correction" and "1.5", the system locates the parameter that needs correction, "depth of cut", in the key operation information of the instruction to be corrected ("machine tool No. 5", "1.5 mm"), and determines its correction value to be "1.5". Finally, the system updates the corresponding key operation information in the instruction context record according to the determined specific operation information, thereby obtaining the updated instruction. For example, the value of "depth of cut" is corrected from the incorrect "7.5 mm" to the correct "1.5 mm".After the update, the system verifies the updated instructions to ensure their logical correctness and security. Once verification is successful, the system submits them to the target device for execution.

[0028] This application constructs a more robust, efficient, and user-friendly enterprise management voice interaction method by organically combining multimodal fusion, instruction context recording, and intelligent correction mechanisms. It effectively solves the pain points of low recognition accuracy and poor error correction efficiency of existing technologies in complex environments, bringing significant efficiency improvements and risk reductions to enterprise management.

[0029] Specifically, in some implementations of the above method, the process of parsing the core purpose and key operational information of the instruction by combining speech signals and auxiliary visual information, and obtaining the recognition confidence level of the key operational information based on the speech signals, can be further refined and implemented. This process includes: The voice signal is input into a preset edge computing unit, which includes a preprocessing module and a preset voice recognition module. The preprocessing module removes known background noise components from the speech signal and converts the speech signal after removing background noise components into an acoustic feature sequence. The acoustic feature sequence is converted into text commands based on the preset speech recognition module; Based on text instructions and auxiliary visual information, the core purpose and key operational information of the instructions are analyzed, as well as the recognition confidence of the key operational information obtained from the text instructions.

[0030] The pre-configured edge computing unit is configured to process data close to the data source to reduce data transmission latency and improve response speed. This unit integrates a preprocessing module and a pre-configured speech recognition module, designed to efficiently process the raw speech signal. Specifically, the preprocessing module performs initial purification of the received speech signal. Its main task is to identify and remove known background noise components from the speech signal, such as environmental noise and equipment operating noise. Noise removal significantly improves the quality of the speech signal, providing a cleaner input for subsequent speech recognition. After removing background noise components, the speech signal is converted into a series of acoustic feature sequences. These acoustic feature sequences are abstract representations of the speech signal in the time and frequency domains, such as Mel-frequency cepstral coefficients (MFCCs), which effectively capture the essential information of the speech. Further, the pre-configured speech recognition module is used to convert the aforementioned acoustic feature sequences into understandable text instructions. This module is typically built based on deep learning models or traditional acoustic models, capable of mapping continuous acoustic features into discrete text sequences. Finally, after obtaining the text instructions, the core purpose and key operational information of the instructions can be analyzed by combining the operator's auxiliary visual information, such as lip movements and gestures. Simultaneously, the recognition confidence level of the key operational information can be obtained based on the text instructions. This multimodal fusion approach effectively improves the accuracy and robustness of instruction parsing. The solution in this application achieves localized and rapid processing of raw speech data by first inputting the speech signal into a preset edge computing unit for processing. The preprocessing module removes background noise and extracts acoustic feature sequences, providing high-quality input to the speech recognition module and ensuring the accuracy of speech recognition. Subsequently, the preset speech recognition module converts the acoustic feature sequences into text instructions, digitizing the speech content. Finally, combining the text instructions and auxiliary visual information for multimodal parsing allows for a more comprehensive and accurate understanding of the operator's intent and the calculation of the recognition confidence level of key operational information, providing a reliable foundation for subsequent instruction processing and correction mechanisms.

[0031] Specifically, in the above method, the steps for establishing the instruction context record include: Obtain the operator's instruction information for a preset number of times. The instruction information includes the processed instructions for the most recent preset number of times and the instructions currently being processed. Based on the instruction information of a preset number of times, multiple instruction data packets are obtained. Each instruction data packet includes a unique identifier, the core purpose of the instruction, the identified key operation information, the identification confidence level of the key operation information, and the instruction status. An instruction context record is created based on multiple instruction data packets.

[0032] The preset number of instruction information refers to the historical instruction data of operators stored and utilized by the system to maintain contextual coherence. This instruction information includes not only the currently processed instruction but also the instruction information processed within the most recent preset number of times. By acquiring this historical and current instruction information, the system can form a more comprehensive understanding of the operator's intentions and operating habits. Furthermore, the instruction data packet is a structured encapsulation unit for each instruction. Each instruction data packet contains a unique identifier for tracking and managing instructions within the system. In addition, the instruction data packet also includes the core purpose of the instruction, i.e., the fundamental intention of the operator issuing the instruction; identified key operational information, i.e., the specific actions or parameters parsed from the operator's instruction; the confidence level of the identification of key operational information, which quantifies the system's accuracy assessment of the identified key operational information; and the instruction status, indicating the current stage of the instruction in the processing flow, such as "processed," "processing," or "pending correction." Thus, by integrating multiple instruction data packets, the system can establish a dynamic and comprehensive instruction context record. This record not only reflects the detailed information of the current instruction but also includes the trajectory of historical operations, providing rich background information for subsequent instruction understanding, correction, and execution. The solution in this application obtains instruction information a preset number of times and structures it into an instruction data packet containing a unique identifier, core purpose, key operation information, identification confidence level and instruction status, thereby establishing an instruction context record.

[0033] In some embodiments of this application described above, the confidence level for identifying key operation information obtained from text instructions can be specifically implemented in the following ways. The confidence level for identifying key operation information obtained from text instructions includes: The confidence score of each word or phrase in the text instruction is obtained based on the text instruction; the average confidence score is calculated by averaging the confidence scores of each word or phrase. The confidence level for identifying key operational information is obtained based on the average confidence score.

[0034] Specifically, in the process of converting acoustic feature sequences into text commands, the pre-defined speech recognition module typically provides a confidence score for each word or phrase in the text command while outputting the text command. This confidence score reflects the speech recognition system's assessment of the accuracy of recognizing that word or phrase. For example, the confidence score of a word might be a value between 0 and 1, with higher values ​​indicating higher recognition accuracy. The average confidence score is calculated by averaging the confidence scores of each word or phrase that constitutes the key operation information. This involves summing the confidence scores of each word or phrase and dividing by the total number of words or phrases to obtain a comprehensive confidence index. For example, if the key operation information consists of three words with confidence scores of 0.8, 0.9, and 0.7, the average confidence score can be calculated as (0.8 + 0.9 + 0.7) / 3. Obtaining the recognition confidence of the key operation information based on the average confidence score can be understood as directly using the calculated average confidence score as the recognition confidence of the key operation information. The aim is to provide a more comprehensive and reliable overall confidence assessment by comprehensively considering the recognition accuracy of each component of the key operational information. This application's solution calculates the average confidence score of each word or phrase in the text instruction, enabling a more detailed evaluation of the recognition reliability of key operational information. Traditional speech recognition systems may only provide the confidence score of the entire instruction, which may not accurately reflect the recognition quality of the key parts of the instruction. By individually assessing the confidence score of each word or phrase constituting the key operational information and calculating the average, it is possible to effectively avoid the impact of recognition errors in non-key parts on the judgment of the confidence score of the key operational information, thereby making the assessment of recognition confidence more accurate.

[0035] To address this, this application further proposes a scheme that, before combining speech signals and auxiliary visual information to parse instructions, performs a quality assessment of the auxiliary visual information and enhances the speech signal based on the assessment results, thereby ensuring the effectiveness of multimodal fusion parsing. Before parsing the core purpose and key operational information of the instructions by combining speech signals and auxiliary visual information, the process also includes: The image quality of the operator's mouth area video stream is obtained based on auxiliary visual information. The image quality includes brightness, contrast, sharpness, and whether there is occlusion. When the image quality is below a preset threshold or there is partial occlusion, the speech signal is enhanced.

[0036] Specifically, auxiliary visual information refers to the operator's lip movements and gestures, acquired through visual sensors such as cameras. Before instruction parsing, the system first performs image quality assessment on the video stream of the operator's mouth area extracted from the auxiliary visual information. Image quality assessment dimensions can include brightness, contrast, sharpness, and the presence of occlusion. Brightness refers to the overall brightness of the image; contrast refers to the grayscale difference between different areas of the image; sharpness refers to the sharpness of image details; and the presence of occlusion determines whether the mouth area is obscured by other objects (such as hands, masks, etc.). These image quality parameters can be calculated and analyzed using image processing algorithms. Brightness can be obtained by calculating the average grayscale value of the image pixels; contrast can be assessed by calculating the standard deviation or histogram distribution of the image pixel grayscale values; sharpness can be measured by calculating the high-frequency components of the image (such as the Laplacian operator, Sobbel operator, etc.); and the presence of occlusion can be identified using a pre-trained object detection model or a deep learning-based semantic segmentation model. A preset threshold is used as a reference standard to determine whether the image quality is acceptable; it can be set according to the actual application scenario and system performance requirements. For example, a lower limit can be set for brightness, contrast, and sharpness, as well as an upper limit for occlusion area. When any of the evaluated image quality parameters falls below a preset threshold, or when there is partial occlusion in the mouth area, it indicates that the auxiliary visual information may be insufficient to provide high-quality lip-reading information. In this case, to compensate for the lack of visual information, the system will enhance the speech signal. Speech signal enhancement can employ various techniques, such as noise reduction, echo cancellation, and speech enhancement algorithms (e.g., spectral subtraction, Wiener filtering, deep learning-based speech enhancement), to improve the signal-to-noise ratio and intelligibility of the speech signal, thereby providing clearer and more reliable speech input for subsequent speech recognition and multimodal fusion parsing.

[0037] In one embodiment, the present application's solution effectively addresses the problem of decreased instruction parsing accuracy due to poor visual information quality by introducing a quality assessment mechanism for auxiliary visual information before multimodal fusion parsing. Specifically, when the image quality (such as brightness, contrast, and sharpness) of the video stream in the operator's mouth area is below a preset threshold, or when there is partial occlusion in the mouth area, the system can promptly identify potential defects in the visual information. Because visual information may not provide effective supplementation in multimodal fusion, the system can specifically enhance the speech signal. By enhancing the speech signal, its signal-to-noise ratio and sharpness are significantly improved, enabling the subsequent speech recognition module to extract key information from higher-quality speech input, ensuring the accuracy of speech channel recognition even when visual information is impaired. This preprocessing mechanism ensures that the multimodal fusion parsing process always relies on at least one high-quality input modality, thereby maintaining the robustness and accuracy of overall instruction parsing. Through the above technical solution, the present application can effectively address the impact of complex and changing environmental factors on the performance of multimodal interaction systems. When auxiliary visual information is of poor quality or obstructed, the system no longer blindly performs multimodal fusion. Instead, it significantly improves the quality of the speech signal through intelligent evaluation and targeted speech enhancement processing. This allows the system to more accurately recognize operator voice commands even in scenarios with limited visual information, thus avoiding command misunderstandings and decreased recognition confidence caused by visual information quality issues. Therefore, this application improves the robustness and adaptability of enterprise management voice interaction methods, reduces the frequency of command correction, and enhances operational efficiency and user experience.

[0038] In one embodiment, assuming the operator issues a voice command to the enterprise management system in a dimly lit environment or with part of their face obscured by their hand, the system acquires a video stream of the operator's mouth area via a camera. Before parsing the video stream along with auxiliary visual information, the system first performs image quality analysis. For example, if the system detects that the average brightness of the mouth area video stream is below a preset brightness threshold, or if an image recognition algorithm detects that the operator's arm is partially obscuring the mouth area, the system will determine that the auxiliary visual information has poor image quality or partial occlusion. To compensate for this deficiency, the system automatically activates a voice signal enhancement module to denoise and enhance the operator's voice signal, for example, by using a deep learning model to dederarize and denoise the speech. The enhanced voice signal has significantly improved clarity and intelligibility, and is then input into the multimodal fusion parsing module along with the imperfect but still usable auxiliary visual information. Even if the lip-reading information provided by the visual information is incomplete, the enhanced voice signal ensures the accuracy of speech recognition, enabling the system to correctly parse the core purpose and key operational information of the operator's command, such as querying inventory quantities. In this way, this application ensures that the system can maintain a high level of instruction recognition accuracy even under adverse visual conditions.

[0039] Specifically, the subsequent voice input will be parsed into supplementary correction information, including: The information in the subsequent speech input is analyzed based on the preset natural language understanding module to obtain the correction value and determine the correction type of the subsequent speech input. The natural language understanding module is used to perform spoken language recognition and processing on the information in the subsequent speech input. Supplementary correction information is obtained based on the correction value and correction type.

[0040] The preset natural language understanding module can be understood as a software or hardware component specifically designed to process human natural language. This module is configured to perform in-depth analysis of the operator's subsequent voice input to identify semantic information. Specifically, its core function is to recognize and process spoken language input, meaning it can not only convert speech into text but also understand the intent, entities, and relationships behind the text. For example, when the operator says "change the quantity to ten," the natural language understanding module can identify "quantity" as the correction type and "ten" as the correction value. Further, the correction value refers to the specific numerical value, text, or other information provided by the operator in subsequent voice input, used to replace or modify a parameter in the instruction to be corrected. The correction type refers to the category of correction the operator intends to make, such as corrections targeting different dimensions like quantity, time, location, or object. Through the parsing of the natural language understanding module, this key information can be accurately extracted from subsequent voice input, thereby constructing complete supplementary correction information. The solution in this application, by introducing a preset natural language understanding module, enables the system to efficiently and accurately parse the operator's subsequent voice input in the state to be corrected. When an operator expresses their intention to make corrections via voice, this module can recognize and process the spoken language, thereby accurately extracting the correction value and type. This mechanism ensures that the system can understand the operator's correction instructions and transform them into structured supplementary correction information, laying the foundation for subsequent determination of specific operational information and instruction updates.

[0041] In this regard, this application further proposes the following steps for obtaining the specific operational information requiring correction based on the supplementary correction information and the key operational information of the instruction to be corrected: Based on the supplementary correction information, the key operation information of the instruction to be corrected is traversed to obtain the positioning instruction with the same correction type as the supplementary correction information, and the key parameter list of the positioning instruction is extracted. The supplementary correction information is matched with the list of key parameters to determine the target parameters to be corrected in order to obtain the specific operational information that needs to be corrected.

[0042] Specifically, traversing the key operational information of the instruction to be corrected based on supplementary correction information refers to the system systematically checking and comparing the identified key operational information in the instruction currently in the correction state after receiving subsequent voice input from the operator and parsing it into supplementary correction information. This process aims to identify instructions or instruction fragments in the key operational information that match the correction type (e.g., numerical correction, text replacement, parameter addition / deletion, etc.) of the supplementary correction information, thereby narrowing down the correction scope and improving the accuracy of the correction. For example, if the correction type is an adjustment to a certain value, the system will prioritize searching for instruction portions in the key operational information that contain numerical parameters. Obtaining the positioning instruction that matches the correction type of the supplementary correction information and extracting the key parameter list of the positioning instruction can be understood as follows: during the traversal process, once an instruction fragment matching the correction type is identified, that fragment is determined to be the positioning instruction. Subsequently, the system will parse and extract all the key parameters that can be corrected from the positioning instruction, forming a key parameter list. This list contains all parameters in the positioning instruction that can potentially be corrected and their current values, laying the foundation for subsequent precise matching. For example, if the positioning command is "set the temperature to 25 degrees," the key parameter list might contain "temperature" and "25." Matching the supplementary correction information with the key parameter list determines the target parameter to be corrected, thus obtaining the specific operational information requiring correction. Specifically, the system compares the correction values ​​contained in the supplementary correction information with each parameter in the key parameter list. The matching process can be based on various strategies; for example, if the correction value is numerical, numerical parameters are matched first; if the correction value is text, semantic similarity matching is performed. Through precise matching, the system can uniquely or with high confidence determine which parameter in the key parameter list is the target parameter that the operator intends to correct. Thus, the target parameter and its correction value together constitute the specific operational information that needs correction. Through the above technical solution, this application can significantly improve the accuracy and efficiency of command correction in enterprise management voice interaction systems. Compared to the potential problems of unclear correction targets or cumbersome correction processes in basic solutions, this solution, through a structured traversal and matching mechanism, ensures that the supplementary correction information can accurately apply to the specific parameters in the command to be corrected, thereby avoiding misoperation and repeated confirmation, and improving the system's intelligence level. As a result, the operator's corrective intent can be understood and executed more accurately, greatly optimizing the user experience and reducing potential business risks caused by misunderstandings of instructions.

[0043] The above process involves matching the supplementary correction information with the list of key parameters to determine the target parameters to be corrected, thereby obtaining the specific operational information that needs to be corrected. This includes: When there are multiple numerical parameters in the key parameter list, the parameter slots with numerical parameters in the key parameter list are searched first. Based on the parameter slots and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. Also, based on the correction type of the supplementary correction information, the semantic relevance is matched with the key parameter list to obtain the semantic relevance score. Based on the semantic relevance score and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. When the list of key parameters contains only one numerical parameter, the target parameter to be corrected is determined based on the numerical parameter and the correction value of the supplementary correction information in order to obtain the specific operational information that needs to be corrected.

[0044] Specifically, "numerical parameters" refer to parameters in the key parameter list whose values ​​can be represented as numbers or quantities, such as quantity, price, time, and size. When multiple such numerical parameters exist in the key parameter list, this application adopts a hierarchical matching strategy to avoid ambiguity in the correction. First, a "parameter slot" can be understood as a predefined data structure or placeholder used to store parameter values ​​of a specific type. Prioritizing the search for parameter slots with numerical parameters means that the system will first focus on parameter positions that are explicitly designed to receive numerical values. For example, if an instruction is "change the order quantity to 5 and set the price to 100", then "quantity" and "price" are two numerical parameters, and the system will prioritize identifying their corresponding parameter slots. Based on this, combined with the correction value of the supplementary correction information, the target parameter to be corrected can be initially determined. For example, if the correction value is "5", and there is only one numerical parameter slot in the instruction that can accept "5", then the parameter corresponding to that parameter slot is the target parameter. Furthermore, to handle more complex correction scenarios, this application also introduces "semantic relevance matching". "Semantic relevance matching" refers to analyzing the semantic relevance between the correction type (e.g., "quantity correction," "price adjustment") of supplementary correction information and each parameter in the key parameter list using natural language processing technology. For example, word vectors, topic models, or deep learning models can be used to calculate the similarity score between the correction type and the parameter name, thus obtaining the "semantic relevance score." This score reflects the semantic closeness between the correction type and a certain parameter. By combining the semantic relevance score and the correction value, the target parameter to be corrected can be determined more accurately, even when the correction value itself may be ambiguous. When the key parameter list contains only one numerical parameter, the correction process is relatively simplified. In this case, since there is no ambiguity in the numerical parameter, the system can directly determine the target parameter to be corrected based on the unique numerical parameter and the correction value of the supplementary correction information. For example, if the instruction is "adjust the temperature to 25 degrees," and the key parameter list contains only one numerical parameter representing temperature, then the correction value "25" will directly apply to that temperature parameter. Through the above technical solutions, this application can significantly improve the accuracy and robustness of enterprise management voice interaction methods in handling instruction correction. Especially in complex scenarios with multiple numerical parameters in the key parameter list, by combining parameter slot recognition and semantic correlation matching, the system can more accurately locate the specific parameter that the operator intends to correct, effectively avoiding erroneous corrections or duplicate confirmations caused by parameter ambiguity, thereby improving correction efficiency and user experience. Furthermore, for scenarios with a single numerical parameter, the matching logic has been simplified, ensuring the timeliness and accuracy of corrections.

[0045] In one embodiment, suppose an operator issues an instruction to change the order quantity to 5 and the price to 100. After parsing, the system obtains key operation information, which includes two numerical parameters: quantity and price. At this time, the key parameter list in the instruction context record will contain these two parameters and their corresponding values. The operator discovers that the quantity is incorrect and issues a correction instruction to change the quantity to 8. At this time, the system receives a subsequent voice input to change the quantity to 8 and parses supplementary correction information, where the correction type is quantity correction and the correction value is 8. Since the key parameter list contains two numerical parameters, quantity and price, the system will prioritize searching for parameter slots with numerical parameter types. The system identifies the parameter slots corresponding to quantity and price. Based on the correction type quantity correction in the supplementary correction information, semantic correlation matching is performed with the key parameter list, and it is found that the semantic correlation score between quantity correction and the quantity parameter is the highest. Combined with the correction value of 8, the system finally determines that the target parameter to be corrected is quantity and corrects it to 8. The operator discovers that the price is incorrect and issues a correction instruction to adjust the price to 120. The system receives a subsequent voice input to adjust the price to 120 and parses supplementary correction information, where the correction type is price adjustment and the correction value is 120. Similarly, the system prioritizes searching for numerical parameter slots. It performs semantic correlation matching between the correction type (price adjustment) and the key parameter list, finding that price adjustment has the highest semantic correlation score with the price parameter. Combined with the correction value of 120, the system ultimately determines that the target parameter to be corrected is price and corrects it to 120. Suppose an operator issues a command to adjust the temperature to 25 degrees Celsius. After parsing, the system obtains key operation information, which contains only one numerical parameter: temperature. The operator then discovers the temperature is incorrect and issues a correction command to adjust the temperature to 30 degrees Celsius. The system receives subsequent voice input to adjust the temperature to 30 degrees Celsius and parses supplementary correction information, where the correction type is temperature adjustment and the correction value is 30. Since the key parameter list only contains one numerical parameter, temperature, the system directly determines that the target parameter to be corrected is temperature based on this numerical parameter and the correction value of 30 in the supplementary correction information, and corrects it to 30.

[0046] In this regard, this application further proposes the following steps for updating the corresponding key operation information in the instruction context record based on specific operation information to obtain the updated instruction: Extract key operation fields from specific operation information; The operation key fields are matched and replaced with the corresponding key operation information in the instruction context record to obtain the updated instruction.

[0047] Specifically, extracting key operation fields from specific operation information refers to identifying and separating the core data units used for the actual modification instructions from the revised and confirmed specific operation information. These key operation fields can be numerical, text, Boolean, or other data types, and they directly correspond to specific parameters or attributes that need to be modified in the instruction. For example, if the specific operation information is to set the temperature to 25 degrees, then 25 degrees or 25 is the key operation field. Matching and replacing the key operation fields with the corresponding key operation information in the instruction context record can be understood as the system searching for the corresponding key operation information that needs to be updated in the instruction context record based on the extracted key operation fields. Once a match is found, the old key operation information value is replaced with the new key operation field value. The purpose is to ensure that the key operation information in the instruction context record can be updated accurately and without errors, thereby forming a complete and accurate updated instruction.

[0048] See Figure 2 , Figure 2 A schematic diagram of an enterprise management voice interaction system based on multimodal fusion provided in one embodiment of this application. The enterprise management voice interaction system 200 based on multimodal fusion includes: The receiving and parsing module 210 is used to receive the operator's instructions containing voice signals and auxiliary visual information, and to parse out the core purpose and key operation information of the instructions by combining the voice signals and auxiliary visual information, as well as to obtain the recognition confidence of the key operation information based on the voice signals. The auxiliary visual information includes the operator's lip movements and gestures. The context recording module 220 is used to establish an instruction context record, which includes the core purpose of the instruction, key operation information, the identification confidence level of the key operation information, and the instruction status. The status update module 230 is used to update the corresponding instruction status in the instruction context record to the state to be corrected when the recognition confidence of key operation information is lower than a preset dynamic threshold, or in response to the operator's explicit correction expression, and send an operation pause instruction to the target device, which is the device controlled by the operator through the instruction; The correction information parsing module 240 is used to receive the correction instruction corresponding to the correction state when the correction state is to be corrected, obtain the operator's subsequent voice input, and parse the subsequent voice input into supplementary correction information, which includes correction type and correction value. The specific operation information determination module 250 is used to obtain the specific operation information that needs to be corrected based on the supplementary correction information and the key operation information of the instruction to be corrected. The update and verification module 260 is used to update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and then submit the updated instruction to the target device for execution.

[0049] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0050] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0051] The foregoing has provided a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined in this application.

Claims

1. A voice interaction method for enterprise management based on multimodal fusion, characterized in that, include: The system receives instructions from the operator that include voice signals and auxiliary visual information, and combines the voice signals and auxiliary visual information to parse out the core purpose and key operational information of the instructions, as well as to obtain the recognition confidence of the key operational information based on the voice signals. The auxiliary visual information includes the operator's lip movements and gestures. Establish an instruction context record, which includes the core purpose of the instruction, the key operation information, the identification confidence level of the key operation information, and the instruction status. When the confidence level of the identification of the key operation information is lower than the preset dynamic threshold, or in response to the operator's explicit correction expression, the corresponding instruction status in the instruction context record is updated to the state to be corrected, and an operation pause instruction is sent to the target device, which is the device controlled by the operator through the instruction; In the state to be corrected, the corresponding correction instruction is received to obtain the operator's subsequent voice input, and the subsequent voice input is parsed into supplementary correction information, which includes correction type and correction value; Based on the supplementary correction information and combined with the key operation information of the instruction to be corrected, the specific operation information that needs to be corrected is obtained; Update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and then submit the updated instruction to the target device for execution.

2. The method according to claim 1, characterized in that, The step of combining the voice signal and the auxiliary visual information to parse the core purpose and key operational information of the instruction, and obtaining the recognition confidence level of the key operational information based on the voice signal, includes: The voice signal is input into a preset edge computing unit, which includes a preprocessing module and a preset voice recognition module. The preprocessing module removes known background noise components from the speech signal and converts the speech signal after removing the background noise components into an acoustic feature sequence. The acoustic feature sequence is converted into text commands based on a preset speech recognition module. Based on the text instructions and the auxiliary visual information, the core purpose and key operational information of the instructions are parsed out, and the recognition confidence of the key operational information is obtained based on the text instructions.

3. The method according to claim 1, characterized in that, The establishment instruction context record includes: Obtain the operator's instruction information for a preset number of times, the instruction information including the most recently processed instruction information and the instruction information currently being processed; Based on the instruction information of the preset number of times, multiple instruction data packets are obtained. Each instruction data packet includes a unique identifier, the core purpose of the instruction, the identified key operation information, the identification confidence level of the key operation information, and the instruction status. An instruction context record is established based on multiple instruction data packets.

4. The method according to claim 2, characterized in that, The step of obtaining the identification confidence level of the key operation information based on the text instruction includes: The confidence score of each word or phrase in the text instruction is obtained based on the text instruction. The average confidence score is obtained by averaging the confidence scores of each word or phrase. The identification confidence level of the key operation information is obtained based on the average confidence score.

5. The method according to claim 1, characterized in that, Before combining the voice signal and the auxiliary visual information to parse the core purpose and key operational information of the instruction, the method further includes: The image quality of the operator's mouth area video stream is obtained based on the auxiliary visual information, and the image quality includes brightness, contrast, sharpness, and whether there is occlusion. When the image quality is below a preset threshold or there is partial occlusion, the speech signal is enhanced.

6. The method according to claim 1, characterized in that, The step of parsing the subsequent voice input into supplementary correction information includes: Based on the information in the subsequent speech input analyzed by the preset natural language understanding module, the correction value is obtained and the correction type of the subsequent speech input is determined. The natural language understanding module is used to perform colloquial recognition and processing on the information in the subsequent speech input. Supplementary correction information is obtained based on the correction value and the correction type.

7. The method according to claim 1, characterized in that, The step of obtaining specific operation information that needs to be corrected based on the supplementary correction information and the key operation information of the instruction to be corrected includes: Based on the supplementary correction information, the key operation information of the instruction to be corrected is traversed to obtain the positioning instruction with the same correction type as the supplementary correction information, and the key parameter list of the positioning instruction is extracted. The supplementary correction information is matched with the list of key parameters to determine the target parameters to be corrected in order to obtain the specific operational information that needs to be corrected.

8. The method according to claim 7, characterized in that, The process of matching the supplementary correction information with the key parameter list to determine the target parameter to be corrected in order to obtain the specific operational information that needs to be corrected includes: When the key parameter list contains multiple numerical parameters, the parameter slots in the key parameter list with numerical parameter types are searched first. Based on the parameter slots and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. Furthermore, the semantic relevance is matched between the correction type of the supplementary correction information and the key parameter list to obtain a semantic relevance score. Based on the semantic relevance score and the correction values ​​of the supplementary correction information, the target parameter to be corrected is determined to obtain the specific operation information that needs to be corrected. When the list of key parameters contains only one numerical parameter, the target parameter to be corrected is determined based on the numerical parameter and the correction value of the supplementary correction information to obtain the specific operation information that needs to be corrected.

9. The method according to claim 1, characterized in that, The step of updating the corresponding key operation information in the instruction context record according to the specific operation information to obtain the updated instruction includes: Extract key operation fields from the specific operation information; The operation key field is matched and replaced with the corresponding key operation information in the instruction context record to obtain the updated instruction.

10. A multimodal fusion-based enterprise management voice interaction system, characterized in that, include: The receiving and parsing module is used to receive the operator's instructions containing voice signals and auxiliary visual information, and to parse the core purpose and key operation information of the instructions by combining the voice signals and the auxiliary visual information, and to obtain the recognition confidence of the key operation information based on the voice signals, wherein the auxiliary visual information includes the operator's lip movements and gestures; The context recording module is used to establish an instruction context record, which includes the core purpose of the instruction, the key operation information, the identification confidence level of the key operation information, and the instruction status. The status update module is used to update the corresponding instruction status in the instruction context record to a state to be corrected when the recognition confidence of the key operation information is lower than a preset dynamic threshold, or in response to the operator's explicit correction expression, and to send an operation pause instruction to the target device, wherein the target device is the device controlled by the operator through the instruction; The correction information parsing module is used to receive the correction instruction corresponding to the correction state in the correction state to obtain the operator's subsequent voice input, and parse the subsequent voice input into supplementary correction information, wherein the supplementary correction information includes correction type and correction value; The specific operation information determination module is used to obtain the specific operation information that needs to be corrected based on the supplementary correction information and the key operation information of the instruction to be corrected. The update and verification module is used to update the corresponding key operation information in the instruction context record according to the specific operation information, obtain the updated instruction, verify the updated instruction, and submit the updated instruction to the target device for execution.

Images