Information interaction processing method and system based on multi-modal context and speech enhancement
By using a multimodal context and speech enhancement-based information interaction processing method, the decoding process of the end-to-end automatic speech recognition model is adjusted and cross-modal referencing resolution is performed, which solves the problems of misidentification and unclear referencing in specific domains of traditional models and achieves high-accuracy information interaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO YUNJIN MICRO INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional general-purpose automatic speech recognition models have a high false recognition rate in specific domains, and natural language processing technology lacks awareness of the current business scenario context, resulting in key information being distorted and the generated instructions or records being unclear and unable to be executed accurately.
An information interaction processing method based on multimodal context and speech enhancement is adopted. The decoding process of the end-to-end automatic speech recognition model is adjusted by dynamically adjusting the bias list, the logarithmic score is adjusted to the additive bias mode, and cross-modal reference resolution is performed to generate structured business instructions.
It significantly improved the accuracy of professional terminology recognition and colloquial reference parsing, greatly enhanced the executability of generated business instructions, and achieved accuracy and reliability in information flow.
Smart Images
Figure CN122157670A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information processing technology, and in particular to an information interaction processing method and system based on multimodal context and speech enhancement. Background Technology
[0002] In highly collaborative work scenarios such as security monitoring, industrial operation and maintenance, and medical care, the accurate flow of information is crucial.
[0003] Traditional general-purpose automatic speech recognition (ASR) models suffer from high false recognition rates when recognizing domain-specific terms (such as device numbers, drug names, fault codes, and transaction codes) due to variations in training data distribution, leading to the incorrect transcription of crucial information. Furthermore, during interactions, people's colloquial expressions frequently contain referential descriptions such as "that device," "the alarm just now," and "this patient." Traditional natural language processing (NLP) techniques lack awareness of the current business context and cannot accurately associate ambiguous references with specific machine-generated events or records, resulting in unclear and unexecutable generated instructions or records. Summary of the Invention
[0004] Therefore, the purpose of this invention is to provide an information interaction processing method based on multimodal context and speech enhancement, so as to solve or at least partially solve the above-mentioned problems existing in the prior art.
[0005] To achieve the above objectives, the first aspect of the present invention provides an information interaction processing method based on multimodal context and speech enhancement, the method comprising the following steps: S101. Trigger the information interaction process according to the preset business configuration or operation instructions; S102. Based on the information interaction process, access multi-source data within the current business cycle and generate a structured data packet containing an entity set including event summaries and domain context; S103. Based on structured data packets, construct a dynamic bias list. During the decoding process of the end-to-end automatic speech recognition model, adjust the logarithmic score of the end-to-end automatic speech recognition model to an additive bias mode according to the dynamic bias list to generate transcribed text. S104. Perform cross-modal dereference on the transcribed text to generate structured business instructions; S105. Update the time-varying bias coefficient in the addition bias mode online based on the success rate of cross-modal reference resolution and the term misidentification rate in business instructions; S106. Perform closed-loop management and retrieval and traceability of structured data packets and business instructions.
[0006] Furthermore, the step of adjusting the logarithmic score of the end-to-end automatic speech recognition model to an additive bias mode during the decoding process based on the dynamic bias list is expressed as follows:
[0007] in, For additive bias mode, This represents the unbiased logarithmic score output by the end-to-end automatic speech recognition model. The bias intensity coefficient, The time-varying bias coefficient, For entities Frequency of occurrence after normalization.
[0008] Furthermore, the cross-modal referencing resolution involves using a multimodal encoder to vector-encode the event summary and domain context in the transcribed text, and combining the candidate confidence and bias usage tags of the end-to-end automatic speech recognition model to calculate the weighted similarity between the vector-encoded event summary and the domain context. Events with a weighted similarity greater than a preset threshold are identified as referential objects, and structured business instructions are generated based on the referential objects.
[0009] Furthermore, the weighted similarity between the event summary and the domain context in the transcribed text is expressed as follows:
[0010] in, The weighted similarity between the event summary and the domain context. For the domain context vector, For the event Event summary vector, In relation to the event The normalized confidence of entity candidates in the associated transcribed text in the candidate confidence score. For instructions and events Are entity candidates in the associated transcribed text amplified due to bias? and These are the weighting coefficients.
[0011] Furthermore, when the maximum weighted similarity is less than a preset threshold, the corresponding structured business instruction is marked as requiring manual confirmation.
[0012] Furthermore, the online update of the time-varying bias coefficient in the additive bias mode based on the success rate of cross-modal reference resolution and the term misidentification rate in business instructions is expressed as follows:
[0013] in, "For the updated time-varying bias coefficients, To limit the results Within the interval, Based on the bias coefficient, and The positive weighting coefficients are... For cross-modal representation of resolution success rate, To resolve the misidentification rate of cross-modal reference terms, and These are the upper and lower limits of the bias coefficient, respectively.
[0014] Furthermore, the cross-modal reference resolution term misidentification rate is estimated by statistically analyzing users' explicit correction behaviors of entities in the transcribed text, or by detecting the number of entities that cannot be aligned between the transcribed text and the entity set; the cross-modal reference resolution success rate is estimated by statistically analyzing users' confirmation behaviors and modification / deletion behaviors of the generated structured business instructions.
[0015] A second aspect of the present invention provides an information interaction processing system based on multimodal context and speech enhancement, the system comprising: Information interaction trigger module: used to trigger the information interaction process according to preset business configuration or operation instructions; Context generation module: Used to access multi-source data within the current business cycle according to the information interaction process, and generate a structured data packet containing an entity set including event summary and domain context; Speech-to-text enhancement module: used to construct a dynamic bias list based on structured data packets, and adjust the logarithmic score of the end-to-end automatic speech recognition model in the decoding process to an additive bias mode based on the dynamic bias list to generate transcribed text; Cross-modal semantic completion and generation module: used to perform cross-modal reference resolution on transcribed text and generate structured business instructions; Adaptive parameter update module: used to update the time-varying bias coefficients in the additive bias mode online based on the success rate of cross-modal reference resolution and the term misidentification rate in the business instructions; Closed-loop management and retrieval module: Used for closed-loop management and retrieval of structured data packets and business instructions.
[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention proposes an information interaction processing method and system based on multimodal context and speech enhancement. By adjusting the logarithmic score of the end-to-end automatic speech recognition model in the decoding process to an additive bias mode according to a dynamic bias list, the recognition accuracy of professional terms is significantly improved, effectively solving the problem of high misrecognition rate of general end-to-end automatic speech recognition models in specific fields.
[0017] 2. This invention proposes an information interaction processing method and system based on multimodal context and speech enhancement. By using event summaries as knowledge anchors for cross-modal reference resolution, it significantly improves the parsing accuracy of colloquial references, thereby substantially improving the executability of generated business instructions. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic flowchart of an information interaction processing method based on multimodal context and speech enhancement provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall structure of an information interaction processing system based on multimodal context and speech enhancement, provided as an embodiment of the present invention. Detailed Implementation
[0020] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.
[0021] Reference Figure 1 This embodiment provides an information interaction processing method based on multimodal context and speech enhancement, the method including the following steps: S101. Trigger the information interaction process according to the preset business configuration or operation instructions.
[0022] This embodiment triggers the information interaction process through business configuration or operation commands, eliminating the need for manual triggering. It can trigger the corresponding information interaction process according to preset business configuration rules and operation commands, thereby realizing the automatic triggering of the interaction process, reducing manual operation steps, improving the overall intelligence and scenario-based nature, and significantly improving information response efficiency.
[0023] S102. Based on the information interaction process, access multi-source data within the current business cycle, generate a structured data packet containing an entity set including event summary and domain context, and extract entities such as device identifier, location identifier and business object code through named entity recognition, and count the frequency of occurrence of each entity.
[0024] This embodiment standardizes and unifies multi-source data formats by generating structured data packets, giving them clear field definitions, hierarchical structures, and organizational forms, significantly reducing the difficulty and processing overhead of subsequent multi-source data parsing. The structured data packets incorporate entity information, clearly defining entity information such as device identifiers, location identifiers, and business object codes, enhancing the semantic expressiveness and understandability of the data. Furthermore, generating structured data packets facilitates storage, retrieval, verification, and traceability, directly supporting closed-loop management and retrieval traceability of the entire process.
[0025] S103. Construct a dynamic bias list based on structured data packets. During the decoding process of the end-to-end automatic speech recognition model, according to the dynamic bias list, if the edit distance between the candidate word and the entity during the Beam Search decoding process is less than or equal to a preset threshold, adjust its logarithmic score to the additive bias mode, generate the transcribed text, and output the N-best candidate confidence and bias usage label of the end-to-end automatic speech recognition model.
[0026] The adjustment of its logarithmic score to an additive bias mode is expressed as follows:
[0027] in, For additive bias mode, This represents the unbiased logarithmic score output by the end-to-end automatic speech recognition model. The bias intensity coefficient, The time-varying bias coefficient, For entities Frequency of occurrence after normalization.
[0028] The The acquisition method is: compute entity The ratio of the frequency of occurrence of an entity to the frequency of the highest entity occurrence in the current business cycle, or the entity's occurrence frequency. The frequency of occurrence is limited to a preset numerical range and mapped to the [0,1] interval.
[0029] The end-to-end automatic speech recognition model is a model obtained through multi-task joint training, and its joint loss function is... for:
[0030] in, For speech recognition loss, For bias confidence-constrained loss, For cross-modal representation of resolution loss, and This is the loss weighting coefficient, with a value ranging from 0.2 to 0.6.
[0031] The bias confidence constraint loss The calculation method is as follows: for each bias-enhanced entity candidate (i.e., term candidate) in the generated transcribed text, let the biased log probability be... The log probability without bias is Define penalty items , means as follows:
[0032] in A positive threshold This is the average value of the penalty term across the batch. The bias confidence constraint loss is used to constrain the end-to-end automatic speech recognition model to term candidates that already have high confidence under unbiased conditions, preventing over-reliance on bias.
[0033] This embodiment improves the probability of recognizing specific terms by adjusting the logarithmic score to an additive bias mode, applying a positive bias to entities to increase their recall, and applying a negative bias to easily confused or misclassified terms to suppress false positives. In acoustically ambiguous scenarios such as low signal-to-noise ratio and heavy accents, the additive bias mode can be used to adjust to the correct decoding path.
[0034] S104. Perform cross-modal dereference on the transcribed text to generate structured business instructions or records, specifically including: The cross-modal referencing resolution involves using a multimodal encoder to vector-encode the event summary and domain context in the transcribed text. This is combined with the N-best candidate confidence score and bias usage label from an end-to-end automatic speech recognition model to calculate the weighted similarity between the vector-encoded event summary and the domain context. Events with a weighted similarity greater than a preset threshold are identified as referential objects, and structured business instructions or records are generated based on these referential objects. When the maximum weighted similarity score is less than the preset threshold, the corresponding structured business instruction is marked as requiring manual confirmation, or the nearest event is selected as the referential object based on temporal and spatial proximity and set as a low priority.
[0035] The weighted similarity between the event summary and the domain context in the transcribed text is calculated as follows:
[0036] in, The weighted similarity between the event summary and the domain context. For the domain context vector, For the event Event summary vector, In relation to the event Normalized confidence of candidate terms in the candidate confidence of the associated transcribed text. For instructions and events Whether the term candidates in the associated transcribed text are amplified due to bias (output 1 if yes, output 0 if no). and This is a weighting coefficient, with a value ranging from 0.1 to 0.5.
[0037] This embodiment transforms fragmented text from multiple sources and modalities into directly executable structured business instructions through cross-modal reference resolution. It accurately locates the real objects referred to by pronouns and omitted components, avoiding misunderstandings due to unclear references and improving the semantic accuracy of the transcribed text. Furthermore, it supports the integration of various multimodal content, associating cross-modal references with the same entity, allowing fragmented information to form a complete semantic chain, and solving problems such as incomplete information in single-modal contexts.
[0038] S105. Based on the success rate of cross-modal reference resolution and the term misidentification rate in the business instructions of the most recent N information interaction processes, the time-varying bias coefficient in the additive bias mode is updated online. User interactions are continuously monitored. If it is found that users frequently modify entities in the transcribed text, the term misidentification rate of cross-modal reference resolution increases, and the time-varying bias coefficient is automatically lowered to reduce dependence on the bias. If it is found that business instructions are frequently canceled or modified, the success rate of cross-modal reference resolution decreases, and the time-varying bias coefficient is appropriately increased to strengthen the association. This is represented as follows:
[0039] in, " For the updated time-varying bias coefficients, To limit the results Within the interval, Based on the bias coefficient, and The positive weighting coefficients are... For cross-modal representation of resolution success rate, To resolve the misidentification rate of cross-modal reference terms, and These are the upper and lower limits of the bias coefficient, respectively.
[0040] The cross-modal reference resolution term misidentification rate is estimated by statistically analyzing users' explicit correction behaviors of key entities (i.e. terms) in the transcribed text, or by detecting the number of entities that cannot be aligned between the transcribed text and the entity set; the cross-modal reference resolution success rate is estimated by statistically analyzing users' confirmation behaviors and modification / deletion behaviors of the generated structured business instructions.
[0041] Traditional additive biasing methods use fixed coefficients, which are difficult to adapt to changes in audio and other scenarios. This embodiment updates the time-varying bias coefficients in the additive biasing method online based on the success rate of terminology resolution and the terminology misrecognition rate. This allows the time-varying bias coefficients to dynamically adjust with changes in the current scenario, improving overall recognition robustness. There is no need for manual adjustments to the bias strength for different business scenarios; this embodiment can be updated online to adapt to changes in noise, domain terminology, etc.
[0042] S106. Perform closed-loop management and retrieval and traceability of structured data packets and business instructions.
[0043] This embodiment implements closed-loop management and retrieval traceability of structured data packets and business instructions throughout the entire process, which is a core technical means to build a standardized, traceable, and highly reliable information flow system. By uniformly controlling, restricting permissions, and recording the entire process of structured data packets and business instructions, a complete closed loop is formed. Closed-loop management can avoid problems such as information distortion, instruction errors, and content loss from the source. Structured data packets have characteristics such as clearly defined fields and semantics, while business instructions have attributes such as clearly defined objectives. Both must undergo multiple stages of review and management within the closed-loop system, including format verification and logical verification. Structured data and business instructions that fail verification cannot enter the next execution stage, avoiding unexpected anomalies. Compared to traditional loose management models, the closed-loop mechanism can significantly reduce a series of problems such as result deviations caused by data anomalies in structured data packets or errors in business instructions, ensuring a high degree of consistency between upstream information and downstream execution.
[0044] Reference Figure 2 Another embodiment of the present invention provides an information interaction processing system based on multimodal context and speech enhancement, the system comprising: Information interaction trigger module: used to trigger the information interaction process according to preset business configuration or operation instructions; Context generation module: Used to access multi-source data within the current business cycle according to the information interaction process, and generate a structured data packet containing an entity set including event summary and domain context; Speech-to-text enhancement module: It is used to construct a dynamic bias list based on structured data packets, and adjust the logarithmic score of the end-to-end automatic speech recognition model in the decoding process to an additive bias mode based on the dynamic bias list, generate transcribed text, and output the N-best candidate confidence and bias usage label of the end-to-end automatic speech recognition model. Cross-modal semantic completion and generation module: used to perform cross-modal reference resolution on transcribed text and generate structured business instructions; Adaptive parameter update module: used to update the time-varying bias coefficients in the additive bias mode online based on the success rate of cross-modal reference resolution and the term misidentification rate in the business instructions; Closed-loop management and retrieval module: Used for closed-loop management and retrieval of structured data packets and business instructions.
[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for information interaction processing based on multimodal context and speech enhancement, characterized in that, The method includes the following steps: S101. Trigger the information interaction process according to the preset business configuration or operation instructions; S102. Based on the information interaction process, access multi-source data within the current business cycle and generate a structured data packet containing an entity set including event summaries and domain context; S103. Based on structured data packets, construct a dynamic bias list. During the decoding process of the end-to-end automatic speech recognition model, adjust the logarithmic score of the end-to-end automatic speech recognition model to an additive bias mode according to the dynamic bias list to generate transcribed text. S104. Perform cross-modal dereference on the transcribed text to generate structured business instructions; S105. Update the time-varying bias coefficient in the addition bias mode online based on the success rate of cross-modal reference resolution and the term misidentification rate in business instructions; S106. Perform closed-loop management and retrieval and traceability of structured data packets and business instructions.
2. The information interaction processing method based on multimodal context and speech enhancement according to claim 1, characterized in that, The step of adjusting the logarithmic score of the end-to-end automatic speech recognition model to an additive bias mode during the decoding process based on the dynamic bias list is expressed as follows: in, For additive bias mode, This represents the unbiased logarithmic score output by the end-to-end automatic speech recognition model. The bias intensity coefficient, The time-varying bias coefficient, For entities Frequency of occurrence after normalization.
3. The information interaction processing method based on multimodal context and speech enhancement according to claim 1, characterized in that, The cross-modal referencing resolution involves using a multimodal encoder to vector-encode the event summary and domain context in the transcribed text, and combining the candidate confidence and bias usage tags of the end-to-end automatic speech recognition model to calculate the weighted similarity between the vector-encoded event summary and the domain context. Events with a weighted similarity greater than a preset threshold are identified as referential objects, and structured business instructions are generated based on the referential objects.
4. The information interaction processing method based on multimodal context and speech enhancement according to claim 3, characterized in that, The weighted similarity between the event summary and the domain context in the transcribed text is calculated as follows: in, The weighted similarity between the event summary and the domain context. For the domain context vector, For the event Event summary vector, In relation to the event The normalized confidence of entity candidates in the associated transcribed text in the candidate confidence score. For instructions and events Are entity candidates in the associated transcribed text amplified due to bias? and These are the weighting coefficients.
5. The information interaction processing method based on multimodal context and speech enhancement according to claim 4, characterized in that, When the maximum weighted similarity is less than a preset threshold, the corresponding structured business instruction is marked as requiring manual confirmation.
6. The information interaction processing method based on multimodal context and speech enhancement according to claim 1, characterized in that, The online update of the time-varying bias coefficient in the additive bias mode based on the success rate of cross-modal reference resolution and the term misidentification rate in business instructions is expressed as follows: in, "For the updated time-varying bias coefficients, To limit the results Within the interval, Based on the bias coefficient, and The positive weighting coefficients are... For cross-modal representation of resolution success rate, To resolve the misidentification rate of cross-modal reference terms, and These are the upper and lower limits of the bias coefficient, respectively.
7. The information interaction processing method based on multimodal context and speech enhancement according to claim 6, wherein the cross-modal reference resolution term misrecognition rate is estimated by statistically analyzing the explicit correction behavior of users on entities in the transcribed text, or by detecting the number of entities that cannot be aligned between the transcribed text and the entity set; and the cross-modal reference resolution success rate is estimated by statistically analyzing the confirmation behavior of users on the generated structured business instructions, and the modification / deletion behavior.
8. An information interaction processing system based on multimodal context and speech enhancement, the system comprising: Information interaction trigger module: used to trigger the information interaction process according to preset business configuration or operation instructions; Context generation module: Used to access multi-source data within the current business cycle according to the information interaction process, and generate a structured data packet containing an entity set including event summary and domain context; Speech-to-text enhancement module: used to construct a dynamic bias list based on structured data packets, and adjust the logarithmic score of the end-to-end automatic speech recognition model in the decoding process to an additive bias mode based on the dynamic bias list to generate transcribed text; Cross-modal semantic completion and generation module: used to perform cross-modal reference resolution on transcribed text and generate structured business instructions; Adaptive parameter update module: used to update the time-varying bias coefficients in the additive bias mode online based on the success rate of cross-modal reference resolution and the term misidentification rate in the business instructions; Closed-loop management and retrieval module: Used for closed-loop management and retrieval of structured data packets and business instructions.