Cross-modal context speech recognition method and system, and storage medium

Abstract

The embodiment of the invention provides a cross-modal context speech recognition method and system and a storage medium, and belongs to the technical field of speech recognition. The method comprises the steps that cross-modal interaction information is collected, and the cross-modal interaction information is preprocessed to generate cross-modal context data; performing incremental training on the pre-trained large model based on preset multi-scene training data to obtain a cross-modal processing model; extracting a cross-modal feature vector from the cross-modal context data based on the cross-modal processing model, and generating a speech recognition result based on a preset fusion rule; and performing scene processing on the speech recognition result based on a preset scene rule to generate final recognition output. By introducing cross-modal context modeling, shunt incremental training and a controlled scene processing link, a speech recognition result better fits scene constraints in a complex service environment, and semantic deviation is reduced.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more specifically to a cross-modal context speech recognition method, a cross-modal context speech recognition system, and a storage medium. Background Technology

[0002] In speech recognition research and engineering applications, most existing technologies still revolve around single-modal audio signals. Mainstream methods typically begin by capturing raw speech through a microphone, followed by a relatively fixed signal processing flow, such as noise reduction, filtering, and endpoint detection. Acoustic features, such as Mel-frequency cepstral coefficients (MFCCs) and filter bank energy (Fbank), are then extracted. These features are subsequently input into acoustic model structures, including convolutional neural networks, recurrent neural networks, or other lightweight deep models, and combined with language models, such as N-gram models or basic Transformer decoders, during the decoding stage to obtain stable text output.

[0003] This kind of recognition process is widely used in smart speakers, mobile voice assistants, and real-time captioning systems. The devices typically rely solely on the audio track itself, without introducing additional environmental cues or historical context. For example, in smart speakers, the internal speech recognition module primarily relies on the combination of acoustic and language models to directly generate command text based on the user's speech snippets, which is then processed by higher-level logic. Similarly, real-time captioning tools often extract audio tracks from videos and convert them into frame-by-frame captions through a separate speech recognition module, without any connection to scene information beyond the audio.

[0004] While these solutions are relatively mature in terms of noise suppression, feature extraction, and basic semantic parsing, they are generally limited by the boundaries of single-modal input. When user statements rely on historical interactions, application states, or task context, these models struggle to perform semantic completion or ambiguity resolution; in scenarios where business processes and recognition are disconnected, the system often requires manual confirmation to continue. Therefore, existing technologies still have significant shortcomings in multi-scenario understanding and cross-modal association. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-modal context speech recognition method, system, and storage medium to at least solve the problem that existing speech recognition methods are prone to semantic bias due to their inability to combine business context and interactive environment.

[0006] To achieve the above objectives, a first aspect of the present invention provides a cross-modal contextual speech recognition method, the method comprising: acquiring cross-modal interaction information and performing preprocessing on the cross-modal interaction information to generate cross-modal contextual data; performing incremental training on a pre-trained large model based on preset multi-scene training data to obtain a cross-modal processing model; extracting cross-modal feature vectors from the cross-modal contextual data based on the cross-modal processing model and generating speech recognition results based on preset fusion rules; and performing scene processing on the speech recognition results based on preset scene rules to generate a final recognition output.

[0007] Optionally, cross-modal interaction information is collected, and preprocessing is performed on the cross-modal interaction information to generate cross-modal context data, including: performing denoising and time-series structuring processing on the speech data in the cross-modal interaction information to generate speech structured data; performing format unification and field extraction processing on the business scenario data in the cross-modal interaction information to generate business scenario structured data; performing feature extraction and parameter normalization processing on the interaction environment data in the cross-modal interaction information to generate environment structured data; and associating the speech structured data, the business scenario structured data, and the environment structured data based on preset indexing rules to generate cross-modal context data.

[0008] Optionally, cross-modal context data is generated by associating the voice structured data, the business scenario structured data, and the environment structured data based on preset indexing rules. This includes: constructing a voice time index based on the timestamp information of the voice structured data; constructing a business status index based on the business status field of the business scenario structured data; constructing an environment parameter index based on the environment parameter field of the environment structured data; and performing index matching processing based on the voice time index, the business status index, and the environment parameter index to combine the voice structured data, business scenario structured data, and environment structured data under the same index conditions to generate cross-modal context data.

[0009] Optionally, incremental training is performed on a pre-trained large model based on preset multi-scenario training data to obtain a cross-modal processing model, including: constructing a speech feature training set based on speech training data in the preset multi-scenario training data; constructing a text feature training set based on text training data in the preset multi-scenario training data; constructing a cross-modal context training set based on context training data in the preset multi-scenario training data; inputting the speech feature training set, the text feature training set, and the cross-modal context training set into the pre-trained large model according to preset training batches and performing parameter update processing to obtain the cross-modal processing model.

[0010] Optionally, the speech feature training set, the text feature training set, and the cross-modal context training set are input into the pre-trained large model according to a preset training batch and parameter update processing is performed to obtain a cross-modal processing model. This includes: constructing a speech training batch based on the speech feature training set and inputting the speech training batch into the speech encoding structure of the pre-trained large model; constructing a text training batch based on the text feature training set and inputting the text training batch into the text encoding structure of the pre-trained large model; constructing a cross-modal context training batch based on the cross-modal context training set and inputting the cross-modal context training batch into the cross-modal association structure of the pre-trained large model; and performing model parameter update processing based on the joint training results of the speech training batch, the text training batch, and the cross-modal context training batch to obtain a cross-modal processing model.

[0011] Optionally, extracting cross-modal feature vectors from the cross-modal context data based on the cross-modal processing model and generating speech recognition results based on preset fusion rules includes: extracting speech feature vectors from the cross-modal context data based on the speech coding structure of the cross-modal processing model; extracting context feature vectors from the cross-modal context data based on the context coding structure of the cross-modal processing model; performing feature fusion processing on the speech feature vectors and the context feature vectors based on preset fusion rules to generate fused feature vectors; and performing semantic parsing processing based on the fused feature vectors to generate speech recognition results.

[0012] Optionally, feature fusion processing is performed on the speech feature vector and the context feature vector based on a preset fusion rule to generate a fused feature vector, including: segmenting the speech feature vector into segments based on existing index fields in the cross-modal context data to form speech segment vectors aligned by index; segmenting the context feature vector according to the same index field to obtain context segment vectors corresponding to the speech segment vectors at the index; after segmentation, generating a set of cross-modal alignment relationships for each group of corresponding segments, and recording the degree of correlation between each segment in matrix form; using the cross-modal alignment relationships as a reference, mapping the speech segment vectors and context segment vectors group by group to obtain the fused feature vector.

[0013] Optionally, scene processing is performed on the speech recognition result based on preset scene rules to generate the final recognition output, including: matching the speech recognition result with corresponding scene rule entries based on the business status field in the cross-modal context data, and generating a scene instruction constraint sequence for constraining the processing steps; performing field recombination and field validation processing on the business fields in the speech recognition result based on the scene instruction constraint sequence to obtain scene processing data consistent with the current business status; performing preset desensitization processing rules on the privacy fields in the scene processing data to generate desensitized processing data; and generating the final recognition output based on the desensitized processing data in the order of the scene instruction constraint sequence.

[0014] A second aspect of the present invention provides a cross-modal contextual speech recognition system, the system comprising: a data acquisition unit for acquiring cross-modal interaction information and performing preprocessing on the cross-modal interaction information to generate cross-modal contextual data; a training unit for performing incremental training on a pre-trained large model based on preset multi-scenario training data to obtain a cross-modal processing model; a processing unit for extracting cross-modal feature vectors from the cross-modal contextual data based on the cross-modal processing model and generating speech recognition results based on preset fusion rules; and a recognition unit for performing scene processing on the speech recognition results based on preset scene rules to generate a final recognition output.

[0015] On the other hand, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned cross-modal contextual speech recognition method.

[0016] Through the above technical solution, this invention retains contextual clues such as business status, environmental conditions, and interactive behaviors in addition to voice input, providing a referable scenario foundation for subsequent recognition processes. Based on this data, multi-scenario training data is used to perform incremental training on a pre-trained large model, enabling the model to simultaneously encode speech features and contextual features, thus possessing cross-modal semantic modeling capabilities. During the inference phase, the model extracts corresponding feature vectors from the cross-modal contextual data and performs joint calculations of speech features and contextual features according to preset fusion rules, ensuring that the recognition result no longer relies on a single audio message but dynamically adjusts according to the scenario content. Finally, by performing structural verification and business field processing on the recognized text through preset scenario rules, it ensures that the output remains consistent with the current business status. Overall, this method forms a cross-modal closed loop from input, training to output, significantly reducing semantic bias in speech recognition in complex scenarios and making the recognition results more closely match real-world scenario requirements.

[0017] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the steps of a cross-modal contextual speech recognition method provided by one embodiment of the present invention; Figure 2 This is a structural diagram of a cross-modal contextual speech recognition system provided in one embodiment of the present invention. Detailed Implementation

[0019] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] Figure 1 This is a flowchart illustrating the steps of a cross-modal contextual speech recognition method provided in one embodiment of the present invention. Figure 1 As shown, this embodiment of the invention provides a cross-modal contextual speech recognition method, the method comprising: Step S10: Collect cross-modal interaction information and perform preprocessing on the cross-modal interaction information to generate cross-modal context data.

[0021] Specifically, the speech data in cross-modal interaction information undergoes denoising and time-series structuring to generate structured speech data; the business scenario data in cross-modal interaction information undergoes format unification and field extraction to generate structured business scenario data; the interaction environment data in cross-modal interaction information undergoes feature extraction and parameter normalization to generate structured environment data; and the structured speech data, the structured business scenario data, and the structured environment data are associated based on preset indexing rules to generate cross-modal context data.

[0022] Furthermore, based on preset indexing rules, the voice structured data, the business scenario structured data, and the environment structured data are associated to generate cross-modal context data, including: constructing a voice time index based on the timestamp information of the voice structured data; constructing a business status index based on the business status field of the business scenario structured data; constructing an environment parameter index based on the environment parameter field of the environment structured data; and performing index matching processing based on the voice time index, the business status index, and the environment parameter index to combine the voice structured data, business scenario structured data, and environment structured data under the same index conditions to generate cross-modal context data.

[0023] In this embodiment of the invention, voice data, business scenario data, and interaction environment data often coexist. They come from different sources, but together they influence the recognition results during actual interaction. A prudent approach is to first categorize and process this data to avoid problems such as mixed dimensions or missing fields in subsequent stages. Only after preliminary processing of cross-modal interaction information can it proceed to the next stage.

[0024] Speech data processing begins with noise reduction. Raw speech is often affected by indoor echoes, ambient background noise, or noise from the equipment itself, so noise reduction algorithms need to be performed in the time-frequency domain. Commonly used methods include spectral subtraction or subband filtering, which can reduce noise components without destroying the core speech information. After noise reduction, the speech needs to be organized into a structured form in chronological order, usually in frames. Each frame retains the speech amplitude, phase, and timestamp, providing a foundation for subsequent pattern recognition. After this series of processing steps, structured speech data is obtained, which is relatively stable and easier to align with other modalities.

[0025] Preprocessing of business scenario data focuses more on organizing structured fields. This data source depends on the specific application, such as page status, business process nodes, and fields generated from previous interactions. To ensure consistent processing of data from different sources, format standardization is first required, consolidating various business fields into parsable key-value structures. Then, field extraction is performed, identifying fields with strong semantic relevance, such as business tags, task types, or the user's current location. This step has no fixed pattern but typically relies on rule tables or lightweight parsers. Ultimately, this results in structured business scenario data, allowing these fields to be referenced during the integration phase.

[0026] The processing of interactive environmental data leans more towards the numerical side. Environmental noise levels, acquisition device parameters, interaction time periods, and network status can all affect the recognition process. Environmental data typically undergoes feature extraction first, transforming raw measurements into feature dimensions. For example, noise measurements can be converted into decibel range labels, and time periods can be converted into periodic parameters. Then, parameter normalization is performed to compress data from different sources into a unified range, preventing imbalances in subsequent vector concatenation. After processing, structured environmental data is obtained, which is generally simple in form but has wide applications.

[0027] Once the three structured data sets are prepared, they need to be combined into cross-modal context data that can be used for fusion. This step relies on pre-defined indexing rules, the core idea of ​​which is to find alignment conditions between voice, business logic, and environment. A commonly used indexing condition is timestamps, as most interaction events have temporal characteristics. Based on the timestamps in the voice structured data, a voice time index can be constructed as the first-level alignment condition.

[0028] Business scenario data typically includes a business status field, describing the business node the user is currently in. This type of field can be transformed into a business status index; it doesn't directly describe time, but rather the contextual relationship from another dimension. Environmental structured data also has usable fields, such as noise levels or device information, which can be used to build environmental parameter indexes. These three indexes essentially form a tagging system with different dimensions.

[0029] Index matching aligns these three types of tags under the same rule. The matching rule typically uses "same index condition" as the trigger point. When the voice time index falls within a certain time period, the business status index points to a certain scene node, and the environment parameter index meets a certain condition, the three types of structured data can be determined to be in the same context. After the matching condition is met, the corresponding data fragments are combined to generate cross-modal context data. The combination method is not limited to splicing; key-value merging or fragment-level data packet construction can also be used. As long as these data maintain their correlation in subsequent stages, it is considered a valid context form.

[0030] Overall, this preprocessing pipeline organizes data from diverse sources into a unified reference framework, explicitly binding the three dimensions of audio, business, and environment. This binding provides a stable foundation for the subsequent fusion stage, enabling the model to interpret the input content from more dimensions.

[0031] In one specific implementation, this step specifically includes: 1) Business / Scenario Data Acquisition: The "Business / Scenario Data Acquisition Unit" of the cross-modal context acquisition module connects to the target scenario system via API. Taking the financial scenario as an example, it collects information such as user identity information, historical interaction records, and current business status.

[0032] 2) Interactive Environment Data Acquisition: The "Interactive Environment Data Acquisition Unit" collects information such as terminal type, operation logs, and environmental noise.

[0033] 3) Data preprocessing: The "data preprocessing unit" performs operations such as format unification, data cleaning, data structuring, and generating index tables.

[0034] Step S20: Perform incremental training on the pre-trained large model based on preset multi-scenario training data to obtain a cross-modal processing model.

[0035] Specifically, a speech feature training set is constructed based on the speech training data in the preset multi-scene training data; a text feature training set is constructed based on the text training data in the preset multi-scene training data; a cross-modal context training set is constructed based on the context training data in the preset multi-scene training data; the speech feature training set, the text feature training set, and the cross-modal context training set are input into the pre-trained large model according to a preset training batch and parameter update processing is performed to obtain a cross-modal processing model.

[0036] Furthermore, the speech feature training set, the text feature training set, and the cross-modal context training set are input into the pre-trained large model according to a preset training batch and parameter update processing is performed to obtain a cross-modal processing model. This includes: constructing a speech training batch based on the speech feature training set and inputting the speech training batch into the speech encoding structure of the pre-trained large model; constructing a text training batch based on the text feature training set and inputting the text training batch into the text encoding structure of the pre-trained large model; constructing a cross-modal context training batch based on the cross-modal context training set and inputting the cross-modal context training batch into the cross-modal association structure of the pre-trained large model; and performing model parameter update processing based on the joint training results of the speech training batch, the text training batch, and the cross-modal context training batch to obtain a cross-modal processing model.

[0037] In this embodiment of the invention, the core objective of the incremental training phase is to adapt the pre-trained large model to a multi-scenario speech recognition environment. While the pre-trained large model already possesses certain expressive capabilities on general corpora, it often lacks specificity for particular business domains, cross-modal associations, and noisy environments. Performing incremental training based on pre-set multi-scenario training data can introduce distribution features more closely aligned with the target scenario while preserving general capabilities. The overall approach involves finely dividing the training data, then feeding it into different encoding structures, and finally updating the parameters using joint loss.

[0038] Multi-scenario training data typically comprises three parts: speech training data, text training data, and contextual training data. Speech training data comes from real-world interactive speech or recorded corpora, covering various speakers, accents, and noise conditions. Text training data corresponds to the annotated text and supplementary text of the speech content, such as business template statements or common question-and-answer patterns. Contextual training data originates from business status, historical interaction records, and environmental parameters, used to describe scenario information beyond speech. These three types of data together constitute the pre-set multi-scenario training data, providing material for subsequently constructing different training sets.

[0039] The construction of a speech feature training set typically begins with feature extraction. Endpoint detection and short-time framing are performed on each speech training sample, and Mel-frequency cepstral coefficients, filter bank energy, or other acoustic features are calculated. The feature sequence is stored along with its corresponding timestamp, forming a speech feature sample with a time dimension. In multi-scene contexts, the speech feature training set intentionally retains labels such as noise type and device type to facilitate conditional modeling during training. After construction, the speech feature training set is divided into a training subset and a validation subset according to a preset ratio.

[0040] The construction of the text feature training set focuses on semantic and lexical information representation. The text training data is first segmented into words or sub-words, and then mapped to word vectors or sub-word vector spaces. A common practice is to use a unified embedding table, employing the same encoding method for texts from different scenarios. For texts containing domain terminology, a domain lexicon can be added to the embedding layer to prevent important words from being fragmented too much. Finally, each text sample is represented as a vector sequence, maintaining an index-aligned relationship with the speech samples, forming the text feature training set.

[0041] The construction of cross-modal context training sets relies more heavily on structured fields. Fields such as business status, page identifier, device information, and environmental noise range in the context training data are usually already structured and only need to be mapped into dense vectors according to preset rules. For example, business status can be mapped to a set of discrete label vectors, and environmental noise ranges can be converted into continuous numerical features. For time-related context data, positional encoding can also be introduced to allow the model to identify the interaction sequence. The final cross-modal context training set consists of multi-dimensional feature vectors, which are associated with the corresponding speech and text samples through a unified index.

[0042] During training, the three types of training sets need to be organized into predefined training batches. The speech feature training set is divided into several speech training batches, each containing a fixed number of speech samples; the text feature training set is divided into text training batches, corresponding to the text annotations of the speech samples in the same batch; the cross-modal context training set is divided into cross-modal context training batches, aligned with the speech and text batches in terms of index. Such a training batch can be viewed as three parallel channels for speech, text, and context, facilitating input into different network structures.

[0043] Pre-trained large models typically consist of three sub-structures: a speech encoding structure, a text encoding structure, and a cross-modal association structure. The speech training batch takes the speech encoding structure as input, performs time-series modeling of acoustic features, and outputs a set of speech representations. The text training batch takes the text encoding structure as input, models word order relationships and context dependencies in the semantic space, and outputs text representations. The cross-modal context training batch takes the cross-modal association structure as input, encodes business states and environmental conditions, and outputs context representations. These three structures share some underlying representations at the parameter level, but can also be configured as independent sub-networks depending on implementation requirements.

[0044] To describe this joint training process, a relatively concise loss expression can be given. For example, the total loss can be defined as: L = α·Lv + β·Lt + γ·Lc.

[0045] Where Lv is the speech recognition-related loss, such as acoustic decoding loss; Lt is the text prediction or language modeling loss; and Lc is the context prediction or alignment constraint loss. α, β, and γ are the weight coefficients of the three losses, used to balance the influence of different modalities during training. The weights can be set empirically or adjusted during the pre-training phase based on validation set performance.

[0046] In each training batch, the speech representation output by the speech coding structure and the text representation output by the text coding structure are aligned at the decoding end or in the joint layer. The alignment method can be attention-based soft alignment or implicit alignment based on connection-time classification. The contextual representation output by the cross-modal association structure is then fused with the speech and text representations through concatenation, weighted summation, or gating mechanisms. The fused representation participates in the loss calculation mentioned above, and during backpropagation, the parameters that need adjustment in all three structures are updated simultaneously.

[0047] Parameter updates typically employ gradient descent-type optimization algorithms. In each iteration, the trainable parameters of the speech coding structure, text coding structure, and cross-modal association structure are updated based on the gradient of the loss function. For relatively stable low-level parameters in pre-trained large models, selective freezing or a lower learning rate can be used to reduce perturbations to general features. For high-level parameters strongly correlated with the scene, a relatively high learning rate is used to accelerate adaptation to new distributions.

[0048] When training data covers multiple business domains across various scenarios, scenario labels can be added to the training batches. These scenario labels, in scalar or vector form, are input into the cross-modal association structure to distinguish different business types or application environments. During training, the model gradually learns to use different feature combinations under different scenario labels, thereby enhancing its cross-scenario transfer capability. After incremental training, the cross-modal processing model can serve multiple scenarios under the same structure.

[0049] In one specific implementation, common domain speech data is collected, each speech is labeled with corresponding text, and context data is associated to construct a multi-scenario dataset. Based on an open-source large model, the bottom 50% of the Transformer layers are frozen (preserving the ability to extract general speech features), and the top 50% of the layers are fine-tuned. The input is speech features (Mel spectrogram) + text features (word embeddings). The Prompt template format is set to [Scenario: XXX Speech Features: {vector} Context: {XXX} Recognition Result: {}]. A small number of samples are collected to fine-tune the training of the Prompt embedding layer. After fine-tuning, the recognition accuracy of this business instruction is much higher than that of the traditional model.

[0050] In another possible implementation, a cross-modal perturbation consistency constraint is added during the incremental training phase. This constraint is somewhat indirect; it uses lightweight perturbations to construct response differences between different modes, thereby enhancing the stability of the cross-modal correlation structure.

[0051] Specifically, a very small perturbation vector is introduced into each of the speech feature training set, text feature training set, and cross-modal context training set, with the perturbation strength kept within a range that does not affect the original semantic representation. The three types of perturbed features are then input into the speech encoding structure, text encoding structure, and cross-modal association structure, respectively, resulting in three sets of perturbed representations. Next, the difference between the representations before and after perturbation is calculated, and a constraint term on this difference is added to the training loss to ensure that the perturbed cross-modal representation remains consistent within a controllable range. This perturbation consistency constraint approximates the feature spaces of different modalities, making the model less prone to local shifts when handling complex scenarios.

[0052] Step S30: Extract cross-modal feature vectors from the cross-modal context data based on the cross-modal processing model and generate speech recognition results based on preset fusion rules.

[0053] Specifically, speech feature vectors are extracted from the cross-modal context data based on the speech coding structure of the cross-modal processing model; context feature vectors are extracted from the cross-modal context data based on the context coding structure of the cross-modal processing model; feature fusion processing is performed on the speech feature vectors and the context feature vectors based on a preset fusion rule to generate a fused feature vector; and semantic parsing processing is performed on the fused feature vectors to generate a speech recognition result.

[0054] Furthermore, based on preset fusion rules, feature fusion processing is performed on the speech feature vector and the context feature vector to generate a fused feature vector. This includes: segmenting the speech feature vector based on existing index fields in the cross-modal context data to form index-aligned speech segment vectors; segmenting the context feature vector according to the same index field to obtain context segment vectors corresponding to the speech segment vectors at the index; after segmentation, generating a set of cross-modal alignment relationships for each group of corresponding segments, and recording the degree of correlation between each segment in matrix form; and mapping the speech segment vectors and context segment vectors group by group with reference to the cross-modal alignment relationships to obtain the fused feature vector.

[0055] In this embodiment of the invention, the voice, business scenario, and environmental information have already been organized into cross-modal context data. Now, it is necessary to extract a computable vector representation from this data. The overall approach is not complicated; the core is to encode separately, then perform controlled fusion, and finally hand it over to the semantic parsing module to complete the text output. The whole process is more like an assembly line, with each step responsible for its own task.

[0056] The speech coding structure undertakes the acoustic modeling task. Cross-modal context data contains structured content related to the original speech, such as frame-level feature sequences, timestamps, and speech segmentation information. The speech coding structure performs temporal modeling based on these feature sequences, typically using multi-layer convolutional networks, recurrent networks, or self-attention networks to compress and reassemble frame-level features. After passing through the speech coding structure, a set of speech feature vectors is obtained, which preserves temporal information while compressing redundant details.

[0057] Context encoding structures handle non-speech components. Cross-modal context data includes business status fields, page identifiers, operation sequences, and environmental noise labels. This content is closely related to the speech content but is expressed in a more structured manner. Context encoding structures typically employ an embedding layer plus several transform network layers to map discrete fields to a continuous vector space, and then combine them according to the order or hierarchical relationship between the fields. After encoding, a set of context feature vectors is obtained, which carries scene-related information.

[0058] Once the speech feature vectors and context feature vectors are prepared, a pre-defined fusion rule needs to be introduced. Simple concatenation often fails to fully utilize contextual information, so a segmented approach with alignment is more reliable. The basic assumption here is that speech segments and contextual segments near the same time slice have a stronger correlation and need to be closer in the vector space. The fusion rule revolves around this assumption, but the implementation path is controllable and discrete.

[0059] Segmentation begins with the speech feature vectors. Cross-modal context data already contains index fields, such as timestamp intervals, session rounds, or interaction step numbers. Based on these index fields, the long sequence of speech features is segmented into several speech fragment vectors, each corresponding to an indexed segment. Segmentation is performed only according to index boundaries, without complex transformations of the internal features, thus ensuring that each fragment can still be traced back to the original speech segment.

[0060] Context feature vectors are also segmented using the same set of index fields. Each context feature vector is assigned to a specific index segment based on the business status timeline or interaction steps, generating a context segment vector corresponding to the speech segment vector in the index. This process does not force length alignment, but rather ensures comparability between speech segments and context segments under the same index. For missing context segments, empty vectors or placeholder encoding can be used to avoid disrupting the overall structure.

[0061] After segmentation, cross-modal alignment needs to be established for each pair of "speech segment vector + context segment vector". Recording the alignment in matrix form facilitates analysis; rows represent positions within speech segments, columns represent positions within context segments, and matrix elements represent the degree of correlation between corresponding positions. The degree of correlation can be calculated based on cosine similarity, dot product, or a lightweight attention mechanism; the choice depends on the overall model complexity requirements. The resulting cross-modal alignment matrix provides a clear weight reference for subsequent fusion.

[0062] The fusion process proceeds primarily along cross-modal alignment relationships. For each corresponding segment, the context segment vector is weighted and aggregated according to the weights in the alignment matrix, and then combined with the speech segment vector. The combination method can be weighted summation, concatenation with linear transformation, or weighted superposition controlled by a gated unit. After each segment is processed, a local fusion vector is generated. The local fusion vectors of all segments are then reassembled in index order to form a complete sequence of fused feature vectors.

[0063] The fused feature vectors then proceed to the semantic parsing stage. The semantic parsing module outputs a sequence of symbols or words based on the temporal order and internal correlation of the fused features. Common approaches include using a sequence-to-sequence structure or an autoregressive decoder to progressively generate text tags from the fused features. Language model constraints can be introduced during parsing to reduce obviously illogical outputs without compromising cross-modal information. Finally, the semantic parsing module provides the speech recognition result, which is the text content that matches the current cross-modal context.

[0064] In one specific implementation, a wavelet transform algorithm is used, combined with environmental noise parameters collected in step one, to remove background noise. Feature extraction is performed by a speech feature extractor, which converts the 16kHz monophonic speech into a 128×300 Mel spectrogram, extracting speech rate (4 syllables / second) and pitch (250Hz) features, and encoding them into a 512-dimensional speech vector. Contextual information is converted into vectors and concatenated using one-hot coding, mean pooling coding, and temporal coding, and then matched with the dimension of the speech vector. The two types of vectors are input into the optimized model, and weights are calculated in the cross-modal attention layer. After obtaining preliminary recognition results, semantic completion is performed by combining rules and context. Finally, historical records are combined, and a mapping library is called for ambiguity resolution.

[0065] Generate recognition results including confidence level and contextual source.

[0066] Step S40: Perform scene processing on the speech recognition results based on preset scene rules to generate the final recognition output.

[0067] Specifically, based on the business status field in the cross-modal context data, corresponding scene rule entries are matched for the speech recognition result, and a scene instruction constraint sequence is generated for constraining the processing steps; based on the scene instruction constraint sequence, field recombination and field validation processing are performed on the business fields in the speech recognition result to obtain scene processing data consistent with the current business status; based on the privacy fields in the scene processing data, preset de-identification processing rules are executed to generate de-identified processing data; based on the de-identified processing data, the final recognition output is generated according to the order of the scene instruction constraint sequence.

[0068] In this embodiment of the invention, the speech recognition result is merely a text string. Without business semantic constraints, it is easy for fields to be missing, meanings to be distorted, or instructions to go out of bounds. The preset scenario rules are used to add a business framework to this text, so that subsequent execution logic has a basis to follow. Overall, this step is more like a structured restructuring than a simple text post-processing.

[0069] The core anchor of scenario rules is the business status field. The cross-modal context data already records the business nodes of the current session, such as the "identity verification stage" in an account opening process or the "result feedback stage" in a work order processing. Based on these business status fields, matching the corresponding scenario rule entries to the speech recognition results allows us to identify a set of constraints and available operations. Each scenario rule entry typically provides a list of required fields, field type constraints, and a set of allowed business actions. After matching, a scenario instruction constraint sequence can be generated to clarify the order and boundaries of subsequent processing.

[0070] The scenario instruction constraint sequence acts as a "script" here. The constraint sequence doesn't need to be overly complex; it's generally a set of ordered instruction labels, such as "extract account field," "validate amount range," and "check permission flags." Each instruction label corresponds to a fixed piece of processing logic, avoiding too many conditional branches in the scenario processing. The generation of the constraint sequence depends on predefined instruction templates in the scenario rule entries; the business status field is only responsible for selecting the template and doesn't directly participate in the logical details. The advantage of this design is that business adjustments mainly modify the rule table, rather than changing the entire processing chain.

[0071] Based on the scene instruction constraint sequence, the main task of this stage is to perform field recombination and field validation on the business fields in the speech recognition results. Field recombination refers to extracting business fields from the freely expressed text according to rules, and then reorganizing them according to the field order and naming method required by the scene rules. For example, if a user says "Please transfer 2,000 yuan from my salary card to my savings card," the field recombination step will sequentially locate the three fields "source account," "target account," and "amount," and map them to standard field names. Field validation checks whether the fields are complete and whether the values ​​conform to the preset format, such as whether the amount is a numeric type and whether the account number matches the length rules. After field recombination and field validation are completed, a scene processing data consistent with the current business state can be obtained.

[0072] Data processed in various scenarios often contains sensitive information, therefore privacy fields need to be treated separately. Pre-defined anonymization rules can be configured by field type; for example, retaining the last four digits of an ID number, the middle four digits of a phone number, and partially replacing or retaining the first character of a name. Anonymization rules can be based on simple mask templates or more granular strategies, such as anonymizing only in log recording scenarios while retaining the complete value in the real-time execution chain. By executing the corresponding anonymization rules item by item based on the privacy fields in the scenario-processed data, an anonymized data set can be generated, ensuring that critical personal information is not exposed during recording and backtracking.

[0073] The generation of the final identification output still relies on the scenario instruction constraint sequence. The order of the constraint sequence determines how the output structure is assembled; for example, business actions are written first, then key fields, and finally necessary context markers are attached. The output format can be a structured instruction object or a text fragment with field labels, depending on the subsequent interface. The key is that every item in the output can be traced back to the preceding scenario processing data or anonymized data, rather than being pieced together temporarily.

[0074] In one specific implementation, text is mapped to API commands, which are then used to trigger operations in business systems. Information such as bank card numbers and ID card numbers is anonymized; for non-privacy-sensitive information, the anonymization function can be disabled, retaining complete command information. An audit report is generated, including the voice storage address, context data, model version, recognition results, and operation records, ensuring traceability.

[0075] Furthermore, data from manual corrections, business records, and user feedback are collected. Error samples are labeled "corrected samples," and correct samples are designated "positive samples." Feedback samples are used to fine-tune the model, updating attention weights and semantic rules. Core metrics are defined for different business scenarios, and subsequent iterative optimizations are based on these core metrics.

[0076] In another possible implementation, a scene conflict detection mechanism is added during the scene processing stage to identify potential contradictions between the speech recognition results and the business status. This mechanism does not involve re-recognition, but rather adds a backoff compensation chain to the scene constraint sequence.

[0077] Specifically, after generating the scene instruction constraint sequence, a set of conflict conditions can be constructed based on the business rule table, such as field omissions, field out-of-bounds errors, and discrepancies between action types and business nodes. When the speech recognition result triggers a conflict condition, a backtracking compensation chain is activated to perform a partial reinterpretation of the recognized text. For example, key fields can be re-examined within a limited scope, or missing values ​​can be filled in based on the context of the previous round of interaction. The execution of the compensation chain does not involve large-scale model inference but is a lightweight, rule-driven operation, typically including three steps: field replacement, field candidate sorting, and context backfilling. After processing, the field reorganization and desensitization process is re-entered to maintain overall structural consistency. This approach is particularly useful in multi-turn dialogues or noisy environments, reducing recognition interruptions caused by business conflicts, and is a structural reinforcement strategy.

[0078] Figure 2 This is a structural diagram of a cross-modal contextual speech recognition system provided in one embodiment of the present invention. Figure 2 As shown, this invention provides a cross-modal contextual speech recognition system, comprising: a data acquisition unit for acquiring cross-modal interaction information and performing preprocessing on the cross-modal interaction information to generate cross-modal contextual data; a training unit for performing incremental training on a pre-trained large model based on preset multi-scene training data to obtain a cross-modal processing model; a processing unit for extracting cross-modal feature vectors from the cross-modal contextual data based on the cross-modal processing model and generating speech recognition results based on preset fusion rules; and a recognition unit for performing scene processing on the speech recognition results based on preset scene rules to generate a final recognition output. The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned cross-modal contextual speech recognition method.

[0079] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0080] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details described above. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe the various possible combinations.

[0081] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the embodiments of the present invention, they should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A cross-modal contextual speech recognition method, characterized in that, The method includes: Collect cross-modal interaction information and perform preprocessing on the cross-modal interaction information to generate cross-modal context data; Incremental training is performed on a pre-trained large model based on pre-set multi-scenario training data to obtain a cross-modal processing model; Based on the cross-modal processing model, cross-modal feature vectors are extracted from the cross-modal context data, and speech recognition results are generated based on preset fusion rules; The speech recognition results are processed according to preset scene rules to generate the final recognition output.

2. The method according to claim 1, characterized in that, Collecting cross-modal interaction information and performing preprocessing on the cross-modal interaction information to generate cross-modal context data includes: Denoising and time-series structuring are performed on the speech data in cross-modal interaction information to generate structured speech data; Perform format unification and field extraction processing on business scenario data in cross-modal interaction information to generate structured business scenario data; Feature extraction and parameter normalization are performed on the interaction environment data in cross-modal interaction information to generate structured environment data; Based on preset indexing rules, the structured voice data, the structured business scenario data, and the structured environment data are associated to generate cross-modal context data.

3. The method according to claim 2, characterized in that, Based on preset indexing rules, the structured speech data, the structured business scenario data, and the structured environment data are associated to generate cross-modal context data, including: A voice time index is constructed based on the timestamp information of the structured voice data; A business status index is constructed based on the business status fields of the structured data of the aforementioned business scenario; An environmental parameter index is constructed based on the environmental parameter fields of the structured environmental data. Based on the voice time index, the service status index, and the environment parameter index, index matching processing is performed to combine the voice structured data, service scenario structured data, and environment structured data under the same index conditions to generate cross-modal context data.

4. The method according to claim 1, characterized in that, Incremental training is performed on a pre-trained large model based on pre-set multi-scenario training data to obtain a cross-modal processing model, including: A speech feature training set is constructed based on the speech training data in the preset multi-scenario training data. A text feature training set is constructed based on the text training data in the preset multi-scenario training data; Construct a cross-modal context training set based on the context training data in the preset multi-scenario training data; The speech feature training set, the text feature training set, and the cross-modal context training set are input into the pre-trained large model according to a preset training batch and parameter update processing is performed to obtain the cross-modal processing model.

5. The method according to claim 4, characterized in that, The speech feature training set, the text feature training set, and the cross-modal context training set are input into the pre-trained large model according to a preset training batch and parameter update processing is performed to obtain a cross-modal processing model, including: Based on the speech feature training set, a speech training batch is constructed and the speech training batch is input into the speech coding structure of the pre-trained large model; Based on the text feature training set, a text training batch is constructed and the text training batch is input into the text encoding structure of the pre-trained large model; Based on the cross-modal context training set, a cross-modal context training batch is constructed and the cross-modal context training batch is input into the cross-modal association structure of the pre-trained large model; Based on the joint training results of the speech training batch, the text training batch, and the cross-modal context training batch, model parameter update processing is performed to obtain a cross-modal processing model.

6. The method according to claim 5, characterized in that, Based on the cross-modal processing model, cross-modal feature vectors are extracted from the cross-modal context data, and speech recognition results are generated based on preset fusion rules, including: The speech coding structure based on the cross-modal processing model extracts speech feature vectors from the cross-modal context data; The context feature vector is extracted from the cross-modal context data based on the context encoding structure of the cross-modal processing model. Based on preset fusion rules, feature fusion processing is performed on the speech feature vector and the context feature vector to generate a fused feature vector; Semantic parsing processing is performed based on the fused feature vectors to generate speech recognition results.

7. The method according to claim 6, characterized in that, Based on preset fusion rules, feature fusion processing is performed on the speech feature vector and the context feature vector to generate a fused feature vector, including: Based on the existing index fields in the cross-modal context data, the speech feature vector is segmented and sliced ​​to form speech slice vectors aligned by the index; The context feature vector is segmented according to the same index field to obtain the context segment vector corresponding to the speech segment vector in the index. After segmentation, a cross-modal alignment relationship is generated for each corresponding segment, and the degree of correlation between each segment is recorded in matrix form. Using cross-modal alignment as a reference, the speech segment vectors and context segment vectors are mapped one group at a time to obtain the fused feature vector.

8. The method according to claim 1, characterized in that, Based on preset scene rules, the speech recognition results are processed to generate the final recognition output, including: Based on the business status field in the cross-modal context data, the corresponding scene rule entries are matched for the speech recognition results, and a scene instruction constraint sequence for the constraint processing steps is generated. Based on the scenario instruction constraint sequence, the business fields in the speech recognition result are subjected to field recombination and field verification processing to obtain scenario processing data consistent with the current business state; Based on the privacy fields in the scenario processing data, a preset de-identification processing rule is executed to generate de-identified processing data; The final recognition output is generated based on the de-identified data in the order of the scene instruction constraint sequence.

9. A cross-modal contextual speech recognition system, characterized in that, The system includes: The acquisition unit is used to acquire cross-modal interaction information and perform preprocessing on the cross-modal interaction information to generate cross-modal context data; The training unit is used to perform incremental training on a pre-trained large model based on preset multi-scenario training data to obtain a cross-modal processing model; The processing unit is used to extract cross-modal feature vectors from the cross-modal context data based on the cross-modal processing model and generate speech recognition results based on preset fusion rules; The recognition unit is used to perform scene processing on the speech recognition results based on preset scene rules to generate the final recognition output.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the cross-modal contextual speech recognition method as described in any one of claims 1-8.

Images