Text history-based reasoning method and device in multi-modal dialogue scene, medium and program product

Abstract

The invention belongs to the technical field of natural language processing, and provides a text history-based reasoning method and device in a multi-modal dialogue scene, a medium and a program product, and the method comprises the steps: obtaining a historical dialogue of voice and text alternation, and carrying out the automatic voice recognition of a voice part in the historical dialogue to obtain a recognition text; extracting non-text feature information at least containing mood, emotion or emphasis when the recognition text is generated, compressing and encoding the non-text feature information into a voice tag, and adding the voice tag to the corresponding recognition text to form historical text information with a semantic compensation mark; and then combining the historical text information and the current user input voice into model input content, inputting the model input content into a multi-modal dialogue model for reasoning, and outputting a corresponding text reply. According to the method, the length of a model input sequence can be shortened, the reasoning efficiency is improved, meanwhile, key non-text information is reserved, the dialogue accuracy is improved, and the multi-modal dialogue interaction experience is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a reasoning method, device, medium and program product based on text history in a multi-modal dialogue scenario. BACKGROUND

[0002] With the rapid development of large language model technology, text dialogue has been difficult to meet the user's demand for natural interaction, and voice dialogue is directly and conveniently applied in daily scenarios, and multi-modal voice dialogue models have emerged and shown excellent performance in the field of voice processing.

[0003] At present, when the multi-modal voice dialogue model reasons, the history content of "voice-text-voice" alternation is often taken as the reasoning basis to generate the next round of text reply. However, this method has obvious defects: the number of tokens corresponding to the voice signal is much larger than that of the text token due to the large number of frames, resulting in too long model input sequence, which directly affects the reasoning speed and efficiency; at the same time, a large number of voice tokens easily interfere with the accurate understanding of the model to the context semantics, causing the accuracy of the reply to decrease, it is difficult to balance the interaction efficiency and semantic integrity of multi-modal dialogue, and it is difficult to fully meet the user's demand for smooth and accurate dialogue. SUMMARY

[0004] To this end, the present application provides a reasoning method, system, electronic device and computer program product based on text history in a multi-modal dialogue scenario to at least partially solve the above technical problems.

[0005] The present application provides a reasoning method based on text history in a multi-modal dialogue scenario, comprising the following method steps: obtaining a history dialogue containing voice and text alternation, and automatically recognizing the voice part in the history dialogue to obtain recognized text; extracting non-text feature information of the voice in the recognized text generation process, wherein the non-text feature information at least includes one of tone, emotion or emphasis; compressing and encoding the non-text feature information to form a voice tag and attach it to the corresponding recognized text to obtain history text information with semantic compensation markers; combining the history text information with the voice input by the current user to form model input content; inputting the model input content into a multi-modal dialogue model for reasoning to output a text reply corresponding to the voice input by the current user.

[0006] Another aspect of the present application also provides a reasoning device based on text history in a multi-modal dialogue scenario, comprising: An acquisition module is configured to acquire a historical dialogue containing voice and text alternation, and to perform automatic speech recognition on a voice part in the historical dialogue to obtain recognized text; A non-text feature information extraction module is configured to extract non-text feature information of the voice during the recognized text generation process, the non-text feature information at least including one of tone, emotion, or emphasis; A compensation marking module is configured to compress and encode the non-text feature information to form a voice tag and attach the voice tag to corresponding recognized text to obtain historical text information with semantic compensation marking; A combination module is configured to combine the historical text information with current user input voice to form model input content; A reasoning module is configured to input the model input content into a multi-modal dialogue model to perform reasoning and output text reply corresponding to the current user input voice.

[0007] Another aspect of the present application also provides an electronic device, which comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-modal dialogue scene based reasoning method based on text history as described above.

[0008] Another aspect of the present application provides a computer readable storage medium having computer program instructions stored thereon, the computer program instructions being executable by a processor to implement the multi-modal dialogue scene based reasoning method based on text history as described above.

[0009] Another aspect of the present application provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the multi-modal dialogue scene based reasoning method based on text history as described above.

[0010] The present application can significantly shorten the model input sequence length to improve the reasoning efficiency by converting the historical dialogue with voice-text-voice alternation in the multi-modal dialogue into text through automatic speech recognition, extracting non-text feature information of the voice, compressing and encoding the non-text feature information into a voice tag, attaching the voice tag to the recognized text to form historical text with semantic compensation marking, and then combining the historical text with current user voice to input the multi-modal dialogue model for reasoning. Meanwhile, the voice tag retains key non-text information such as tone, emotion, emphasis, and rhythm, and in combination with model attention weighting and prompt control signal, can improve the model's understanding of context semantics and the adaptation degree of reply to user needs, and balance the reasoning efficiency and dialogue accuracy to optimize the multi-modal dialogue interaction experience. BRIEF DESCRIPTION OF DRAWINGS

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a reasoning method based on text history in a multimodal dialogue scenario provided by an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the non-text feature information extraction process provided in an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram of the structure of a text history-based reasoning device in a multimodal dialogue scenario provided by an embodiment of the present invention.

[0015] Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The technical solutions of this application will be described in detail below with reference to various embodiments.

[0018] like Figure 1 As shown in the diagram, this invention discloses a reasoning method based on text history in a multimodal dialogue scenario, including the following method steps: S101, acquire historical dialogues that alternate between speech and text, and perform automatic speech recognition on the speech portion of the historical dialogues to obtain the recognized text; S102, extracting non-textual feature information of the speech during the text recognition generation process, wherein the non-textual feature information includes at least one of tone, emotion, or emphasis; S103, the non-text feature information is compressed and encoded to form a speech tag and attached to the corresponding recognized text to obtain historical text information with semantic compensation mark; S104, combine the historical text information with the voice input by the current user to form the model input content; S105, input the model input content into the multimodal dialogue model for inference, and output a text response corresponding to the voice input by the current user.

[0019] In some embodiments, for S101, firstly, all user speech segments to be identified are separated from the speech-text alternating history dialogue to ensure no omissions or confusion.

[0020] Specifically, using the dialogue round number as an index, the historical dialogue database is traversed to extract the voice files input by the user in each round. For example, for a certain scenario's historical dialogue, two voice segments to be recognized are extracted: the voice of user in round 1 and the voice of user in round 2. At the same time, the dialogue ID, round number, and previous model text response content corresponding to each voice segment are recorded to form a voice segment-meta-information association table, avoiding confusion between voices from different dialogues and different rounds.

[0021] Since different users' voice files may have inconsistent sampling rates, bit depths, and number of channels, this embodiment uses a unified format to adapt to the subsequent speech recognition model.

[0022] Specifically, a uniform sampling rate is used to resample all speech segments to 16kHz (the standard input sampling rate of mainstream speech recognition models), and the audio sampling points are adjusted through interpolation algorithms (such as linear interpolation) to ensure that there is no significant distortion in sound quality. Unify bit depth and vocal tract, convert speech segments into 16-bit mono WAV format, take left channel data for stereo speech (or calculate the average value of stereo channels) to eliminate the impact of vocal tract differences on recognition. Speech segment duration trimming: For speech segments that are too long (>10s), they are divided into multiple sub-segments (each sub-segment is 2-5s long) according to semantic pauses (silent segments are identified by silence detection algorithm) to avoid recognition delay caused by long speech.

[0023] Next, based on the standardized user speech segments, the speech is converted into text using automatic speech recognition technology, while ensuring that the recognition results are consistent with the speech semantics.

[0024] Specifically, based on end-to-end speech recognition models or speech recognition modules embedded in existing mainstream multimodal dialogue models, these models have been trained with large-scale speech data and have a high accuracy rate in recognizing everyday conversational speech (accuracy ≥ 95% in clear speech scenarios).

[0025] Optionally, the recognized text is combined with the text portion of the historical dialogue to form a complete textualized historical dialogue. Specifically, based on the previously recorded speech segment-meta-information association table, the recognized text of each speech segment replaces the user speech portion in the original historical dialogue. For example, if the original historical dialogue is "Round 1 User Speech - Model Text 1 - Round 2 User Speech - Model Text 2", the replacement becomes "Round 1 Recognized Text - Model Text 1 - Round 2 Recognized Text - Model Text 2", ensuring that each recognized text segment accurately corresponds to the round position of the original speech and maintaining the "text-text alternation" historical dialogue structure.

[0026] In some embodiments, for S102, a tool that supports fine-grained acoustic feature acquisition is selected to ensure that the core acoustic features of speech, such as fundamental frequency (F0), energy, speech rate, and pause duration, can be accurately captured. These features are the core inputs of the subsequent emotion recognition sub-model. Preferably, the non-textual feature information is extracted by an emotion recognition sub-model, which is trained based on acoustic features and used to output the corresponding emotion category or tone category.

[0027] Specifically, an emotion recognition sub-model is built based on a deep learning architecture, and a network structure suitable for sequence feature processing (such as a CNN+LSTM hybrid architecture) is selected. CNN is used to extract local key information in acoustic features (such as volume fluctuations in a short period of time), and LSTM is used to capture the temporal correlation of acoustic features (such as continuous changes in tone) to adapt to the temporal characteristics of speech signals.

[0028] Optionally, the training process of the emotion recognition sub-model includes, for example, the following specific steps: For each speech record in the training set, it is converted into numerical features that the model can process using acoustic feature extraction tools. For example, these include... For each speech, it is segmented into speech frames with a frame length of 20ms and a frame shift of 10ms. For each frame, 13-dimensional Mel-frequency cepstral coefficients (MFCC, reflecting the spectral characteristics of speech), 1-dimensional fundamental frequency (F0, associated with intonation and emotion, such as F0 rising when angry), 1-dimensional root mean square energy (RMS, reflecting volume strength and associated with emphasis features), and 2-dimensional speech rate and pause features (speech rate = number of syllables / speech duration, pause features = number of pauses / speech duration, associated with hesitant tone) are extracted to form 17-dimensional basic features for each frame. The extracted basic features are Z-score standardized (to eliminate feature scale differences between different speech sounds), and the 17-dimensional features are reduced to 12-dimensional features through principal component analysis (PCA) (retaining more than 95% of feature information), reducing the computational load of the model and avoiding overfitting. All frame features of a single speech are concatenated in chronological order to form a temporal feature matrix of frame number × 12 dimensions, which serves as the input data format for the emotion recognition sub-model.

[0029] A three-layer emotion recognition sub-model architecture consisting of CNN, LSTM, and a fully connected layer is constructed. The first layer comprises two convolutional layers (3×3 kernel size, ReLU activation function) to extract local key patterns of acoustic features (such as the feature pattern corresponding to a sudden increase in volume). The second layer is an LSTM layer (64 hidden units) to capture the temporal dependencies of features (such as the interrogative tone corresponding to a sustained rise in intonation). The third layer is a fully connected layer (the number of output units equals the total number of non-text feature categories, such as 3 categories of tone + 4 categories of emotion + 3 categories of emphasis = 10 categories, Softmax activation function) to output the probability of each category. The cross-entropy loss function (adapted to multi-class classification tasks) is adopted, and the optimizer is Adam (with an initial learning rate of 0.001, dynamically adjusted according to the validation set loss). The training epochs are set to 50 epochs. After each epoch, the model accuracy is evaluated on the validation set. If the validation set accuracy does not improve for 5 consecutive epochs, training is stopped to avoid overfitting.

[0030] In this embodiment, during the synchronous process of automatic speech recognition (ASR) generating recognized text, the trained emotion recognition sub-model is invoked to extract non-textual feature information of the speech in real time, such as... Figure 2 The diagram illustrates the process for extracting non-textual feature information.

[0031] Specifically, in S201, the speech and the recognized text are aligned in time. For example, when ASR is performed to generate the recognized text, the timestamp information of the speech is recorded simultaneously. For example, the speech is segmented into "vocabulary-level speech segments" according to the word boundaries of the recognized text (e.g., if the recognized text is "I want to know the price of product A", it is segmented into three vocabulary-level speech segments: "I want to know", "product A", and "price", with each segment labeled with the corresponding speech start and end time). This ensures that the non-text features extracted subsequently can be accurately associated with the specific content of the recognized text.

[0032] Next, in step S202, non-text features are extracted from the segmented lexical-level speech fragments and the complete speech, specifically including: Tone and emotion feature extraction: The acoustic features of the complete speech (processed according to the pre-acoustic feature engineering process) are input into the emotion recognition sub-model. The model outputs the probability distribution of tone category (such as "question") and emotion category (such as "hesitation"). The category with the highest probability (≥0.7) is selected as the final result. If the highest probability is <0.7, it is marked as tone unknown or emotion unknown to avoid misjudgment. Emphasis Feature Extraction: For each lexical-level speech segment, extract its energy features and compare them with the average energy of the complete speech. If the energy of a segment is ≥ 1.5 times the average energy, and the probability of the "emphasis" category output by the sub-model is ≥ 0.8, then the corresponding word of the segment is determined to be an "emphasis word" (e.g., the energy of the "price" segment exceeds the standard, and it is marked as "emphasis-price"); if the energy of the entire speech segment is ≥ 1.2 times the average energy, then it is determined to be a sentence-level emphasis and marked as "emphasis-entire sentence"; if no segment energy exceeds the standard, it is marked as "no emphasis".

[0033] S203, the extracted non-text features are bound to the identified text to form a structured output of identified text and non-text feature labels, exemplarily including, The label format is defined using the format "[feature type: feature value]", such as tone label "[tone: question]", emotion label "[emotion: hesitation]", and emphasis label "[emphasis: price]", to ensure that the labels are clear and parsable; Association rules attach tone and emotion tags to the beginning of the recognized text (reflecting the attributes of the whole sentence), and emphasis tags are attached to the corresponding words (reflecting local attributes). For example, after recognizing the text "I want to know the price of product A" and combining it with non-text features, the output is "[tone: question][emotion: hesitation] I want to know the price of product A [emphasis: price]"; Output synchronization: Ensure that the non-text feature extraction and ASR generation of recognized text are completed synchronously, with a time difference of ≤100ms (to avoid affecting the real-time performance of multimodal dialogue). Finally, the recognized text and non-text feature labels are jointly fed into the subsequent speech label generation stage.

[0034] Preferably, the non-textual feature information further includes prosodic features, wherein the prosodic features include at least one of intonation variation, pause length, or volume intensity.

[0035] Specifically, intonation changes directly reflect the user's tone (such as questioning or affirmation), and are extracted by analyzing the changes in the fundamental frequency (F0, which determines pitch). The specific process is as follows: 1. Baseband data acquisition and preprocessing For example, the fundamental frequency (F0) is extracted frame by frame from the standardized speech segment. The fundamental frequency range is set to 50-500Hz (covering the normal fundamental frequency range of adult male and female voices). Invalid frames with a fundamental frequency of 0 (corresponding to silent segments) are removed to obtain the fundamental frequency value of each frame. The fundamental frequency smoothing process employs a moving average filter (window size of 5 frames) to process the raw fundamental frequency data, eliminating noise in the fundamental frequency fluctuations (such as fundamental frequency abrupt changes caused by sudden airflow) and obtaining a smooth fundamental frequency variation curve. 2. Intonation change pattern recognition Key node localization involves locating the fundamental frequency values ​​of key nodes on the fundamental frequency change curve, including the speech start frame, speech end frame, and frame before the semantic pause. The fundamental frequency difference and rate of change between nodes are then calculated (rate of change = (fundamental frequency of the next node - fundamental frequency of the previous node) / number of frames between the two nodes). The type of intonation change is determined based on the fundamental frequency change rate. For example, if the difference between the fundamental frequency of the end frame and the fundamental frequency of the start frame is greater than 50 Hz and the change rate is greater than 2 Hz / frame, it is determined as "rising intonation" (common in interrogative tones); if the difference is less than -30 Hz and the change rate is less than -1.5 Hz / frame, it is determined as "falling intonation" (common in affirmative tones); if the absolute value of the difference is less than 20 Hz and the absolute value of the change rate is less than 0.5 Hz / frame, it is determined as "stable intonation" (common in declarative tones).

[0036] Specifically, the length of the pause reflects the user's thinking state (such as hesitation or fluency), which is extracted by analyzing the duration of silent segments in the speech. The specific process is as follows: 1. Silent segment detection Energy calculation involves calculating the short-time energy of each frame of the standardized speech segment (energy per frame = sum of squares of all sampling points in that frame / frame length) to obtain the energy change curve. The silence threshold is determined by setting the silence energy threshold to 15% of the average energy based on the average energy of the speech segment. Frames with short-term energy less than the threshold are judged as silence frames, and consecutive silence frames form a silence segment. 2. Pause Length Classification and Filtering Length calculation: Count the number of frames in each silent segment, and calculate the actual duration of the silent segment (duration = number of frames × 20ms) by combining the frame length (20ms). For example, the length of the pause is divided into three categories: a pause of 0.2-0.5s is a "short pause" (normal semantic interval, no special meaning); a pause of >0.5s and <2s is a "long pause" (reflecting user hesitation or thinking); and a pause of ≥2s is an "extra-long pause" (may be a speech interruption, which should be marked as an invalid pause). "Extremely long pauses" and "silent segments at the beginning and end of speech" are removed, and only "short pauses" and "long pauses" within the speech are retained. The start and end times of each valid pause are recorded to provide a temporal basis for subsequent association with the recognized text.

[0037] Specifically, volume reflects the user's emphasis (such as highlighting key information), which is extracted by analyzing changes in speech energy. The specific process is as follows: 1. Energy characteristic calculation Short-time energy extraction involves calculating the short-time energy of each frame of the standardized speech segment (using the same energy calculation method as in pause length extraction) to obtain the energy value for each frame. Energy normalization is achieved by dividing the energy value of all frames by the average energy of the speech segment to obtain the normalized energy (value range 0-5), which eliminates the influence of the difference in the absolute value of the speech volume of different users (such as the energy comparison between loud and soft users). 2. Volume strength mode recognition Energy peak localization involves locating consecutive frames with energy values ​​greater than 2.5 (i.e., 1.5 times higher than the average energy) in the normalized energy curve, forming an "energy peak segment," which corresponds to the part of the speech where the volume increases. If a speech segment contains an "energy peak segment" and the peak segment duration accounts for more than 15% of the total speech duration, it is judged as "volume increase" (reflecting the user emphasizing a certain part of the content); if the normalized energy of all frames is <0.8 (i.e., below 20% of the average energy) and there is no energy peak segment, it is judged as "volume decrease" (reflecting the user's tone is gentle); if the normalized energy is mostly between 0.8 and 1.2 and there is no obvious peak segment, it is judged as "volume stable"; By combining the temporal information of the recognized text generated by ASR (such as word boundary timestamps), the "energy peak segment" is mapped to the specific words in the recognized text (such as the peak segment corresponding to the speech "price", then "price" is marked as an emphasized word), thus realizing the binding of volume strength features with text content.

[0038] Preferably, the extracted intonation changes, pause lengths, and volume intensity prosodic features are integrated to form structured prosodic feature information, providing input for subsequent generation of speech tags and construction of semantic compensation tags of historical text.

[0039] Specifically, for example, features are recorded in the format of [prosodic feature type: feature value: associated text position], such as "[intonation change: rising: whole sentence]", "[pause length: long pause: after 'scheme']", "[volume strength: increase: 'price']", where "associated text position" clearly defines the recognition text range (whole sentence or specific words) corresponding to the feature, ensuring that the correlation between the feature and the text is clear; The integrated prosodic features are output synchronously with the ASR-generated recognition text and other non-textual feature information extracted in the aforementioned embodiments to ensure alignment in the time dimension.

[0040] For example, by combining the text timestamps output by ASR, "long pauses" (prosodic features) are associated with the word "solution" in the recognized text, and "hesitation" is simultaneously marked; "volume enhancement" (prosodic features) is associated with the word "price", and "emphasis-price" is simultaneously marked, ensuring that non-textual information is accurately bound to textual content.

[0041] In some embodiments, for S103, in order to avoid redundant information increasing the burden on the model input, before compression encoding, the extracted non-text features (including tone, emotion, emphasis and prosody features) are first screened, and only features that have core value for dialogue understanding are retained.

[0042] Specifically, non-textual features that cannot reflect user intent or emotion should be removed, such as short pauses (0.2-0.5s), steady volume, and neutral declarative tone in prosodic features. These features do not help the model understand the dialogue, and retaining them will increase coding redundancy. Retain features that are directly related to the user's core needs or deep state, such as emphasis features (e.g., "emphasis-price"), emotion features (e.g., "anger" "hesitation"), and key prosodic features (e.g., "sentence-end intonation rises → question tone"). The absence of these features will lead to the model misjudging the user's intent. In this embodiment, preferably, the filtered features are sorted according to their degree of impact on the accuracy of model inference to avoid coding confusion when multiple features conflict. For example, First priority, such as emphasis-based features (e.g., "emphasis-price"), which are directly related to content that users are most concerned about, should be encoded first; The second priority is, for example, emotional characteristics (such as "hesitation" or "anger"), which reflect the user's deep state and influence the choice of response strategy; The third priority, for example, is key tone / prosodic features (such as "interrogative tone" and "long pauses"), which help to understand the user's expressed intention; If a single speech tag corresponds to multiple non-text features (such as "emphasis-price and interrogative tone"), the first two core features are retained according to priority (to avoid excessively long encoded labels), thus shortening the token sequence and improving efficiency.

[0043] In this embodiment, the voice tags follow the compression coding principle of short sequences and high semantic density, transforming the filtered non-text features into a concise and parsable tag format.

[0044] Specifically, to ensure that the model can quickly identify label types, a structured encoding format of [feature category: core information] is designed, for example, where: The feature category should be clearly defined using 2-3 Chinese characters, such as "emphasis", "emotion", "tone", and "rhythm". The core information conveys key content with the fewest characters, such as highlighting the corresponding words with emphasis features ("price"), labeling the specific category with emotion features ("hesitation"), labeling the type with tone features ("question"), and labeling the key expression with prosodic features ("long pause"). For example, the emphasis feature is encoded as “[Emphasis: Price]”, the emotion feature is encoded as “[Emotion: Hesitation]”, the interrogative tone is encoded as “[Tone: Interrogative]”, and the long pause prosodic feature is encoded as “[Prosodic: Long Pause]”.

[0045] Preferably, for scenarios where a single speech corresponds to multiple non-text features, a merging encoding rule is designed to further compress the label length. For example, features of the same category are merged. If multiple features belong to the same category (such as "long pause and rising intonation" in prosodic features), they are merged into a single label and encoded as "[prosodic: long pause and rising intonation]", instead of "[prosodic: long pause] and [prosodic: rising intonation]". Cross-category feature simplification: If features belong to different categories and have different priorities (such as "emphasis-price and interrogative tone"), retain the core information of the features with higher priority and merge them into "[emphasis: price|tone: interrogative]", using "|" to separate different categories to avoid increasing the number of labels; Optionally, the total number of characters in the voice tag corresponding to a single voice message is ≤15 (e.g., “[emphasis: price|emotion: hesitation]” has 13 characters), ensuring that the tag length is much shorter than the original voice token (the file mentions that the voice token is much longer than the text token due to the large number of frames), thus maintaining the advantage of short sequences.

[0046] In this embodiment, the compressed and encoded speech tags are accurately appended to the corresponding positions in the recognized text to ensure that the model can associate the tags with the text content. The specific implementation is as follows: First, based on the process of generating recognized text using Automatic Speech Recognition (ASR), the following timing information is recorded synchronously to provide a basis for alignment: This word timestamp is used by ASR to mark the "speech start time - end time" of each word when generating the recognized text (e.g., in the recognized text "I want to know the price of product A", the speech timestamp for "price" is "3.2s-3.8s"). Non-text feature timestamps: When extracting non-text features, record the "voice timestamp" of the feature occurrence (e.g., "emphasis-price" corresponds to a voice timestamp of "3.2s-3.8s", and "hesitation emotion" corresponds to a voice timestamp of "2.5s-2.8s"). The timestamps of the speech tags are matched with the timestamps of the text words. If the overlap is ≥80%, the tag is determined to correspond to the word. If the feature covers the entire speech segment (such as "interrogative tone"), the tag is determined to correspond to the entire sentence of recognized text.

[0047] Next, based on the type of non-textual features, the additional positions of the speech tags in the recognized text are determined to ensure clear semantic associations. For emphasis tags, for example, they can be appended to the corresponding words. For instance, if the text "price" corresponds to "[emphasis: price]", the appended text becomes "price[emphasis: price]", which clearly indicates the specific content that the user is emphasizing. For emotion-related tags, for example, they can be appended before the corresponding semantic fragment. For instance, the "hesitation emotion" corresponding to the identified text "I think this solution" becomes "[emotion: hesitation] I think this solution" after appending, reflecting the user's state when expressing that fragment. Tone / prosody tags, for example, can be appended to the beginning of the entire sentence recognition text. For instance, "interrogative tone" corresponds to the recognition text "Is this solution feasible?", which, after appending, becomes "[tone: interrogative] Is this solution feasible?", reflecting the expressive intent of the entire sentence; For example, if a single sentence needs to be appended with multiple tags (such as "[tone: question] and [emotion: hesitation]"), arrange them in the order of "tone / prosody tag → emotion tag → emphasis tag" to avoid confusion. Example: "[tone: question][emotion: hesitation] I want to know the price [emphasis: price]".

[0048] Finally, the identified text and voice tags are integrated to form historical text information with semantic compensation markers, specifically, The initial format of the historical content was "voice-text-voice-text". After being replaced with recognized text and tagged, the format was upgraded to "(recognized text + voice tag)-(model text)-(recognized text + voice tag)-(model text)".

[0049] In some embodiments, for S104-S105, preferably, the voice tags are input into the multimodal dialogue model as independent tokens, without occupying the number of recognition text tokens.

[0050] The total number of tokens in the combined content (text tokens from historical text + tag tokens + acoustic feature tokens from the current speech) is counted to ensure that the total number of tokens is less than or equal to the context window threshold of the multimodal dialogue model (e.g., 2048 tokens). If the threshold is exceeded, the earliest dialogue turn in the historical text is truncated first (keeping the most recent 3-5 core dialogue turns), while the acoustic feature dimension of the current speech is compressed (e.g., reducing the MFCC feature from 40 dimensions to 20 dimensions) to ensure that the input sequence length is compliant.

[0051] The combined historical text and current speech are input into a multimodal dialogue model, which then outputs a corresponding text response through the model's cross-modal understanding and generation capabilities.

[0052] Specifically, the first step is to convert the combined content into a format that the model can recognize, for example, Text modal input: The sequence of "user text (including tags) - model text" in historical text information is converted into a sequence of text tokens supported by the model. Voice tags (such as "[emphasis: price]") are treated as "special tokens". These tag tokens are pre-registered in the model's vocabulary to ensure that the model can recognize their semantics (rather than splitting them into ordinary character tokens). Speech modal input: The basic acoustic features of the current user's speech are converted into speech feature tensors supported by the model, with the dimensions matching the speech input layer of the model (e.g., if the model's speech input layer is "time step × feature dimension", then the acoustic features are adjusted to tensors of the corresponding dimensions); if the model supports "direct input of speech waveform", then the standardized speech WAV file is converted into a waveform tensor to avoid feature loss.

[0053] Preferably, in the cross-modal attention layer of the model, the attention weight of the current speech features is increased to ensure that the model prioritizes the immediate needs of the current user; at the same time, the attention weight of speech tag tokens in historical text (such as "[emphasis: price]") is also increased to help the model quickly locate the core information in historical dialogues (such as "price" that the user previously focused on). An incremental inference strategy is adopted, in which the model only calculates the inference part corresponding to the current speech, reuses the intermediate results of inference of historical text information (such as the semantic encoding vector of historical text), avoids recalculating the complete historical text for each inference, and improves the inference speed (shortening the inference time). At the same time, the model's context memory module is enabled to store the core semantics of historical text information (such as the type of product the user is interested in and their price requirements) in the memory unit. When reasoning about the current voice response, the memory information is directly called to ensure that the response is consistent with the logic of the historical dialogue (such as if the user asked about the price of product A before, and now asks "Are there any discounts?", the model can clearly understand that "discount" refers to "discount on the price of product A").

[0054] After the model inference is completed, a text response corresponding to the current user's voice is generated. Preferably, the voice tag can be converted into a prompting control signal during the model output stage to guide the generated response to have a specific tone of soothing, encouraging or emphasizing.

[0055] Check if the reply responds to non-textual features in the historical text (e.g., if the user has a "hesitant" emotion tag in the historical text, the reply should include reassuring statements, such as "If you still have questions about product B, please explain in detail"); if it does not respond, supplement the corresponding statements and output the reply to ensure that the reply matches the user's emotions and tone; The generated text responses are converted into natural and fluent conversational expressions, avoiding stiff machine language. The response length is controlled (1-3 sentences, ≤20 characters per sentence) to match the interactive rhythm of multi-turn dialogues. The final output text response is (e.g., "Product B currently has no direct discounts, but you can enjoy a 10% discount when you buy 2 items. Would you like to know more about the promotion?"). It is understood that the expression style can be adapted to specific application scenarios, and this invention does not impose any limitations.

[0056] Figure 3 An inference device 300 based on text history in a multimodal dialogue scenario is shown. This device embodiment is similar to... Figure 1 Corresponding to the method embodiment shown, this device can be specifically applied to various electronic devices, specifically including, The acquisition module 301 is used to acquire historical dialogues that alternate between speech and text, and to perform automatic speech recognition on the speech portion of the historical dialogues to obtain the recognized text. The non-text feature information extraction module 302 is used to extract non-text feature information of the speech during the process of generating the recognized text. The non-text feature information includes at least one of tone, emotion, or emphasis. The compensation tagging module 303 is used to compress and encode the non-text feature information to form a speech tag and attach it to the corresponding recognized text to obtain historical text information with semantic compensation tags. The combination module 304 is used to combine the historical text information with the voice input by the current user to form the model input content; The inference module 305 is used to input the model input content into the multimodal dialogue model for inference and output a text response corresponding to the voice input by the current user.

[0057] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0058] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and smart bands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0059] Figure 4 The diagram illustrates the structure of an apparatus suitable for implementing the methods and / or technical solutions in the embodiments of this application. The apparatus 400 includes a central processing unit 401, which can perform various appropriate actions and processes based on a program stored in a read-only memory 402 or a program loaded from a storage portion 408 into a random access memory 403. Various programs and data required for system operation are also stored in the RAM 403. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0060] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 407 including cathode ray tubes, liquid crystal displays, LED displays, OLED displays, etc., and speakers, etc.; a storage section 408 including one or more computer-readable media such as hard disks, optical disks, magnetic disks, semiconductor memory, etc.; and a communication section 409 including network interface cards such as LAN cards, modems, etc. The communication section 409 performs communication processing via a network such as the Internet.

[0061] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 401, it performs the functions defined in the methods of this application.

[0062] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0063] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0064] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0065] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0066] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0072] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0074] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A text history based reasoning method in a multi-modal dialogue scenario, characterized in that, The method comprises the following steps: obtain historical dialogue containing voice and text alternately, and perform automatic speech recognition on the voice part in the historical dialogue to obtain recognized text; extract non-text feature information of the voice during the generation of the recognized text, the non-text feature information at least including one of tone, emotion or emphasis; compress and encode the non-text feature information to form voice tags and attach them to the corresponding recognized text to obtain historical text information with semantic compensation marks; combine the historical text information with the current user input voice to form model input content; input the model input content into a multi-modal dialogue model for reasoning, and output a text reply corresponding to the current user input voice.

2. The text history-based reasoning method in a multi-modal dialogue scene according to claim 1, wherein the non-text feature information further includes prosodic features, and the prosodic features include at least one of intonation variation, pause length or volume strength.

3. The text history-based reasoning method in a multi-modal dialogue scene according to claim 1, wherein the non-text feature information is extracted by an emotion recognition sub-model, and the emotion recognition sub-model is obtained based on acoustic feature training and is used to output corresponding emotion categories or tone categories.

4. The text history-based reasoning method in a multi-modal dialogue scene according to claim 1, wherein the multi-modal dialogue model performs attention weighting based on the voice tags when generating the text reply, so as to improve the response capability to historical tone or emotion.

5. The text history-based reasoning method in a multi-modal dialogue scene according to claim 1, wherein the voice tags can be converted into prompt control signals in the model output stage, which are used to guide the generated reply to have specific tones such as appeasement, encouragement or emphasis.

6. The text history-based reasoning method in a multi-modal dialogue scene according to claim 1, wherein the voice tags are input into the multi-modal dialogue model in the form of independent tokens, without occupying the number of recognized text tokens. comprise: an acquisition module, configured to obtain historical dialogue containing voice and text alternately, and perform automatic speech recognition on the voice part in the historical dialogue to obtain recognized text; a non-text feature information extraction module, configured to extract non-text feature information of the voice during the generation of the recognized text, the non-text feature information at least including one of tone, emotion or emphasis; a compensation mark module, configured to compress and encode the non-text feature information to form voice tags and attach them to the corresponding recognized text to obtain historical text information with semantic compensation marks; a combination module, configured to combine the historical text information with the current user input voice to form model input content; 7. A reasoning device based on text history in a multimodal dialogue scenario, characterized in that, a reasoning module, configured to input the model input content into a multi-modal dialogue model for reasoning, and output a text reply corresponding to the current user input voice.

8. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein, ​ ​ ​ ​ ​ The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of any one of claims 1-6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-6.

Images