Training method and device of multi-modal speech understanding model, speech understanding method and device

Abstract

The application provides a training method, a speech understanding method and device of a multi-modal speech understanding model, and relates to the technical field of cargo transportation. The training method comprises obtaining target text intention sample data of a logistics scene, and collecting real driver speech data; a timbre library of the real driver speech data is constructed; the target text intention sample data and the timbre library are fused, and multi-source speech training data is generated based on the fusion result; a first multi-modal large model is used to perform offline inference on real speech service data of the logistics scene, and distillation data is obtained; an initial multi-modal small model is trained based on the multi-source speech training data and the distillation data, and a multi-modal speech understanding model is obtained. The scheme of the application can reduce the training and deployment cost of the model, and make the model lightweight and land-based.

Description

Technical Field

[0001] This application relates to the field of cargo transportation technology, and in particular to a training method, speech understanding method and apparatus for a multimodal speech understanding model. Background Technology

[0002] In actual logistics scenarios, drivers prefer to communicate by phone to quickly confirm key information such as cargo availability, transportation needs, and prices. Therefore, a large amount of business interaction information exists in voice form.

[0003] In related technologies, semantic understanding and modeling of speech data can be implemented through two main approaches: one is a cascaded solution of "Automatic Speech Recognition (ASR) + large text model"; the other is an end-to-end multi-modal large model solution. The "ASR + large text model" cascaded solution relies on ASR to transcribe speech into text, and then the large text model performs semantic understanding and dialogue generation on the transcribed text. However, in freight scenarios, drivers' accents vary significantly, making this method prone to transcription errors and error accumulation. Furthermore, paralinguistic information such as tone, speech rate, and pauses are easily overlooked during transcription, leading to the loss of key speech information. Therefore, choosing the end-to-end multi-modal large model solution directly models the relationship between speech and semantics, addressing the problems encountered with the "ASR + large text model" cascaded solution.

[0004] However, existing end-to-end multimodal large models rely on large-scale parameter models to achieve good results, which leads to a significant increase in training and inference costs as the model size increases. Furthermore, they are difficult to deploy on a large scale in actual logistics operations, especially in real-time voice interaction scenarios where cost issues are even more prominent. Summary of the Invention

[0005] This application provides a training method, a speech understanding method, and an apparatus for a multimodal speech understanding model, in order to solve the problems of high training costs and lack of lightweight deployment capabilities of large multimodal models used for semantic recognition in related technologies.

[0006] Firstly, this application provides a training method for a multimodal speech understanding model, comprising: Acquire sample data of target text intent in logistics scenarios and collect real driver voice data; Construct a timbre library of the real driver's voice data; The target text intent sample data and the timbre library are fused together, and multi-source speech training data is generated based on the fusion result; Offline inference was performed on real voice business data in logistics scenarios using the first multimodal large model to obtain distilled data; The initial multimodal mini-model is trained based on the multi-source speech training data and the distillation data to obtain a multimodal speech understanding model.

[0007] In one optional design, training an initial multimodal mini-model based on the multi-source speech training data and the distillation data to obtain a multimodal speech understanding model includes: The initial multimodal small model is trained using the distillation data to achieve capability alignment, thereby obtaining the first multimodal small model. The first multimodal mini-model is trained with full parameter fine-tuning using the multi-source speech training data to obtain the second multimodal mini-model; The multi-source speech training data is input into the second multimodal small model to obtain the first prediction result output by the second multimodal small model; The first prediction result is checked for consistency with the label data of the multi-source speech training data, and the multi-source speech training data is optimized based on the consistency check result to obtain the target training sample data. The second multimodal mini-model is trained using the target training sample data to obtain the multimodal speech understanding model.

[0008] In an optional design, the step of performing a consistency check between the first prediction result and the label data of the multi-source speech training data, and optimizing the multi-source speech training data based on the consistency check result to obtain target training sample data, includes: The consistency verification result is obtained by comparing the first prediction result with the label data of the multi-source speech training data. The data in the multi-source speech training data whose consistency check results are consistent are determined as the first training sample data, and the data in the multi-source speech training data whose consistency check results are inconsistent are determined as the second training sample data. The second training sample data is input into at least two second multimodal large models to obtain the second prediction result output by each second multimodal large model; Voting is performed on the second prediction results output by each of the second multimodal large models, and the second training sample data is updated according to the voting results to obtain the third training sample data; The first training sample data and the third training sample data are integrated to obtain the target training sample data.

[0009] In one optional design, constructing the timbre library of the real driver's voice data includes: The real driver's voice data is anonymized and segmented to obtain initial voice data; The initial speech data is subjected to timbre feature extraction to obtain the target timbre features; The target timbre features are aggregated to obtain the timbre library of the real driver's voice data.

[0010] In one optional design, the fusion of the target text intent sample data and the timbre library, and the generation of multi-source speech training data based on the fusion result, includes: The target text intent sample data and the timbre library are fused using timbre cloning technology to obtain multimodal first speech training data; The first speech training data is enhanced to obtain the second speech training data. The first speech training data and the second speech training data are integrated to obtain the multi-source speech training data.

[0011] In one optional design, obtaining target text intent sample data for a logistics scenario includes: Obtain initial textual intent data for the logistics scenario; The initial text intent data is structured to obtain semantically standardized target text intent sample data.

[0012] Secondly, this application provides a speech understanding method, including: Acquire voice data from logistics scenarios; The voice data of the logistics scenario is input into the multimodal speech understanding model to obtain the speech understanding information output by the multimodal speech understanding model. The multimodal speech understanding model is trained based on the training method of the multimodal speech understanding model as described in any of the first aspects above.

[0013] Thirdly, this application provides a training device for a multimodal speech understanding model, comprising: The data acquisition module is used to acquire target text intent sample data in logistics scenarios and collect real driver voice data; The timbre library construction module is used to construct the timbre library of the real driver's voice data; The data fusion module is used to fuse the target text intent sample data and the timbre library, and generate multi-source speech training data based on the fusion result; The data inference module is used to perform offline inference on real voice business data in logistics scenarios using the first multimodal large model to obtain distilled data; The model training module is used to train the initial multimodal mini-model based on the multi-source speech training data and the distillation data to obtain a multimodal speech understanding model.

[0014] Fourthly, this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the training method for the multimodal speech understanding model as described in any of the first aspects above, or to implement the speech understanding method as described in the second aspect above.

[0015] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the multimodal speech understanding model as described in any of the first aspects above, or implements the speech understanding method as described in the second aspect above.

[0016] The multimodal speech understanding model training method, speech understanding method, and apparatus provided in this application first acquire target text intent sample data from a logistics scenario and collect real driver speech data to construct a voice library of real driver speech data. Then, the target text intent sample data and the voice library are fused, and multi-source speech training data is generated based on the fusion result. This method can reverse-generate speech data with intent labels from existing text data, and construct large-scale speech training data with intent labels at low cost. Then, the first multimodal large model is used to perform offline inference on real speech business data from a logistics scenario to obtain distilled data. Finally, the initial multimodal small model is trained based on the multi-source speech training data and the distilled data to obtain the multimodal speech understanding model, thus realizing the training of the multimodal speech understanding model. In this way, through knowledge distillation, the capabilities of the large-scale model (the first multimodal large model) can be transferred to the small-scale model (the initial multimodal small model). This allows the small-scale model to approximate the performance of the large-scale model while maintaining a low inference cost, thereby reducing the training and deployment costs of the multimodal speech understanding model and enabling the model to be deployed in a lightweight manner. Moreover, the multi-source speech training data used in the model training process is generated by reverse engineering from existing text data, which makes up for the lack of intent-labeled speech training data and the extremely high labeling cost in the field of freight transportation, further reducing the training cost of the model. Attached Figure Description

[0017] Figure 1 One of the flowcharts illustrating the training method for the multimodal speech understanding model provided in this application embodiment; Figure 2 A second schematic flowchart illustrating the training method for the multimodal speech understanding model provided in this application embodiment; Figure 3A flowchart illustrating the speech understanding method provided in this application embodiment; Figure 4 A schematic diagram of the structure of a training device for a multimodal speech understanding model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the speech understanding device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c alone can mean: a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c. a, b, and c can be single or multiple. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0019] The terms “center,” “longitudinal,” “lateral,” “up,” “down,” “left,” “right,” “front,” and “rear,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0020] The terms "connected" and "connected" should be interpreted broadly. For example, in circuit structures, "connected" or "connected" can refer not only to physical connections but also to electrical or signal connections. This could be a direct connection (physical connection) or an indirect connection via at least one intermediate component, as long as the circuit is connected. It could also refer to the internal connection between two components. Similarly, a signal connection can refer to a connection via a circuit or a medium, such as radio waves. Those skilled in the art will understand the specific meaning of these terms in this application based on the specific circumstances.

[0021] In actual logistics scenarios, drivers prefer to communicate by phone to quickly confirm key information such as cargo availability, transportation needs, and prices. Therefore, a large amount of business interaction information exists in the form of voice.

[0022] The cascaded solution of "ASR + text big model" relies on Automatic Speech Recognition (ASR) technology to first transcribe speech into text, and then the text big model performs semantic understanding and dialogue generation. However, this method has the following problems: (1) In the context of freight transportation, drivers' accents vary significantly, and a large number of technical terms and complex environmental noise are involved, which leads to a decrease in the recognition accuracy of ASR. Once an error occurs in the transcription, it will be directly transmitted to the subsequent model, resulting in a deviation in semantic understanding. Moreover, this method lacks an effective error correction mechanism, which ultimately leads to the propagation and accumulation of errors; (2) Paralinguistic information such as tone, speed, and pauses in speech are easily ignored during the transcription process. This loss of key speech information will affect the accurate judgment of the user's true intention. Therefore, the cascaded solution of "ASR + text big model" cannot achieve a good semantic understanding effect in complex speech environments.

[0023] Based on this, in one embodiment, speech can be used as input, and an end-to-end multi-modal large model can be employed to recognize and understand the speech. This directly models the relationship between speech and semantics, solving the problems encountered in the cascaded "ASR + text large model" scheme. Compared to the cascaded "ASR + text large model" scheme, it has stronger semantic understanding capabilities and greater development potential. However, when using end-to-end multimodal large models for semantic understanding, there are still the following shortcomings: (1) End-to-end multimodal large models rely on large-scale parameter models to achieve better results, which leads to a significant increase in training and inference costs as the model size increases. Furthermore, it is difficult to deploy on a large scale in actual logistics business and lacks lightweight implementation capabilities. In particular, the cost problem is more prominent in real-time voice interaction scenarios. (2) End-to-end multimodal large models rely on a large amount of labeled voice data for training. However, in the field of cargo transportation, the acquisition of real call data is restricted by privacy protection and compliance requirements, and the cost of labeling data is high, resulting in a lack of large-scale, high-quality "voice + intent label" data, which seriously restricts the improvement of model performance. (3) Voice data mainly relies on the collection of real scenarios or a small amount of labeled data. The data construction method is simple and it is difficult to cover the multi-noise environment, multi-accent differences and complex interaction scenarios in actual business, resulting in insufficient robustness of the model in the real environment.

[0024] In view of this, embodiments of this application provide a training method for a multimodal speech understanding model for logistics voice dialogue scenarios. On the one hand, it reverse-engineers intent-labeled speech data from existing text data in the logistics scenario, constructing large-scale intent-labeled speech training data at low cost. Then, based on this speech training data and knowledge distillation technology, while controlling the model inference cost, it transfers the semantic understanding capabilities of large-scale multimodal models to small-scale models, achieving "high performance + low cost" and practical multimodal semantic understanding capabilities. On the other hand, it designs a multi-stage collaborative model fine-tuning strategy and introduces a pseudo-label optimization mechanism based on model prediction consistency and multi-model voting, which can improve data quality and model generalization ability.

[0025] The training method for the multimodal speech understanding model provided in this application can be applied to electronic devices or to a training device for the multimodal speech understanding model installed in such electronic devices. This training device can be implemented through software, hardware, or a combination of both. The electronic devices may include at least one of the following, but are not limited to: servers, mobile phones, computers, in-vehicle terminals, tablet computers, wearable devices, smart home devices, augmented reality (AR) devices, and virtual reality (VR) devices. Servers may include independent servers, virtual servers, and cluster servers.

[0026] The following example uses an electronic device as the executing entity, combined with... Figures 1-2 The training method for the multimodal speech understanding model provided in the embodiments of this application will be described in detail.

[0027] Figure 1 This illustration shows one of the flowcharts of a training method for a multimodal speech understanding model provided in an embodiment of this application. (Refer to...) Figure 1 As shown, the training method for this multimodal speech understanding model may include the following steps 110-150.

[0028] Step 110: Obtain target text intent sample data in the logistics scenario and collect real driver voice data.

[0029] The target text intent sample data can be text intent data obtained from logistics scenarios, or text intent data obtained from logistics scenarios after semantic standardization. Real driver voice data consists of real driver business interaction voices collected from online operations, such as voice data of drivers communicating via telephone on freight transportation platforms to confirm cargo information, transportation needs, and prices.

[0030] In one embodiment, obtaining target text intent sample data for a logistics scenario may include: obtaining text intent data for the logistics scenario from a text intent classification database, and determining the text intent data as target text intent sample data.

[0031] In another embodiment, obtaining target text intent sample data for a logistics scenario may include: obtaining initial text intent data for the logistics scenario; and performing structured processing on the initial text intent data to obtain semantically standardized target text intent sample data.

[0032] Specifically, initial text intent data for logistics scenarios can be obtained from a text intent classification database. The text intent classification database is used to store the text intent data for logistics scenarios.

[0033] It is understandable that electronic devices can randomly obtain the required amount of text intent data from the text intent classification database, or obtain text intent data from the text intent classification database based on set rules (such as the principle of full type coverage).

[0034] Step 120: Construct a timbre library of real driver voice data.

[0035] After obtaining real driver voice data, you can directly extract timbre features from the data, and then aggregate and model these features to obtain a timbre library of real driver voice data. Alternatively, you can first perform preprocessing such as anonymization and noise reduction on the obtained real driver voice data, then extract timbre features, and finally aggregate and model these features to obtain a timbre library of real driver voice data.

[0036] For example, in one embodiment, constructing a timbre library of real driver voice data may include: performing voice desensitization and voice segmentation on the real driver voice data to obtain initial voice data; extracting timbre features from the initial voice data to obtain target timbre features; and aggregating the target timbre features to obtain a timbre library of real driver voice data.

[0037] The extracted target timbre features may include accent features, speech rate features, noise level features, and driver attribute features, which may include the driver's gender, age, and region.

[0038] For example, the target timbre features can be aggregated in different feature combinations to obtain a timbre library of real driver voice data.

[0039] For example, the constructed timbre library can cover all different types of timbre features, which can be combined in any form, and each timbre is evenly distributed. That is, the constructed timbre library has the characteristics of diversity and uniformity of each timbre. For example, the timbre library can include accents A, B, and C, with an equal number of audio samples for each accent, and each accent's audio samples can correspond to different combinations of features such as speech rate features, noise level indicators, and driver attribute features.

[0040] In this way, by building a timbre library of real driver voice data, a diverse range of timbre resources can be provided for subsequent voice generation.

[0041] Step 130: Fuse the target text intent sample data and the timbre library, and generate multi-source speech training data based on the fusion result.

[0042] In one embodiment, fusing target text intent sample data and a voice library, and generating multi-source speech training data based on the fusion result, may include: fusing target text intent sample data and a voice library using voice cloning technology to obtain multimodal first speech training data; and determining the first speech training data as multi-source speech training data.

[0043] In another embodiment, fusing the target text intent sample data and the timbre library, and generating multi-source speech training data based on the fusion result, may include: fusing the target text intent sample data and the timbre library using timbre cloning technology to obtain multimodal first speech training data; enhancing the first speech training data to obtain second speech training data; and integrating the first speech training data and the second speech training data to obtain multi-source speech training data.

[0044] Voice cloning technology refers to the technique of analyzing the voice characteristics of a specific person using artificial intelligence algorithms to generate a highly similar "replica voice". By fusing target text intent sample data and voice library through voice cloning technology, the target text intent sample data can be transformed into speech data with a "specified driver voice", thereby generating large-scale, intent-labeled, multimodal first speech training data.

[0045] Enhancement processing of the first speech training data may include at least one of the following, but is not limited to: background noise separation, background noise reconstruction, adding noise (including environmental noise and call noise), speech rate perturbation, and frequency perturbation.

[0046] In this way, by enhancing the first speech training data, the diversity of the data can be improved. Then, by integrating the first speech training data and the second speech training data, the coverage of the data in real and complex scenarios can be improved, resulting in multi-source speech training data with strong coverage.

[0047] Step 140: Use the first multimodal large model to perform offline inference on real voice business data in the logistics scenario to obtain distilled data.

[0048] Among them, the first type of multimodal large language model (MLLMs) is a class of artificial intelligence models that can process and understand multiple data modalities simultaneously, such as DeepSeek and Wenxin Yiyan.

[0049] Electronic devices can acquire real-world voice business data from logistics scenarios and input this data into a first multimodal large-scale model. The first multimodal large-scale model then performs offline semantic understanding on this data, generating high-quality semantic label data and intermediate representation information. This semantic label data and intermediate representation information are identified as distilled data. The intermediate representation information represents the process information of the first multimodal large-scale model's thinking and reasoning about the real-world voice business data.

[0050] Step 150: Train the initial multimodal small model based on multi-source speech training data and distillation data to obtain a multimodal speech understanding model.

[0051] After obtaining multi-source speech training data and distillation data, electronic devices can use these data to train an initial multimodal mini-model, which is then designated as the multimodal speech understanding model. The initial multimodal mini-model refers to an artificial intelligence model with a small number of parameters that can simultaneously process and understand multiple data modalities.

[0052] A multimodal small model is defined in contrast to a multimodal large model. In the embodiments of this application, a multimodal large model can refer to an artificial intelligence model with 1B to several hundredB (billions to hundreds of billions) of parameters that can simultaneously process and understand multiple data modalities. A multimodal small model can refer to an artificial intelligence model with 1M to 1B (millions to hundreds of millions) of parameters that can simultaneously process and understand multiple data modalities.

[0053] In one embodiment, training an initial multimodal mini-model based on multi-source speech training data and distillation data to obtain a multimodal speech understanding model may include: using distillation data to perform capability alignment training on the initial multimodal mini-model to obtain a first multimodal mini-model; using multi-source speech training data to perform full-parameter fine-tuning training on the first multimodal mini-model to obtain a second multimodal mini-model, and determining the second multimodal mini-model as the multimodal speech understanding model.

[0054] Among them, using distilled data to perform capability alignment training on the initial multimodal small model can be done through full parameter fine-tuning, enabling the initial small model to learn the semantic expression capabilities of the first multimodal large model, thereby obtaining high semantic understanding capabilities with a smaller parameter scale, and providing a model foundation for low-latency inference.

[0055] After obtaining the first multimodal small model, the multi-source speech training data generated in step 130 is used to further train the first multimodal small model with full-parameter fine-tuning, such as supervised fine-tuning (SFT) training. This can inject relevant intent understanding capabilities from the logistics field into the first multimodal small model, strengthen the joint modeling capability between speech and semantics, and enable the first multimodal small model to directly perform intent recognition and semantic understanding from speech. At the same time, the first multimodal small model continues to learn speech data covering multiple noise, multiple accents, and multiple scene conditions, thereby enabling the first multimodal small model to stably extract key information in complex environments and improve the semantic understanding capability and robustness of the finally trained multimodal speech understanding model.

[0056] In another embodiment, the initial multimodal small model is trained based on multi-source speech training data and distillation data to obtain a multimodal speech understanding model, which may include the following steps one to five.

[0057] Step 1: Use distillation data to perform capability alignment training on the initial multimodal small model to obtain the first multimodal small model.

[0058] Among them, the initial multimodal small model can be an artificial intelligence model with 1M~1B (millions to hundreds of millions) of parameters and capable of processing and understanding multiple data modalities simultaneously.

[0059] The electronic device can use the distillation data obtained in step 140 to perform full-parameter fine-tuning training on the initial multimodal small model, enabling the initial multimodal small model to learn the semantic representation ability of the first multimodal large model, ultimately obtaining a first multimodal small model that approximates the performance of the first multimodal large model. In this way, through model distillation, high semantic understanding ability can be obtained with a relatively small parameter scale, providing a model foundation for low-latency inference.

[0060] Step 2: Use multi-source speech training data to fine-tune the first multimodal small model with all parameters to obtain the second multimodal small model.

[0061] After model distillation is completed, the electronic device can use the multi-source speech training data generated in step 130 to continue full-parameter fine-tuning training of the first multimodal small model, injecting relevant intent understanding capabilities in the logistics field into the first multimodal small model, strengthening the joint modeling capability between speech and semantics. At the same time, the first multimodal small model continues to learn speech data covering multiple noise, multiple accents and multiple scene conditions, thereby enabling the trained second multimodal small model to still stably extract key information in complex environments, improving the semantic understanding capability and robustness of the trained second multimodal small model.

[0062] Step 3: Input the multi-source speech training data into the second multimodal small model to obtain the first prediction result output by the second multimodal small model.

[0063] After completing the two-stage model training based on distillation data and multi-source speech training data, the electronic device can enter the data optimization and retraining stage, using the second multimodal small model to infer the multi-source speech training data and obtain the first prediction result output by the second multimodal small model.

[0064] Step 4: Perform consistency verification between the first prediction result and the label data of the multi-source speech training data, and optimize the multi-source speech training data based on the consistency verification result to obtain the target training sample data.

[0065] The multi-source speech training data is a speech dataset with intent labels. After the electronic device obtains the first prediction result output by the second multimodal small model, it can perform a consistency check between this first prediction result and the label data of the multi-source speech training data. Then, based on the consistency check result, the multi-source speech training data can be filtered, dividing it into consistent data and inconsistent data. Consistent data is directly retained as high-confidence training samples; inconsistent data undergoes further optimization. For example, it can be inferred multiple times using another multimodal large model, selecting the prediction result with the highest confidence as the intent label to update the inconsistent data; or, multiple different multimodal large models can be used to infer the inconsistent data separately, and the prediction results are then voted on, with the prediction result with the highest confidence being used as the intent label to update the inconsistent data.

[0066] Specifically, in one embodiment, step four involves performing a consistency check between the first prediction result and the label data of the multi-source speech training data, and optimizing the multi-source speech training data based on the consistency check result to obtain the target training sample data. This step may include: The consistency of the first prediction result with the label data of the multi-source speech training data is verified to obtain the consistency verification result. The data in the multi-source speech training data with consistent consistency verification results are determined as the first training sample data, and the data in the multi-source speech training data with inconsistent consistency verification results are determined as the second training sample data. The second training sample data is input into at least two second multimodal large models to obtain the second prediction result output by each second multimodal large model. The second prediction results output by each second multimodal large model are voted on, and the second training sample data is updated according to the voting results to obtain the third training sample data. The first training sample data and the third training sample data are integrated to obtain the target training sample data.

[0067] The second multimodal large model can be the same as or different from the first multimodal large model.

[0068] For example, for consistency verification, if the similarity between the first prediction result and the label data of the multi-source speech training data is greater than a preset similarity threshold, then the consistency verification result between the first prediction result and the label data is determined to be consistent; otherwise, the consistency verification result is inconsistent.

[0069] For example, updating the second training sample data based on the voting results to obtain the third training sample data may include: taking the second prediction result with the most votes as the label data corresponding to the second training sample data, so as to update the second training sample data and obtain the third training sample data.

[0070] For example, for each second training sample data, three different second multimodal large models (denoted as model M1, model M2, and model M3) can be used for inference. The second training sample data is input into model M1, model M2, and model M3 respectively, and the prediction result of each model is obtained. Assuming that model M1 votes for intention label A, model M2 votes for intention label B, and model M3 votes for intention label A, the label data of the second training sample data can be updated to intention label A, thus updating the second training sample data and obtaining third training sample data with the label of intention label A. Alternatively, assuming that models M1, model M2, and model M3 vote for different intention labels, meaning that the confidence level of the prediction results of each model is the same, the second training sample data can be kept unchanged.

[0071] Step 5: Train the second multimodal mini-model using the target training sample data to obtain the multimodal speech understanding model.

[0072] After obtaining the target training sample data, the electronic device can use this data to further supervise and fine-tune the second multimodal mini-model trained in step two, ultimately obtaining the desired multimodal speech understanding model. This further reduces the impact of noise labels on the model, improving its stability and generalization ability.

[0073] The multi-stage collaborative fine-tuning method described in steps one through five constructs a multi-source data system consisting of "distilled data + synthesized speech data + data augmentation + a small amount of manually labeled data," and employs a phased training strategy: The first stage is capability alignment, using distilled data from the first multimodal large model to train the initial multimodal small model, aligning the initial multimodal small model with high-order semantic understanding capabilities, thus obtaining the first multimodal small model; the second stage is basic capability injection, injecting speech data generated based on timbre cloning into the model, and continuing supervised fine-tuning training of the first multimodal small model (an open-source base model), strengthening the joint modeling capability between speech and semantics, enabling the first multimodal small model to directly perform intent recognition and semantic understanding from speech. In this stage, the cloned speech data can also be subjected to noise addition, speed variation, and other modifications. After frequency-enhanced data processing, the data is input into the first multimodal small model for training to improve data diversity and model robustness, ultimately resulting in a well-trained second multimodal small model. The third stage involves using the base model (second multimodal small model) trained in the second stage to perform rapid inference on the generated multi-source speech training data, obtain model prediction results, and perform consistency comparison with the original label data. Data with prediction results consistent with the original label data are directly used for subsequent training, while inconsistent data is fused by voting from multiple multimodal large models to generate more reliable target label data. Finally, the consistent and inconsistent data are combined to construct the target training sample data, which is then re-tuned under supervision based on the second multimodal small model obtained in the second stage to obtain the final multimodal speech understanding model.

[0074] Through this multi-stage coordinated fine-tuning strategy, the trained multimodal speech understanding model can simultaneously possess the following characteristics: semantic understanding capabilities approaching those of large multimodal models; strong robustness to real-world speech environments; and good cross-scene generalization ability. In other words, it achieves a high-performance, lightweight, and highly generalizable multimodal speech understanding model.

[0075] The training method for the multimodal speech understanding model provided in this application first acquires target text intent sample data from a logistics scenario and collects real driver speech data to construct a voice library of real driver speech data. Then, the target text intent sample data and the voice library are fused, and multi-source speech training data is generated based on the fusion result. This method can reverse-generate speech data with intent labels from existing text data, and construct large-scale speech training data with intent labels at low cost. Then, the first multimodal large model is used to perform offline inference on real speech business data from a logistics scenario to obtain distilled data. Finally, the initial multimodal small model is trained based on the multi-source speech training data and the distilled data to obtain the multimodal speech understanding model, thus realizing the training of the multimodal speech understanding model. In this way, through knowledge distillation, the capabilities of the large-scale model (the first multimodal large model) can be transferred to the small-scale model (the initial multimodal small model). This allows the small-scale model to approximate the performance of the large-scale model while maintaining a low inference cost, thereby reducing the training and deployment costs of the multimodal speech understanding model and enabling the model to be deployed in a lightweight manner. Moreover, the multi-source speech training data used in the model training process is generated by reverse engineering from existing text data, which makes up for the lack of intent-labeled speech training data and the extremely high labeling cost in the field of freight transportation, further reducing the training cost of the model.

[0076] Training methods for multimodal speech understanding models based on the above embodiments, Figure 2 This is a second schematic flowchart illustrating the training method of the multimodal speech understanding model provided in this application embodiment. (Refer to...) Figure 2 As shown, the training method for this multimodal speech understanding model may include the following steps 201-216.

[0077] In this example embodiment, the electronic device can first enter the data construction stage of steps 201-203 to perform a multi-source data preparation process for text and voice.

[0078] Step 201: Obtain sample data of target text intent in the logistics scenario.

[0079] Electronic devices can obtain text intent data of logistics scenarios from text intent classification databases to obtain initial text intent data. Then, the initial text intent data is organized and structured to transform the original business corpus into standardized semantic representations, resulting in semantically standardized target text intent sample data, which can provide a unified semantic foundation for subsequent multimodal fusion.

[0080] Step 202: Collect and process real driver voice data.

[0081] Electronic devices can collect real drivers' business interaction voices from online business, such as voice data of drivers communicating via telephone on freight transportation platforms to confirm cargo information, transportation needs, and prices. This data can be used to obtain real driver voice data. Then, the real driver voice data can be preprocessed with voice desensitization and noise reduction. After that, the preprocessed voice data can be segmented and timbre features extracted to obtain target timbre features, thereby obtaining basic voice units that can be used for modeling.

[0082] Step 203: Build a sound library.

[0083] After the electronic device extracts the target timbre features from the real driver's voice data, it aggregates and models these target timbre features to obtain a timbre library of the real driver's voice data, which can provide diverse timbre resources for subsequent voice generation.

[0084] After completing the basic data preparation in steps 201-203, the speech generation and enhancement stage proceeds to steps 204-206.

[0085] Step 204: Sound cloning.

[0086] Electronic devices can use timbre cloning technology to fuse the target text intent sample data obtained in step 201 with the timbre library constructed in step 203 to generate speech data with specified timbre features, thereby obtaining first speech training data with clear intent labels.

[0087] Step 205: Data augmentation.

[0088] The electronic device further enhances the first speech training data generated in step 204 to improve the data's coverage in real complex scenarios, thereby obtaining the second speech training data.

[0089] The enhancement processing may include at least one of the following, but is not limited to: background sound separation, background sound reconstruction, adding noise (including environmental noise and call noise), speech rate perturbation and frequency perturbation.

[0090] For example, the electronic device can also enhance a portion of the first speech training data generated in step 204 to obtain the second speech training data. For instance, a preset number of the first speech training data can be randomly selected for enhancement.

[0091] Step 206: Construct multi-source speech training data.

[0092] The electronic device can integrate the first speech training data obtained in step 204 and the second speech training data obtained in step 205 to form the multi-source speech training data required for the first stage of training.

[0093] In this way, through the speech generation and enhancement in steps 204-206, the data can simultaneously possess diversity and structure at both the semantic and acoustic layers, thus providing a foundation for the subsequent model to learn efficient contextual representations.

[0094] Next, the electronic device can enter the multi-stage collaborative training process of steps 207-216.

[0095] Step 207: Obtain real voice business data in logistics scenarios.

[0096] Real-world voice communication data in logistics scenarios consists of large-scale online Q&A data obtained from online business platforms within the logistics sector, including driver-generated voice interactions. This data can be saved in real time to create a real-world voice communication dataset. Electronic devices can then extract all or part of this dataset to obtain the desired real-world voice communication data.

[0097] Step 208: First multimodal large model offline inference.

[0098] The electronic device can utilize the first multimodal large model to perform offline reasoning (offline semantic understanding) on ​​the real voice service data obtained in step 207, generating high-quality semantic labels (pseudo-labels) and intermediate representation information, and identifying this semantic label data and intermediate representation information as distilled data. The intermediate representation information refers to the process information of the first multimodal large model's thinking and reasoning on the real voice service data. In this way, high-quality label data can be generated through reasoning.

[0099] Step 209: First stage of model training.

[0100] The electronic device can use the distillation data obtained in step 208 to perform the first stage of capability alignment training on the initial multimodal small model. Through full parameter fine-tuning, the initial multimodal small model learns the semantic representation capabilities of the first multimodal large model, thus obtaining the first multimodal small model. In this way, the model can achieve high semantic understanding capabilities with a relatively small parameter scale, completing model distillation and providing a model foundation for low-latency inference.

[0101] Step 210: Second stage of model training.

[0102] After model distillation is completed, the electronic device can use the multi-source speech training data constructed in step 206 to perform a second stage of full parameter fine-tuning training on the first multimodal small model obtained in step 209, injecting the ability to understand intentions related to the logistics field into the first multimodal small model to obtain the second multimodal small model.

[0103] Because the multi-source speech training data used in this training phase covers multiple noise levels, accents, and scene conditions, the model can still stably extract key information in complex environments, thereby reducing the model's reliance on redundant information during inference and providing support for subsequent context compression and efficient inference.

[0104] After completing the first and second stages of model training, you can proceed to the data optimization and retraining stages described in steps 211-216.

[0105] Step 211: Model consistency verification.

[0106] The electronic device can use the second multimodal small model trained in step 210 to perform inference prediction on the multi-source speech training data constructed in step 206, and perform consistency verification between the first prediction result output by the second multimodal small model and the label data of the multi-source speech training data.

[0107] Step 212: Data filtering.

[0108] After the electronic device performs a consistency check between the first prediction result output by the second multimodal small model and the label data of the multi-source speech training data, it can filter the multi-source speech training data constructed in step 206 based on the consistency check result, dividing the multi-source speech training data into consistent data and inconsistent data. For consistent data, step 213 is executed; for inconsistent data, step 214 is executed.

[0109] Step 213: Retain the data.

[0110] For the consistent data segmented in step 212, the electronic device can directly retain it as a high-confidence training sample to obtain the first training sample data.

[0111] Step 214: Multi-model voting fusion.

[0112] For the inconsistent data identified in step 212, the electronic device can use at least two second-mode multimodal large models to infer the inconsistent data, generate new label data for the inconsistent data through voting fusion, and update the label of the inconsistent data with the new label data to update the inconsistent data, thus obtaining the third training sample data. In this way, multiple multimodal large models with strong semantic understanding capabilities can be used to further infer and vote on the inconsistent data, improving the confidence of the label data of the inconsistent data, thereby optimizing the inconsistent data.

[0113] At least two of the second multimodal large models can be the same large model, or they can be partially the same or completely different large models.

[0114] Step 215: Construct target training sample data.

[0115] The electronic device can integrate the first training sample data retained in step 213 with the third training sample data optimized in step 214 to form optimized high-quality target training sample data.

[0116] Step 216: Third stage model training.

[0117] The electronic device can continue to perform a third stage of supervised fine-tuning training on the second multimodal small model obtained in step 210 based on the target training sample data obtained in step 215, so as to obtain the final multimodal speech understanding model required.

[0118] This further reduces the impact of noise labels on the model, improving its stability and generalization ability. The third stage of processing allows the model to gradually converge to a higher-quality data distribution during training, while reducing the waste of computational resources caused by ineffective learning.

[0119] The training method for the multimodal speech understanding model provided in this application can be applied to logistics voice dialogue scenarios. Through the joint design of data construction, model capability transfer, and multi-stage collaborative fine-tuning, it can improve the model's understanding and generalization capabilities in real and complex speech scenarios while reducing training and inference costs. This constructs a multimodal speech understanding model that combines high performance, low cost, and strong generalization capabilities. Furthermore, data construction, model capability transfer, and multi-stage collaborative fine-tuning can be connected through structured data and a unified context design mechanism, forming a progressively advanced technical system. On the one hand, by transferring the semantic understanding capabilities of the large multimodal model to the small multimodal model through knowledge distillation, the training and deployment costs of the model are reduced while ensuring semantic understanding performance, achieving a practical multimodal speech understanding capability with "high performance + low cost". On the other hand, by using timbre cloning and data augmentation, large-scale and high-quality intention-labeled speech data is generated from existing text intent data, overcoming the problem of a lack of intention-labeled speech data and extremely high annotation costs in the logistics field, thus reducing data costs. At the same time, by combining distilled data and synthesized multi-source speech training data, the model is fine-tuned in multiple stages, and a pseudo-label optimization mechanism based on model prediction consistency and multi-model voting is introduced to further improve data quality and the model's generalization ability.

[0120] After obtaining the multimodal speech understanding model based on the training method of the multimodal speech understanding model described in any of the above embodiments, the multimodal speech understanding model can be deployed in the actual logistics business system for tasks such as logistics voice dialogue understanding, intent recognition and related decision support, so as to achieve a balance between high performance and low latency.

[0121] Specifically, this application also provides a speech understanding method, which can be applied to an electronic device or to a semantic understanding device installed in the electronic device. The semantic understanding device can be implemented through software, hardware, or a combination of both. The electronic device may include at least one of the following, but is not limited to: a server, mobile phone, computer, in-vehicle terminal, tablet computer, wearable device, smart home device, augmented reality (AR) device, and virtual reality (VR) device. The server may include a standalone server, a virtual server, and a cluster server.

[0122] Figure 3 A flowchart illustrating the speech understanding method provided in an embodiment of this application is shown below. Figure 3 As shown, the speech understanding method may include the following steps 310-320.

[0123] Step 310: Obtain voice data for the logistics scenario.

[0124] Voice data in logistics scenarios includes voice data of drivers communicating via telephone on freight transportation platforms to confirm cargo information, transportation needs, and prices. Electronic devices can collect drivers' voice data from freight transportation platforms in real time.

[0125] Step 320: Input the voice data of the logistics scenario into the multimodal speech understanding model to obtain the speech understanding information output by the multimodal speech understanding model.

[0126] The multimodal speech understanding model is trained using the training method described in any of the above embodiments, and will not be repeated here.

[0127] After acquiring voice data from a logistics scenario, electronic devices can input this voice data into a multimodal speech understanding model. The model then performs semantic and intent understanding on the voice data to obtain speech understanding information.

[0128] The speech understanding method provided in this application embodiment uses the multimodal speech understanding model provided in this application embodiment to understand speech data, thereby reducing costs and improving the understanding ability and accuracy of speech understanding in complex speech scenarios.

[0129] This application also provides a training device for a multimodal speech understanding model. Figure 4 This paper shows a schematic diagram of the structure of a training device for a multimodal speech understanding model provided in an embodiment of this application. (Refer to...) Figure 4 As shown, the training device for this multimodal speech understanding model may include: The data acquisition module 410 is used to acquire target text intent sample data in logistics scenarios and collect real driver voice data; The timbre library construction module 420 is used to build a timbre library based on real driver voice data. The data fusion module 430 is used to fuse the target text intent sample data and the timbre library, and generate multi-source speech training data based on the fusion result; The data inference module 440 is used to perform offline inference on real voice business data in the logistics scenario using the first multimodal large model to obtain distilled data; The model training module 450 is used to train the initial multimodal small model based on multi-source speech training data and distillation data to obtain a multimodal speech understanding model.

[0130] In one embodiment, the model training module 450 may include: a first training unit, used to perform capability alignment training on an initial multimodal mini-model using distilled data to obtain a first multimodal mini-model; a second training unit, used to perform full-parameter fine-tuning training on the first multimodal mini-model using multi-source speech training data to obtain a second multimodal mini-model; a prediction unit, used to input multi-source speech training data into the second multimodal mini-model to obtain a first prediction result output by the second multimodal mini-model; a verification and optimization unit, used to perform consistency verification between the first prediction result and the label data of the multi-source speech training data, and optimize the multi-source speech training data according to the consistency verification result to obtain target training sample data; and a third training unit, used to train the second multimodal mini-model using the target training sample data to obtain a multimodal speech understanding model.

[0131] In one embodiment, the verification and optimization unit may be specifically used to: perform consistency verification between the first prediction result and the label data of the multi-source speech training data to obtain a consistency verification result; determine the data in the multi-source speech training data with consistent consistency verification results as the first training sample data, and determine the data in the multi-source speech training data with inconsistent consistency verification results as the second training sample data; input the second training sample data into at least two second multimodal large models to obtain the second prediction result output by each second multimodal large model; vote on the second prediction result output by each second multimodal large model, update the second training sample data according to the voting result, and obtain the third training sample data; and integrate the first training sample data and the third training sample data to obtain the target training sample data.

[0132] In one embodiment, the timbre library construction module 420 can be specifically used to: perform speech desensitization and speech segmentation on real driver speech data to obtain initial speech data; extract timbre features from the initial speech data to obtain target timbre features; and aggregate the target timbre features to obtain a timbre library of real driver speech data.

[0133] In one embodiment, the data fusion module 430 is specifically used to: fuse the target text intent sample data and the timbre library using timbre cloning technology to obtain multimodal first speech training data; enhance the first speech training data to obtain second speech training data; and integrate the first speech training data and the second speech training data to obtain multi-source speech training data.

[0134] In one embodiment, the data acquisition module 410 is specifically used to: acquire the initial text intent data of the logistics scenario; and perform structured processing on the initial text intent data to obtain semantically standardized target text intent sample data.

[0135] The training device for the multimodal speech understanding model provided in this application embodiment has the same implementation principle and beneficial effects as the training method for the multimodal speech understanding model provided in the above embodiments, and will not be repeated here.

[0136] This application also provides a speech understanding device. Figure 5 A schematic diagram of the structure of the speech understanding device provided in an embodiment of this application is shown, with reference to... Figure 5 As shown, the voice understanding device may include: The voice acquisition module 510 is used to acquire voice data in logistics scenarios; The speech understanding module 520 is used to input speech data from the logistics scenario into the multimodal speech understanding model to obtain speech understanding information output by the multimodal speech understanding model; wherein, the multimodal speech understanding model is trained based on the training method of the multimodal speech understanding model as described in any of the above embodiments.

[0137] The speech understanding device provided in this application embodiment has the same implementation principle and beneficial effects as the speech understanding method provided in the above embodiments, and will not be repeated here.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0139] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the training method for the multimodal speech understanding model described in any of the above method embodiments, or the steps of the speech understanding method described in any of the above method embodiments, which will not be repeated here.

[0140] For example, Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 can communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the steps of the training method of the multimodal speech understanding model provided in any of the above method embodiments, or execute the steps of the speech understanding method provided in any of the above method embodiments, which will not be elaborated here.

[0141] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] Based on the methods described in any of the above embodiments, this application also provides a computer-readable storage medium, which may be a non-transitory computer-readable storage medium. For example, a non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, and an optical data storage device, etc. The storage medium stores computer instructions for executing the steps of the training method for the multimodal speech understanding model described in any of the above embodiments, or for executing the steps of the speech understanding method described in any of the above method embodiments, which will not be elaborated further here.

[0143] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0144] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations thereof that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

Claims

1. A training method for a multimodal speech understanding model, characterized in that, include: Acquire sample data of target text intent in logistics scenarios and collect real driver voice data; Construct a timbre library of the real driver's voice data; The target text intent sample data and the timbre library are fused together, and multi-source speech training data is generated based on the fusion result; Offline inference was performed on real voice business data in logistics scenarios using the first multimodal large model to obtain distilled data; The initial multimodal mini-model is trained based on the multi-source speech training data and the distillation data to obtain a multimodal speech understanding model; The step of training the initial multimodal mini-model based on the multi-source speech training data and the distillation data to obtain the multimodal speech understanding model includes: The initial multimodal small model is trained using the distillation data to achieve capability alignment, thereby obtaining the first multimodal small model. The first multimodal mini-model is trained with full parameter fine-tuning using the multi-source speech training data to obtain the second multimodal mini-model; The multi-source speech training data is input into the second multimodal small model to obtain the first prediction result output by the second multimodal small model; The first prediction result is checked for consistency with the label data of the multi-source speech training data, and the multi-source speech training data is optimized based on the consistency check result to obtain the target training sample data. The second multimodal mini-model is trained using the target training sample data to obtain the multimodal speech understanding model.

2. The training method for the multimodal speech understanding model according to claim 1, characterized in that, The step of performing a consistency check between the first prediction result and the label data of the multi-source speech training data, and optimizing the multi-source speech training data based on the consistency check result to obtain target training sample data, includes: The consistency verification result is obtained by performing a consistency check between the first prediction result and the label data of the multi-source speech training data. The data in the multi-source speech training data whose consistency check results are consistent are determined as the first training sample data, and the data in the multi-source speech training data whose consistency check results are inconsistent are determined as the second training sample data. The second training sample data is input into at least two second multimodal large models to obtain the second prediction result output by each second multimodal large model; Voting is performed on the second prediction results output by each of the second multimodal large models, and the second training sample data is updated according to the voting results to obtain the third training sample data; The first training sample data and the third training sample data are integrated to obtain the target training sample data.

3. The training method for the multimodal speech understanding model according to claim 1 or 2, characterized in that, The construction of the timbre library for the real driver's voice data includes: The real driver's voice data is anonymized and segmented to obtain initial voice data; The initial speech data is subjected to timbre feature extraction to obtain the target timbre features; The target timbre features are aggregated to obtain the timbre library of the real driver's voice data.

4. The training method for the multimodal speech understanding model according to claim 1 or 2, characterized in that, The process of fusing the target text intent sample data and the timbre library, and generating multi-source speech training data based on the fusion result, includes: The target text intent sample data and the timbre library are fused using timbre cloning technology to obtain multimodal first speech training data; The first speech training data is enhanced to obtain the second speech training data. The first speech training data and the second speech training data are integrated to obtain the multi-source speech training data.

5. The training method for the multimodal speech understanding model according to claim 1 or 2, characterized in that, The acquisition of target text intent sample data in the logistics scenario includes: Obtain initial textual intent data for the logistics scenario; The initial text intent data is structured to obtain semantically standardized target text intent sample data.

6. A speech understanding method, characterized in that, include: Acquire voice data from logistics scenarios; The voice data of the logistics scenario is input into the multimodal speech understanding model to obtain the speech understanding information output by the multimodal speech understanding model. The multimodal speech understanding model is trained based on the training method of the multimodal speech understanding model as described in any one of claims 1 to 5.

7. A training device for a multimodal speech understanding model, characterized in that, include: The data acquisition module is used to acquire target text intent sample data in logistics scenarios and collect real driver voice data; The timbre library construction module is used to construct the timbre library of the real driver's voice data; The data fusion module is used to fuse the target text intent sample data and the timbre library, and generate multi-source speech training data based on the fusion result; The data inference module is used to perform offline inference on real voice business data in logistics scenarios using the first multimodal large model to obtain distilled data; The model training module is used to train the initial multimodal mini-model based on the multi-source speech training data and the distillation data to obtain a multimodal speech understanding model. The model training module includes: The first training unit is used to perform capability alignment training on the initial multimodal small model using the distillation data to obtain the first multimodal small model. The second training unit is used to perform full-parameter fine-tuning training on the first multimodal small model using the multi-source speech training data to obtain the second multimodal small model. The prediction unit is used to input the multi-source speech training data into the second multimodal small model to obtain the first prediction result output by the second multimodal small model; The verification and optimization unit is used to verify the consistency between the first prediction result and the label data of the multi-source speech training data, and optimize the multi-source speech training data according to the consistency verification result to obtain the target training sample data. The third training unit is used to train the second multimodal mini-model using the target training sample data to obtain the multimodal speech understanding model.

8. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the multimodal speech understanding model as described in any one of claims 1 to 5, or implements the speech understanding method as described in claim 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the multimodal speech understanding model as described in any one of claims 1 to 5, or implements the speech understanding method as described in claim 6.

Images