Machine learning model training method and apparatus, dialogue method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a machine learning model training method and apparatus, a dialogue method and apparatus, a device, and a medium. The machine learning model training method comprises: acquiring speech information and text information corresponding to the speech information, and respectively coding the text information and the speech information to obtain text coded information and speech coded information; fusing the text coded information and the speech coded information to obtain fused coded information; according to the fused coded information, predicting time information, in the speech information, corresponding to each text unit in the text information; and according to the predicted time information and a position annotation, in the speech information, of a speech unit corresponding to each text unit in the text information, pretraining a first machine learning model to obtain a pre-trained model.

Description

[0001] Machine Learning Model Training, Dialogue Method, Apparatus, Device, and Medium This disclosure claims priority to Chinese patent application No. 202310397072.3, filed with the China Patent Office on April 11, 2023, entitled "Machine Learning Model Training, Dialogue Method, Apparatus, Device, and Medium," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of information technology, and more particularly to a machine learning model training, dialogue method, apparatus, device, and medium. Background Currently, intelligent customer service can engage in dialogue with users in some business scenarios, providing them with services such as business consultation and transaction processing. Specifically, intelligent customer service can be implemented using a machine learning model deployed on a computing device, such as a computer or computer cluster. However, current machine learning models do not accurately understand user speech. SUMMARY To address or at least partially address the aforementioned technical issues, the present disclosure provides a machine learning model training and dialogue method, apparatus, device, and medium to improve a pre-trained model's understanding of voice and text information, thereby improving the accuracy of the pre-trained model's understanding of user speech. In a first aspect, embodiments of the present disclosure provide a machine learning model training method, comprising: obtaining voice information and text information corresponding to the voice information; encoding the text information and voice information separately to obtain text encoding information and voice encoding information; fusing the text encoding information and voice encoding information to obtain fused encoding information; predicting the time information corresponding to each text unit in the text information in the voice information based on the encoding information corresponding to the text information in the fused encoding information; and pre-training a first machine learning model based on the predicted time information and the positional annotation of the voice unit corresponding to each text unit in the text information in the voice information. In a second aspect, an embodiment of the present disclosure provides a machine learning model training method, the method comprising: obtaining sample data of a target task, the sample data comprising data to be processed and annotated results of the data to be processed, the data to be processed comprising sample speech and sample text corresponding to the sample speech; inputting the data to be processed into a second machine learning model so that the second machine learning model outputs a processed result, the second machine learning model being a pre-trained model obtained according to the training method described in the first aspect; training the second machine learning model according to the processed results output by the second machine learning model and the annotated results of the data to be processed to obtain a trained machine learning model.In a third aspect, embodiments of the present disclosure provide an intelligent dialogue method, the method comprising: obtaining historical dialogues between a target user and an intelligent dialogue system, and text information corresponding to the historical dialogues; and

[0002] Inputting the historical conversation and the text information into a trained machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, and key information, wherein the machine learning model is trained according to the method described in the second aspect; and conducting an intelligent conversation with the target user based on at least one of the target user's emotion, mood, intention, and key information. In a fourth aspect, embodiments of the present disclosure provide a machine learning model training device, comprising: an acquisition module configured to acquire voice information and text information corresponding to the voice information; an encoding module configured to encode the text information and the voice information, respectively, to obtain text encoding information and voice encoding information; a fusion processing module configured to fuse the text encoding information and the voice encoding information to obtain fused encoding information; a prediction module configured to predict the time information corresponding to each text unit in the text information in the voice information based on the encoding information corresponding to the text information in the fused encoding information; and a pre-training module configured to pre-train a first machine learning model based on the predicted time information and the position annotation of the voice unit corresponding to each text unit in the text information in the voice information. In a fifth aspect, embodiments of the present disclosure provide an electronic device, comprising: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first, second, or third aspects. In a sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first, second, or third aspects. The machine learning model training and dialogue methods, apparatuses, devices, and media provided in embodiments of the present disclosure obtain voice information and text information corresponding to the voice information, and encode the text information and voice information, respectively, to obtain text encoding information and voice encoding information. Furthermore, the text encoding information and voice encoding information are fused to obtain fused encoding information. Based on the fused encoding information, the time information corresponding to each text unit in the text information in the voice information is predicted. Then, based on the predicted time information and the position annotation of the voice unit corresponding to each text unit in the text information in the voice information, a first machine learning model is pre-trained to obtain a pre-trained model.This allows the pre-trained model to align speech and text information in temporal order. For example, finer-grained text units in text information can be aligned with finer-grained speech units in speech information. This improves the pre-trained model's understanding of both speech and text information, and thus increases the accuracy of the pre-trained model's understanding of user utterances. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings herein are incorporated into and constitute a part of this specification. They illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. To more clearly illustrate the embodiments of the present disclosure or technical solutions in the prior art, the following briefly describes the drawings required in the embodiments or description of the prior art. Obviously, those skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 is a flow chart of a model pre-training method provided by an embodiment of the present disclosure; Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure; Figure 3 is a schematic diagram of a machine learning model provided by an embodiment of the present disclosure; Figure 4 is a schematic diagram of a conversation between a user and an intelligent dialogue system provided by an embodiment of the present disclosure; Figure 5 is a schematic diagram of time series prediction provided by another embodiment of the present disclosure; Figure 6 is a schematic diagram of a machine learning model provided by another embodiment of the present disclosure; Figure 7 is a schematic diagram of the construction of a first loss function provided by another embodiment of the present disclosure; Figure 8 is a schematic diagram of multiple types of sample data provided by another embodiment of the present disclosure; Figure 9 is a flow chart of a model pre-training method provided by another embodiment of the present disclosure; Figure 10 is a schematic diagram of a machine learning model provided by another embodiment of the present disclosure; Figure 11 is a flow chart of a model training method provided by another embodiment of the present disclosure; Figure 12 is a flow chart of an intelligent dialogue method provided by another embodiment of the present disclosure; Figure 13 is a structural schematic diagram of a model pre-training device provided by an embodiment of the present disclosure; Figure 14 is a structural schematic diagram of a model training device provided by an embodiment of the present disclosure; Figure 15 is a structural schematic diagram of an intelligent dialogue device provided by an embodiment of the present disclosure; Figure 16 is a structural schematic diagram of an electronic device embodiment provided by an embodiment of the present disclosure. DETAILED DESCRIPTION To provide a clearer understanding of the aforementioned objectives, features, and advantages of the present disclosure, the following further describes the embodiments of the present disclosure. It should be noted that, unless conflicting, the embodiments and features of the embodiments may be combined. The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure, but the present disclosure may also be implemented in other ways than those described herein. It is apparent that the embodiments described in this specification represent only a portion of the embodiments of the present disclosure, and do not represent all of the embodiments.It should be noted that the voice information (including but not limited to user voice collected by terminal devices) and text information (including but not limited to text information corresponding to user voice) referred to in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals will be provided for users to choose to authorize or refuse. Furthermore, the machine learning model training and dialogue methods provided in this disclosure involve the following explanations, which are explained in detail below: Joint Speech and Semantic Modeling: The system uses both speech and corresponding text transcription as input. Guided by tasks such as speech-text alignment, it achieves a joint improvement in speech and text understanding. Pre-training: A key method that allows the model to perform unsupervised learning on large-scale corpora, significantly improving the results of downstream tasks. Temporality: This refers to the temporal order of the input sequence. Both speech and text have temporal order. Navigation: The user calls the voice robot. Outbound Calling: The voice robot calls the user. Typically, intelligent customer service can engage in conversations with users in certain business scenarios, providing services such as business consultation and processing. Specifically, intelligent customer service can be implemented using a machine learning model deployed on a computing device, such as a computer or computer cluster. However, current machine learning models lack high accuracy in understanding user speech. To address this issue, embodiments of the present disclosure provide a machine learning model training method, which is described below in conjunction with specific embodiments. Figure 1 is a flow chart of the machine learning model training method provided in embodiments of the present disclosure. The machine learning model training method can specifically be a pre-training method. The method can be executed by a machine learning model training device, which can be implemented using software and / or hardware and can be deployed in an electronic device, such as a server or terminal. Terminals specifically include mobile phones, computers, or tablets. The server can specifically be a cloud server, and the machine learning model training method can be executed by the cloud. The cloud can be equipped with several computing nodes (cloud servers), each of which has processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide a particular service, and a single computing node can also provide one or more services. The cloud can provide this service by providing a service interface, which users can call to access the corresponding service. Service interfaces include software development kits (SDKs) and application programming interfaces (APIs).In addition, the machine learning model training method described in this embodiment can be applied to the application scenario shown in FIG2 . As shown in FIG2 , this application scenario includes a terminal 21 and a server 22. Server 22 can use the machine learning model training method described in this embodiment to pre-train a first machine learning model. After pre-training, the first machine learning model is obtained as a pre-trained model. Furthermore, the pre-trained model can be retained in server 22, or server 22 can deploy the pre-trained model to terminal 21 or another server. This method is described in detail below in conjunction with FIG2 . As shown in FIG1 , the specific steps of this method are as follows:

[0003] S101. Acquire voice information and text information corresponding to the voice information. For example, the server 22 may be provided with an intelligent dialogue system. Terminal 21, as shown in FIG2 , may collect user voice and send the user voice to server 22. The intelligent dialogue system may provide a voice response based on the user voice. Furthermore, server 22 feeds back the voice response of the intelligent dialogue system to terminal 21. Similarly, the user of terminal 21 may engage in a voice dialogue with the intelligent dialogue system. During the dialogue, server 22 may acquire voice information, which may be a voice dialogue between the user and the intelligent dialogue system. Furthermore, server 22 may convert the voice dialogue into corresponding text information through Automatic Speech Recognition (ASR). Specifically, the voice information acquired by server 22 may be the user voice, and the text information may be a text transcription of the user voice. Specifically, the voice information and the text information corresponding to the voice information may serve as input to a first machine learning model. In this embodiment, the text information may include at least one sentence. The voice information may include the voice of each sentence in the at least one sentence, or include the voice of part of the at least one sentence.

[0004] S102. Encode the text information and the voice information, respectively, to obtain text encoding information and voice encoding information. In this embodiment, the first machine learning model may include a text encoder and a voice encoder. The text encoder may be configured to encode the text information to obtain text encoding information. The voice encoder may be configured to encode the voice information to obtain voice encoding information. In one feasible implementation, the text encoder may encode each sentence included in the text information to obtain text encoding information, where the text encoding information includes a representation vector for each sentence. Similarly, the voice encoder may encode the voice of each sentence in the voice information to obtain voice encoding information, where the voice encoding information includes a representation vector for each sentence. In another feasible implementation, encoding the text information and the speech information separately to obtain text encoding information and speech encoding information includes: encoding at least one encoding unit included in each text unit in the text information using a text encoder to obtain text encoding information, wherein the text encoding information includes a representation vector for each encoding unit; and encoding each speech segment in the speech information using a speech encoder to obtain speech encoding information, wherein the speech encoding information includes a representation vector for each speech segment. For example, each sentence in the text information is composed of text units, which may be characters, words, phrases, subwords, or single words. For example, a sentence is "They won five games successively". When words are used as text units, the sentence can be divided into 5 words, namely "he", "they", "win", "have", and "five games". Each text unit may also include at least one encoding unit. For example, "he" includes one character, so "he" includes one encoding unit, namely "he" itself. "Five games" includes two characters. If one character is used as one encoding unit, "five games" includes two encoding units. Therefore, the sentence "They won five games successively" includes 6 encoding units, namely "he", "they", "win", "have", "five", and "game". Therefore, when the sentence "They won five games successively" is encoded by a text encoder, the 6 encoding units can be encoded separately. After encoding, each encoding unit corresponds to a representation vector. For another example, a sentence is "They won five games successively". The sentence includes a total of 5 words. Each word is regarded as a text unit. Since the first four words are shorter, each of the four words can be regarded as a coding unit.However, the last word, "successively," is relatively long. Therefore, it can be divided into two coding units: "success" and "ively." Therefore, when encoding the sentence "They won five games successively" using a text encoder, the six coding units, "They," "won," "five," "games," "success," and "ively," can be encoded separately. After encoding, each coding unit corresponds to a representation vector. Since the text information is not limited to a single sentence; each sentence can include multiple text units, and each text unit can also include at least one coding unit, when encoding the text information using a text encoder, each coding unit in the text information can be encoded to obtain text encoding information, which includes the representation vectors for each coding unit. Furthermore, the speech of each sentence in the speech information can be a 10-second audio signal. Before speech encoding, the speech of each sentence can be split into multiple speech segments. For example, each speech segment can be 1 second long. Furthermore, each speech segment in the speech information is encoded by a speech encoder to obtain speech coding information, where the speech coding information includes a representation vector of each speech segment.

[0005] S103. Fuse the text encoding information and the speech encoding information to obtain fused encoding information. In this embodiment, the first machine learning model may further include a modal fusion layer. After obtaining the text encoding information and speech encoding information as described above, the text encoding information and the speech encoding information may be concatenated, and the concatenated result may be used as input to the modal fusion layer. The modal fusion layer then fuses the text encoding information and the speech encoding information to obtain fused encoding information, which may also be recorded as a joint representation vector. Specifically, the input and output of the modal fusion layer have a one-to-one correspondence. For example, if the input of the modal fusion layer includes representation vectors for m coding units and representation vectors for n speech segments, that is, the input of the modal fusion layer includes m+n representation vectors, then the output of the modal fusion layer also includes m+n representation vectors, and there is a one-to-one correspondence between the m+n representation vectors in the input and the m+n representation vectors in the output. Each representation vector output by the modal fusion layer contains both text and speech components. Therefore, each representation vector output by the modality fusion layer can be recorded as a fused representation vector.

[0006] 5104. Predict the time information corresponding to each text unit in the text information in the speech information according to the coding information corresponding to the text information in the fused coding information. For example, the text information includes m coding units, the speech information includes n speech segments, and the input of the modality fusion layer includes the representation vectors of m coding units and the representation vectors of n speech segments. Since the input and output of the modality fusion layer are in one-to-one correspondence, the first m representation vectors output by the modality fusion layer are the coding information corresponding to the text information, and the last n representation vectors output by the modality fusion layer are the coding information corresponding to the speech information. Since each representation vector output by the modality fusion layer contains both text components and speech components, the first machine learning model can predict the time information corresponding to each text unit in the text information in the speech information according to the first m representation vectors output by the modality fusion layer. For example, the time information corresponding to each text unit in the speech information includes the start time and the end time corresponding to the text unit in the speech information. It can be understood that text units such as "he", "they", "win", "five games" are respectively information of the text type, each text unit corresponds to a speech word, and the speech word is information of the speech type. The speech word corresponding to a text unit can be the pronunciation of the text unit by the user or the intelligent dialogue system. Therefore, the start time corresponding to the text unit in the speech information is specifically the start time corresponding to the speech word corresponding to the text unit in the speech information (for example, the start moment of pronunciation), and the end time corresponding to the text unit in the speech information is specifically the end time corresponding to the speech word corresponding to the text unit in the speech information (for example, the end moment of pronunciation).

[0007] S105. Pre-train the first machine learning model based on the predicted time information and the position of the speech unit corresponding to each text unit in the text information in the speech information, obtaining a pre-trained model. Specifically, in this embodiment, the position of the speech unit corresponding to each text unit in the text information in the speech information can be pre-annotated. Furthermore, a first loss function is constructed based on the time information corresponding to each text unit in the text information in the speech information predicted by the first machine learning model, as well as the position of the speech unit corresponding to each text unit in the text information in the speech information, and then pre-train the first machine learning model based on the first loss function. For example, in this embodiment, the text information is "They won five games," and the speech information is the speech of "They won five games." In this embodiment, the position of the speech unit corresponding to each text unit in the text information in the speech information can be pre-annotated. For example, the text information includes five text units, namely, "he", "we", "win", "have", and "five". The voice information includes phonetic words corresponding to "he", "we", "win", "have", and "five", respectively, and the phonetic words are recorded as phonetic units. For example, the phonetic word corresponding to "he" is the pronunciation of "he". In this embodiment, the starting position (for example, the starting time or starting moment) and the ending position (for example, the ending time or ending moment) of the phonetic word corresponding to "he" in the voice information can be pre-marked. Similarly, the starting position and the ending position of the phonetic words corresponding to "we", "win", "have", and "five" in the voice information can be pre-marked. Furthermore, a first loss function is constructed based on the start and end times of "he," "we," "win," "have," and "five" predicted by the first machine learning model in the voice information, as well as the pre-annotated start and end positions of "he," "we," "win," "have," and "five" in the voice information. Specifically, a first difference is calculated based on the start times of "he," "we," "win," "have," and "five" predicted by the first machine learning model in the voice information, as well as the pre-annotated start positions of the corresponding voice words of "he," "we," "win," "have," and "five" in the current voice.According to the end times of "he", "they", "won", "five games" predicted by the first machine learning model in the speech information, and the end positions of the corresponding speech words of "he", "they", "won", "five games" pre-annotated in the current speech, calculate the second difference. Further, construct a first loss function based on the first difference and the second difference. And pre-train the first machine learning model according to the first loss function to obtain a pre-trained model. In the embodiments of the present disclosure, by obtaining speech information and the text information corresponding to the speech information, and encoding the text information and the speech information respectively, text encoding information and speech encoding information are obtained. Further, perform fusion processing on the text encoding information and the speech encoding information to obtain the encoded information after fusion processing. According to the encoded information after fusion processing, predict the time information corresponding to each text unit in the text information in the speech information, and then pre-train the first machine learning model according to the predicted time information and the position annotation of the corresponding speech unit of each text unit in the text information in the speech information to obtain a pre-trained model. So that the pre-trained model can align the speech information and the text information in time sequence. For example, the alignment between finer-grained text units in the text information and finer-grained speech units in the speech information, thereby improving the understanding ability of the pre-trained model for speech information and text information, and thus improving the accuracy of the pre-trained model for understanding user speech. As shown in FIG. 3 is a schematic structural diagram of the first machine learning model. Specifically, the first machine learning model includes a text encoder, a speech encoder, and a modality fusion layer. The pre-training method will be introduced in detail below with reference to FIG. 3. Specifically, the input of the first machine learning model includes text information and speech information, that is, the input of the first machine learning model is multi-modal information. Optionally, the speech information includes historical speech and current speech, and the text information includes historical text and current text. That is to say, for each type of modal information, both historical information and current information are included. For example, when the speech information is a speech conversation between a user and an intelligent dialogue system, as time goes by, the conversation content is constantly increasing. Therefore, during the conversation, some speech can be used as historical speech, and some speech can be used as current speech. Similarly, since the text information is the text transcription of the speech conversation, some sentences in the text information can be used as historical text, and some sentences can be used as current text. As shown in FIG. 4 is a schematic diagram of a conversation between a user and an intelligent dialogue system. Assume that the conversation content shown in FIG. 4 is text transcription.The text transcription of each sentence spoken by the user can be recorded as one round of text, and the text transcription of each sentence spoken by the intelligent dialogue system can be recorded as one round of text. Therefore, the dialogue content shown in Figure 4 is 8 rounds of text, namely "AAAAAA".

[0008] "BBBBBB", "CCCC", "DDDD", "EEEEEE", "FFFFFF", "GGGGG",

[0009] "HHHHH" is a round of text. In addition, in this embodiment, the first 7 rounds of text in the 8 rounds of text can be recorded as historical texts, and the last round of text, that is, the text "HHHHH", can be recorded as the current text. Similarly, since each round of text in the eight rounds corresponds to a speech, and one speech is recorded as one round of speech, eight rounds of text correspond to eight rounds of speech. The first seven rounds of speech are recorded as historical speech, and the last round of speech, i.e., the speech corresponding to "HHHHH," is recorded as the current speech. Furthermore, the maximum length of a round of speech can be 10 seconds. Since text and speech are two different modalities, the storage space occupied by text and speech of the same round may be different. For example, speech of the same round may occupy a larger storage space. Therefore, text and speech of the same round can be used as inputs to the first machine learning model, or text of the first round and speech of the second round can be used as inputs to the first machine learning model, where the first round is greater than the second round. For example, in this embodiment, eight rounds of text and two rounds of speech can be selected as inputs to the first machine learning model as shown in FIG3 . Specifically, the eight rounds of text include seven rounds of historical text and one round of current text. The two rounds of speech include one round of historical speech and one round of current speech. Furthermore, the current round of text and the current round of speech are aligned. For example, the current round of text is "HHHHH," and the current round of speech is the speech corresponding to "HHHHH." Therefore, the historical text shown in FIG3 can be the seven rounds of historical text, and the current text shown in FIG3 can be the one round of current text. The historical speech shown in FIG3 can be the one round of historical speech, and the current speech shown in FIG3 can be the one round of current speech. That is, the seven rounds of historical text and the one round of current text constitute text information, and the one round of historical speech and the one round of current speech constitute speech information. Furthermore, the text information is encoded using a text encoder to obtain text encoding information, and the speech information is encoded using a speech encoder to obtain speech encoding information. As shown in FIG3, to distinguish the seven rounds of historical text from the one round of current text, a preset symbol <s> can be added at the beginning of the seven rounds of historical text, and a preset symbol < / s> can be added between the seven rounds of historical text and the one round of current text. Alternatively, A preset symbol, < / s>, is added between each adjacent round of text. This means that < / s> is set to separate each round of text. Similarly, preset symbols like vs> and v / s> can be used to separate the historical speech from the current speech, or to separate each round of speech.Furthermore, a text encoder encodes <s>, the seven rounds of historical text described above, < / s>, and the current text described above. A speech encoder encodes the historical speech described above and the current speech described above. For example, for the seven rounds of historical text and the current text described above, each round of text can be a sentence. Each sentence is composed of text units, and each text unit can also include at least one encoding unit. Therefore, the historical text and current text shown in Figure 3 can be divided into several encoding units, for example, box 31 represents any encoding unit. Furthermore, the preset symbol <s> can be used as an encoding unit, and the preset symbol < / s> can be used as an encoding unit. As shown in Figure 3, after the text encoder encodes each encoding unit, text encoding information is obtained. This text encoding information includes representation vectors for each encoding unit. For example, this text encoding information includes the representation vectors within box 32 shown in Figure 3, where each small box within box 32 (for example, box 33) represents a representation vector for a coding unit. In this embodiment, the historical speech and the Danghu speech shown in FIG3 are each 10-second audio signals. Before encoding, the historical speech can be split into multiple speech segments, and the current speech can also be split into multiple speech segments. For example, each speech segment can be 1 second long. As shown in FIG3 , the speech encoder encodes each speech segment in the historical speech, and the speech encoder encodes each speech segment in the current speech, to obtain speech encoding information. This speech encoding information includes representation vectors for each speech segment. For example, this speech encoding information includes the representation vectors within box 34 shown in FIG3 . Each small box within box 34 (e.g., box 35) represents a representation vector for a speech segment. As shown in FIG3 , the text encoding information represented by box 32 and the speech encoding information represented by box 34 are concatenated, and the concatenated result is used as input to the modal fusion layer. The modal fusion layer fuses the text encoding information and the speech encoding information to obtain fused encoding information, which can also be recorded as a joint representation vector. For example, the coded information after the fusion process includes the respective representation vectors in the box 36 as shown in FIG3 , and each small box (eg, box 37) in the box 36 represents a representation vector output by the modality fusion layer.For example, box 33 represents the representation vector of the preset character <s>, and box 37 represents the first representation vector output by the modality fusion layer. This first representation vector is the result of a fusion of text encoding information and speech encoding information. Specifically, this first representation vector is the result of a fusion of the representation vector of the preset character <s>, the representation vectors of each encoding unit in the historical text, the representation vector of the preset character < / s>, the representation vector of each encoding unit in the current text, the representation vector of each speech segment in the historical speech, and the representation vector of each speech segment in the current speech. Therefore, the first representation vector contains not only text components but also speech components. Similarly, the other representation vectors output by the modality fusion layer contain both text and speech components. Furthermore, a portion of the representation vectors in box 36 shown in FIG3 is extracted, for example, the representation vectors corresponding to the current text output by the modality fusion layer, namely, the representation vectors in box 38. Furthermore, based on the representation vectors in box 38, the corresponding time information of each text unit in the current text in the current speech is predicted. For example, the current speech is 10 seconds long. After normalizing the length of the current speech, the start position of the current speech can be recorded as 0, and the end position can be recorded as lo. As shown in Figure 3, the start time of each text unit in the current text in the current speech is predicted by fully connected layer 1, and the end time of each text unit in the current text in the current speech is predicted by fully connected layer 2. For example, as shown in Figure 5, assuming the current text is "They won five times," based on the representation vector corresponding to the current text output by the modal fusion layer, the start time (e.g., 0.05) and end time (e.g., 0.16) corresponding to "he" in the current speech can be predicted, as can the start time (e.g., 0.18) and end time (e.g., 0.32) corresponding to "men" in the current speech. Similarly, the start time (e.g., 0.067) and end time (e.g., 0.97) corresponding to "five times" in the current speech can be predicted. For example, the current text and the current speech are samples, so the position of the speech unit corresponding to each text unit in the current text in the speech information can be pre-annotated. For example, the current speech includes speech words corresponding to "he," "men," "win," "le," and "five times," respectively. These speech words are recorded as speech units. For example, the speech word corresponding to "he" is the pronunciation of "he." In this embodiment, "he" can be pre-annotated. The starting position (such as the starting time or starting moment) and the ending position (such as the ending time or ending moment) of the corresponding speech word in the current speech.Similarly, the start and end positions of the voice words corresponding to "们", "赢", "了", and "五把" can be pre-annotated in the current voice. Further, according to the start times of "他", "们", "赢", "了", "五把" predicted by the fully connected layer 1 in the current voice, and the start positions of the voice words corresponding to "他", "们", "赢", "了", "五把" pre-annotated in the current voice, a first difference is constructed. According to the end times of "他", "们", "赢", "了", "五把" predicted by the fully connected layer 2 in the current voice, and the end positions of the voice words corresponding to "他", "们", "赢", "了", "五把" pre-annotated in the current voice, a second difference is constructed. Further, a first loss function is constructed based on the first difference and the second difference, and the first machine learning model shown in Figure 3 is pre-trained through this first loss function to obtain a pre-trained model. In the above embodiment, the fused encoded information includes the fused representation vectors corresponding to each encoding unit respectively, and the fused representation vectors corresponding to each voice segment respectively; according to the fused encoded information, predicting the time information corresponding to each text unit in the text information in the voice information includes: if the text unit includes one encoding unit, then according to the fused representation vector corresponding to the encoding unit, predicting the start time and end time corresponding to the text unit in the voice information; if the text unit includes multiple encoding units, then according to the fused representation vector corresponding to the first encoding unit in the multiple encoding units, predicting the start time corresponding to the text unit in the voice information, and according to the fused representation vector corresponding to the last encoding unit in the multiple encoding units, predicting the end time corresponding to the text unit in the voice information. As shown in Figure 6, the fused representation vectors output by the modality fusion layer include the fused representation vector corresponding to the preset character vs>, the fused representation vectors corresponding to each encoding unit in the historical text respectively, the fused representation vector corresponding to the preset character < / s>, the fused representation vectors corresponding to each encoding unit in the current text respectively, and the fused representation vectors corresponding to each voice segment respectively. Specifically, if there are 8 rounds of historical text and 8 rounds of historical voice, and the 8 rounds of historical text and 8 rounds of historical voice are aligned, then according to the fused representation vectors corresponding to each encoding unit in the historical text, the time information corresponding to each text unit in the historical text in the historical voice can be predicted.For example, based on the fused representation vectors corresponding to each coding unit in the first round of historical text, the start and end times corresponding to each text unit in the first round of historical speech are predicted. Similarly, the start and end times corresponding to each text unit in the eighth round of historical text are predicted. Furthermore, since the current text and current speech are aligned, the start and end times corresponding to each text unit in the current text in the current speech can be predicted based on the fused representation vectors corresponding to each coding unit in the current text. For example, if the current text is "They won five rounds," "he," "we," "win," "le," "five," and "put" are coding units. The fused representation vectors output by the modality fusion layer include fused representation vectors corresponding to "he," "we," "win," "le," "five," and "put." For example, A in Figure 6 represents the fused representation vector corresponding to "he," and similarly, F represents the fused representation vector corresponding to "put." Since "he", "they", "win", and "have" are text units respectively, and the text unit includes a coding unit, for example, "he" is a text unit, and the text unit includes a coding unit, and the coding unit is "he", that is, "he" is both a text unit and a coding unit included in the text unit. Therefore, according to the fused representation vector A corresponding to "he", the start time and end time corresponding to "he" in the current speech can be predicted. For example, the fully connected layer 1 shown in Figure 6 includes the parameter W. start , the fully connected layer 2 includes the parameter Wend, and the start time of "he" in the current speech predicted by the fully connected layer 1 is recorded as W start A, the end time of the word "he" predicted by the fully connected layer 2 in the current speech is recorded as W endA. Similarly, the fully-connected layer 1 can also predict the start times of "们", "赢", and "了" in the current speech respectively, and the fully-connected layer 2 can also predict the end times of "们", "赢", and "了" in the current speech respectively. Additionally, since "五把" is a text unit, but this text unit includes multiple coding units. For example, "五" is the first coding unit in this text unit, and "把" is the last coding unit in this text unit. Therefore, based on the fused representation vector corresponding to the first coding unit, the start time of "五把" in the current speech can be predicted, and based on the fused representation vector corresponding to the last coding unit, the end time of "五把" in the current speech can be predicted. For example, the fused representation vector corresponding to "五" is denoted as E, and the fused representation vector corresponding to "把" is denoted as F. Then, the start time of "五把" in the current speech predicted by the fully-connected layer 1 is denoted as W start E, and the end time of "五把" in the current speech predicted by the fully-connected layer 2 is denoted as W endF. For example, Figure 5 shows the start time of each text unit predicted by fully connected layer 1 and the end time of each text unit predicted by fully connected layer 2. Optionally, pre-training a first machine learning model based on the predicted time information and the positional annotation of the speech unit corresponding to each text unit in the text information in the speech information includes: constructing a first loss function based on the start time and end time corresponding to the text unit in the speech information, and the pre-annotated start and end positions of the speech unit corresponding to the text unit in the speech information; and pre-training the first machine learning model based on the first loss function. Optionally, the first loss function includes a first difference and a second difference, the first difference being the difference obtained based on the start time and start position, and the second difference being the difference obtained based on the end time and end position. For example, as shown in Figure 7, the duration of the current speech is 10 seconds, i.e., the current speech starts at the 0th second and ends at the 10th second. In this embodiment, the start and end positions of the speech units corresponding to each text unit in the current text, such as "They won five rounds," can be pre-annotated. For example, the speech unit corresponding to "he" in the current speech starts at 0.6 seconds and ends at 1.7 seconds. Similarly, the speech unit corresponding to "wuba" in the current speech starts at 6.9 seconds and ends at 9.9 seconds. Since the start time and end time predicted by fully connected layer 1 and fully connected layer 2 are both values ​​between 0 and 1, this embodiment can normalize the duration of the current speech to ensure that they match the values ​​between 0 and 1 in magnitude. For example, the duration of "Dang Hu" speech can be normalized to 1 second, that is, the start position of the current speech is normalized to 0.6 seconds, and the end position of the current speech is normalized to 1 second. This normalizes the start and end positions of the speech units corresponding to each text unit in "They won five rounds" within the current speech. For example, after normalization, the speech unit corresponding to "he" starts at 0.06 seconds and ends at 0.17 seconds within the current speech. Similarly, the speech unit corresponding to "five rounds" starts at 0.69 seconds and ends at 0.99 seconds within the current speech. In other words, during the normalization process, the total length is divided by the duration of the current speech (e.g., 10 seconds).As shown in Figure 7, for the same text unit in the current text, such as "he", the start time corresponding to "he" predicted by the fully connected layer 1 in the current speech is 0.05, while the start position of the speech unit corresponding to the pre-annotated "he" in the current speech is at the 0.06th second. That is, there is a certain difference between this start time and this start position, and this difference is denoted as the first difference. At the same time, the end time corresponding to "he" predicted by the fully connected layer 2 in the current speech is 0.16, while the end position of the speech unit corresponding to the pre-annotated "he" in the current speech is at the 0.17th second. That is, there is a certain difference between this end time and this end position, and this difference is denoted as the second difference. In addition, for other text units in the current text, there are similar first and second differences. Furthermore, this embodiment takes predicting the start time and end time of each text unit in the current text as an example for illustrative purposes. In other embodiments, it is also possible to predict the time information corresponding to each text unit in the historical text in the historical speech. For example, according to the fused representation vectors respectively corresponding to each coding unit in the first-round historical text, predict the start time and end time corresponding to each text unit in the first-round historical text in the first-round historical speech, and so on, predict the start time and end time corresponding to each text unit in the eighth-round historical text in the eighth-round historical speech. Therefore, for each text unit in each round of historical text, there are similar first and second differences. Therefore, this embodiment can construct a first loss function based on the first and second differences respectively corresponding to each text unit in multiple rounds of text (including historical text and current text). Further, pre-train the first machine learning model as shown in Figure 3 according to this first loss function. For example, this first loss function £ can be expressed as the following formula: The first loss function uses the mean squared error (MSE), which is the square of the difference between the predicted result (e.g., the predicted start or end time) and the true position (e.g., the pre-annotated start or end position). i represents the i-th round of text, m represents the round number or the number of texts. j represents the j-th text unit in the i-th round of text, e.g., the / -th word. n represents the number of words in the i-th round of text. d represents the parameters included in the fully connected layer 1, md represents the parameters included in the fully connected layer 2, j represents the fused representation vector corresponding to the first encoding unit in the j-th text unit in the i-th round of text, and t represents the fused representation vector corresponding to the last encoding unit in the j-th text unit in the i-th round of text. If the j-th text unit includes one encoding unit, then the first encoding unit and the last encoding unit here are the same encoding unit, that is, t and t are the same. If the j-th text unit includes multiple encoding units, then the first encoding unit and the last encoding unit here are different, that is, t and t are different. It can be understood that during the pre-training process, t and t are parameters to be trained. start h s . represents the start time of the jth text unit in the i-th round of text predicted by the fully connected layer 1 in the i-th round of speech, W end h eijIt represents the end time of the Jth text unit in the i-th round of text predicted by the fully connected layer 2 in the i-th round of speech. S" represents the starting position of the jth text unit in the i-th round of text, which is pre-annotated, in the i-th round of speech before normalization, and ⑶ represents the ending position of the jth text unit in the i-th round of text, which is pre-annotated, in the i-th round of speech before normalization. Scale represents the duration of the Z-th round of speech, or represents the maximum duration in multiple rounds of speech, and Scale is set to perform the normalization process as described above. In addition, when pre-training the first machine learning model shown in Figure 3 or Figure 6 according to the first loss function, the parameters of each to-be-trained structure shown in Figure 3 or Figure 6, such as the text encoder, the speech encoder, the modal fusion layer, the fully connected layer 1, and the fully connected layer 2, can be iteratively updated until the number of iterations reaches a preset number or the parameters converge. For example, based on the similarity or difference between the prediction result output by the first machine learning model and the pre-annotated true position, the parameters of each to-be-trained structure shown in Figure 3 or Figure 6 are adjusted so that the prediction result output by the first machine learning model and the pre-annotated true position gradually become similar in subsequent iterative training processes. For example, Parameter adjustment can be guided based on the gradient data between the prediction result output by the first machine learning model and the pre-annotated true position. The original meaning of a gradient is a vector (vector), indicating that the directional derivative of a function at that point reaches its maximum value along that direction. That is, the function changes fastest and has the highest rate of change along that direction (the direction of the gradient) at that point. Based on this principle, parameter adjustment can be guided so that the prediction result output by the first machine learning model approaches the pre-annotated true position. As described above, the sample data used for pre-training the first machine learning model includes text information and speech information. For example, the text information includes seven rounds of historical text and one round of current text, and the speech information includes one round of historical speech and one round of current speech. The seven rounds of historical text and one round of current text are eight consecutive sentences in the same conversation. Similarly, one round of historical speech and one round of current speech are two consecutive speech in the same conversation. This means that there is semantic coherence between the seven rounds of historical text and the one round of current text. There is semantic coherence between the 1st round of historical speech and the 1st round of current speech. In other embodiments, three other types of sample data can be derived from this sample data. For example, the first type of derived sample data can be obtained by replacing the 1st round of current text in this sample data with a sentence from another conversation. This results in the semantic coherence between the 7th round of historical text and the 1st round of current text being lost, but the semantic coherence between the 1st round of historical speech and the 1st round of current speech remains.For another example, replacing the current speech in the sample data with a speech from another conversation results in a second type of derived sample data. This results in a loss of semantic coherence between the historical speech and the current speech, but a loss of semantic coherence between the seven historical texts and the current text. For another example, replacing the current text in the sample data with a sentence from another conversation and replacing the current speech in the sample data with a speech from another conversation results in a third type of derived sample data. This results in a loss of semantic coherence between the seven historical texts and the current text, as well as between the historical speech and the current speech. If the original sample data is recorded as sample data with a label of 0 as shown in Figure 8, then the other three types of sample data derived from this original sample data are, respectively, sample data with labels of 1, 2, and 3 as shown in Figure 8. Here, Query1 represents the historical text, Query2 represents the current text, Query3 represents the historical speech, Query4 represents the current speech, Query5 represents the current text after being replaced with other sentences, and Query6 represents the current text after being replaced with other sentences. and Indicates the current voice after being replaced with another voice. In the process of pre-training the first machine learning model, the pre-training method further includes the following steps as shown in FIG9:

[0010] S901: Based on the coded information after the fusion process, determine whether the historical speech matches the current speech to obtain a first determination result, and determine whether the historical text matches the current text to obtain a second determination result. In this embodiment, sample data with Label 0, Label 1, Label 2, or Label 3 can be used as It is understandable that if the current text is Query 1, then the historical text matches the current text, that is, the speech between the historical text and the current text is coherent, and the historical text and the current text have obvious contextual relevance in semantic content. At this time, the historical text and the current text are the context of the same conversation. If the current text is Query 2, then the historical text matches the current text, that is, the speech between the historical text and the current text is coherent, and the historical text and the current text have obvious contextual relevance in semantic content. and, then the historical text and the current text do not match, that is, the historical text and the current text are not in the same conversation context. Similarly, if the current speech is Query (, then the historical speech and the current speech match, that is, the speech between the historical speech and the current speech is coherent, and the historical speech and the current speech have obvious contextual relevance in semantic content. In this case, the historical speech and the current speech are in the same conversation context. If the current speech is Query (, then the historical speech and the current speech do not match, that is, the historical speech and the current speech are not in the same conversation context. Therefore, the first judgment result described above has two results: one result is that the historical speech and the current speech match, and the other result is that the historical speech and the current speech do not match. Similarly, the second judgment result also has two results: one result is that the historical text and the current text match, and the other result is that the historical text and the current text do not match.

[0011] S902. Predict the type of an input sample based on the first judgment result and the second judgment result, where the input sample includes the speech information and the text information. For example, the fully connected layer 3 may predict the type of the input sample based on the first judgment result and the second judgment result. For example, if the first judgment result indicates that the historical speech and the current speech match, and the second judgment result indicates that the historical text and the current text match, the fully connected layer 3 predicts that the input sample is a sample with a label of 0. If the first judgment result indicates that the historical speech and the current speech match, and the second judgment result indicates that the historical text and the current text do not match, the fully connected layer 3 predicts that the input sample is a sample with a label of 1. If the first judgment result indicates that the historical speech and the current speech do not match, and the second judgment result indicates that the historical text and the current text match, the fully connected layer 3 predicts that the input sample is a sample with a label of 2. If the first judgment result indicates that the historical speech and the current speech do not match, and the second judgment result indicates that the historical text and the current text do not match, the fully connected layer 3 predicts that the input sample is a sample with a label of 3. That is, the fully connected layer 3 can predict whether the input sample is a sample with Label 0, Label 1, Label 2, or Label 3 based on the first and second judgment results. In one possible implementation, the fully connected layer 3 can directly output the predicted type of the input sample, for example, Label = 0, Label = 1, Label = 2, or Label = 3. In another possible implementation, the fully connected layer 3 can be a four-class classification task, that is, the output of the fully connected layer 3 can be four probability values, which respectively represent the probabilities of the input sample being a sample with Label 0, Label 1, Label 2, or Label 3. Furthermore, the type corresponding to the maximum probability value among the four probability values ​​is used as the type of the input sample predicted by the fully connected layer 3.

[0012] S903. Based on the predicted input sample type and the pre-labeled input sample type, a second loss function is constructed, and the first machine learning model is pre-trained based on the second loss function. This embodiment may also pre-label whether the historical speech and current speech in the input sample actually constitute contextual information, and pre-label whether the historical text and current text in the input sample actually constitute contextual information. That is, this embodiment may pre-label the actual type of the input sample, i.e., whether the input sample has Label 0, Label 1, Label 2, or Label 3. Furthermore, this embodiment may construct a second loss function based on the input sample type predicted by the fully connected layer 3 and the pre-labeled actual type of the input sample. This second loss function may be a cross-beam loss function for classification. Furthermore, the first machine learning model shown in FIG10 is pre-trained based on this second loss function. This embodiment uses the fused encoded information to determine whether the historical speech and current speech match, obtaining a first determination result, and to determine whether the historical text and current text match, obtaining a second determination result. The first machine learning model is enabled to predict the type of the input sample based on the first and second judgment results. Furthermore, the input sample type is pre-labeled based on whether the historical speech and the current speech actually constitute contextual information, as well as whether the historical text and the current text actually constitute contextual information. A second loss function is then constructed based on the pre-labeled input sample type and the input sample type predicted by the first machine learning model, and the first machine learning model is pre-trained based on the second loss function to obtain a pre-trained model. This enables the pre-trained model to recognize contextual information, which helps improve the accuracy of the pre-trained model in user intent recognition, sentiment analysis, and other aspects. It is understood that in this embodiment, the first machine learning model can be pre-trained based on either the first or second loss function alone, or the first and second loss functions can be used together. Furthermore, in addition to the aforementioned time series prediction tasks (e.g., predicting the start and end times of a text unit in speech) and cross-modal dialogue response selection tasks (e.g., predicting the type of an input sample), the first machine learning model can also be pre-trained using other tasks. For example, the first machine learning model can also be pre-trained through a mask prediction task. For example, this embodiment can mask some words in historical text or current text, and / or mask some speech segments in historical speech or current speech.The masked samples are then input into the first machine learning model. Furthermore, the masked partial words and / or speech parts are predicted based on the fused representation vector output by the modality fusion layer. A third loss function is then constructed based on the predicted masked partial words and / or speech parts and the actual masked partial words and / or speech parts, and the first machine learning model is pre-trained based on this third loss function. This utilizes different modal information, such as text and speech, to model the interaction between the two modalities. It will be appreciated that pre-training the first machine learning model yields a pre-trained model, which can serve as a model foundation. Sample data from downstream tasks can then be used to train or fine-tune the pre-trained model, resulting in a trained machine learning model that can then be used to perform downstream tasks. Figure 11 is a flow chart of a machine learning model training method according to another embodiment of the present disclosure. This machine learning model training method can be a method for training or fine-tuning the pre-trained model. In this embodiment, the method comprises the following specific steps:

[0013] S1101: Obtain sample data for a target task. The sample data includes data to be processed and annotation results for the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech. For example, the pre-trained model may be stored in server 22. Server 22 may further train or fine-tune the pre-trained model. Specifically, server 22 may obtain sample data for a target task, which may be the downstream task described above. The sample data includes data to be processed and annotation results for the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech. The annotation results for the data to be processed are pre-annotated user emotion categories, user mood categories, user intent, or key information from the sample speech or sample text based on the sample speech and / or sample text. Optionally, the target task includes at least one of the following: a sentiment analysis task, an emotion recognition task, an intent analysis task, or a key information collection task. For example, when the target task is sentiment analysis, the sample data acquired by server 22 includes sample speech, sample text corresponding to the sample speech, and the pre-labeled user emotion category corresponding to the sample speech or sample text. Specifically, the sample text includes historical text and current text, and the sample speech includes historical speech and current speech.

[0014] S1102: Input the data to be processed into a second machine learning model, so that the second machine learning model outputs a processed result. The second machine learning model is a pretrained model obtained according to the pretraining method described above. For example, the historical text, current text, historical speech, and current speech are input into the second machine learning model, i.e., the pretrained model. The text encoder, speech encoder, and modality fusion layer in the pretrained model process the inputs of the pretrained model and predict the user emotion category corresponding to the sample speech or text based on the fused representation vector (e.g., the first fused representation vector) output by the modality fusion layer. The user emotion category may be used as the processing result output by the pretrained model.

[0015] S1103: Train the second machine learning model based on the processed results output by the second machine learning model and the annotated results of the data to be processed to obtain a trained machine learning model. For example, the pre-trained model is trained or fine-tuned based on the user emotion category corresponding to the sample speech or sample text predicted by the pre-trained model and the user emotion category corresponding to the pre-annotated sample speech or sample text, thereby obtaining a trained machine learning model. Similarly, the pre-trained model can be trained or fine-tuned using sample data from an emotion recognition task, an intent analysis task, or a key information collection task. It will be appreciated that in this embodiment, the pre-trained model can be trained or fine-tuned using the emotion analysis task, the emotion recognition task, the intent analysis task, or the key information collection task individually. Alternatively, the pre-trained model can be trained or fine-tuned using the emotion analysis task, the emotion recognition task, the intent analysis task, and the key information collection task in combination. Furthermore, the pre-trained model is not limited to being trained or fine-tuned using the aforementioned tasks; for example, other tasks can also be used to train or fine-tune the pre-trained model. For example, the sample speech represents a conversation between a user and an intelligent dialogue system. During this conversation, the user and the intelligent dialogue system may discuss key information. For example, in a flight booking scenario, the intelligent dialogue system may ask the user what day and time of flight they require, and what type of ticket (e.g., discounted or non-discounted). These information, such as the day and time, and the type, are all key information. Therefore, this embodiment can train or fine-tune the pre-trained model using the key information predicted and pre-labeled by the pre-trained model. It is understood that when training or fine-tuning the pre-trained model based on sentiment analysis, emotion recognition, intent analysis, and / or key information collection tasks, new fully connected layers can be deployed in the pre-trained model, eliminating the need for fully connected layers 1, 2, and 3 as described above. This embodiment, for the first time, performs joint modeling based on speech semantics in a conversational scenario. Based on the temporal characteristics of speech, the first machine learning model is pre-trained using the time series prediction task. Furthermore, a cross-modal dialogue response selection task is introduced to pre-train the first machine learning model. The resulting pre-trained model not only aligns speech and text information in temporal order but also identifies contextual information. Experiments have shown that the solution provided by this embodiment improves existing speech and semantic modeling methods by 5-20 percentage points, enabling more accurate understanding of user speech and enhancing the user experience.Therefore, in use, the solution described in this embodiment not only learns more fine-grained and accurate speech-to-text alignment, but also captures more speech information, better helping the system understand user intent and improving the user experience. Furthermore, the method described in this embodiment shows improvements over both text-only and speech-only models, with a greater improvement over speech models, demonstrating that our model effectively facilitates joint modeling of speech and semantics. Furthermore, our method significantly improves upon existing state-of-the-art pre-training methods for joint speech-semantics modeling. Even for non-pre-trained methods, our approach surpasses the respective state-of-the-art (SOTA) methods by more than 1 percentage point on all tasks, and achieves a 3.5 percentage point improvement on the Multimodal Dataset (MOSI) dataset. Furthermore, on public datasets, this solution achieves a 10.1% improvement in accuracy compared to text-only methods, and even a 5-6% improvement compared to the most advanced multimodal methods. Figure 12 is a flow chart of the intelligent dialogue method provided by another embodiment of the present disclosure. This method can be performed by a cloud server. For example, the cloud server may be deployed with the trained or fine-tuned machine learning model described above. The cloud server may use this trained or fine-tuned machine learning model to implement the intelligent dialogue method described in this embodiment. In this embodiment, the method includes the following specific steps:

[0016] S1201: Obtain historical conversations between a target user and an intelligent dialogue system, as well as text information corresponding to the historical conversations. For example, the intelligent dialogue system may be pre-deployed on the cloud server, and the cloud server may obtain historical conversations between the target user and the intelligent dialogue system, as well as text information corresponding to the historical conversations.

[0017] S1202: Input the historical conversation and the text information into a trained machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, and key information. The machine learning model is trained according to the above method. For example, the cloud server may input the historical conversation and the text information into a trained or fine-tuned machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, and key information.

[0018] S1203: Conduct an intelligent dialogue with the target user based on at least one of the target user's emotion, mood, intention, and key information. For example, the cloud server may conduct an intelligent dialogue with the target user based on at least one of the target user's emotion, mood, intention, and key information. For example, if the last sentence in the historical dialogue is a question from the target user, the cloud server may generate a reply to the question based on at least one of the target user's emotion, mood, intention, and key information, and feed the reply back to the target user's terminal. If the last sentence in the historical dialogue is a reply from the intelligent dialogue system, the cloud server may generate a question based on at least one of the target user's emotion, mood, intention, and key information, and feed the question back to the target user's terminal, thereby allowing the target user to continue the dialogue with the intelligent dialogue system in the cloud server. This embodiment obtains historical conversations between a target user and an intelligent dialogue system, as well as the textual information corresponding to these historical conversations. These historical conversations and textual information are then input into a trained machine learning model, causing the model to output at least one of the target user's emotions, moods, intentions, and key information. This improves the machine learning model's accuracy in understanding the target user. Furthermore, based on at least one of the target user's emotions, moods, intentions, and key information, an intelligent dialogue is conducted with the target user. This allows the machine learning model to accurately understand the target user and thus engage in a precise dialogue with the target user. Figure 13 is a schematic diagram of the structure of a machine learning model training device provided in an embodiment of the present disclosure. The machine learning model training device provided by the embodiment of the present disclosure can execute the processing flow provided by the embodiment of the machine learning model training method. As shown in Figure 13, the machine learning model training device 130 includes: an acquisition module 131, which is configured to acquire voice information and text information corresponding to the voice information; an encoding module 132, which is configured to encode the text information and the voice information respectively to obtain text encoding information and voice encoding information; a fusion processing module 133, which is configured to fuse the text encoding information and the voice encoding information to obtain the fused encoding information; a prediction module 134, which is configured to predict the time information corresponding to each text unit in the text information in the voice information based on the encoding information corresponding to the text information in the fused encoding information; a pre-training module 135, which is configured to pre-train the first machine learning model based on the predicted time information and the position annotation of the voice unit corresponding to each text unit in the text information in the voice information.Optionally, when the encoding module 132 encodes the text information and the voice information respectively to obtain text encoding information and voice encoding information, it is specifically configured to: encode at least one encoding unit included in each text unit in the text information by a text encoder to obtain text encoding information, where the text encoding information includes a representation vector of each encoding unit; and encode each voice segment in the voice information by a voice encoder to obtain voice encoding information, where the voice encoding information includes a representation vector of each voice segment. Optionally, the fused encoding information includes fused representation vectors corresponding to the respective encoding units and fused representation vectors corresponding to the respective voice segments; when the prediction module 134 predicts the time information corresponding to each text unit in the text information in the voice information based on the fused encoding information, it is specifically set to: if the text unit includes one encoding unit, then according to the fused representation vector corresponding to the encoding unit, predict the start time and end time corresponding to the text unit in the voice information; if the text unit includes multiple encoding units, then according to the fused representation vector corresponding to the first encoding unit among the multiple encoding units, predict the start time corresponding to the text unit in the voice information, and according to the fused representation vector corresponding to the last encoding unit among the multiple encoding units, predict the end time corresponding to the text unit in the voice information. Optionally, when pre-training the first machine learning model based on the predicted time information and the positional annotations of the speech units corresponding to each text unit in the text information in the speech information, the pre-training module 135 is specifically configured to: construct a first loss function based on the start time and end time corresponding to the text unit in the speech information, and the pre-annotated start and end positions of the speech units corresponding to the text units in the speech information; and pre-train the first machine learning model based on the first loss function. Optionally, the first loss function includes a first difference and a second difference, wherein the first difference is a difference obtained based on the start time and the start position, and the second difference is a difference obtained based on the end time and the end position.Optionally, the speech information includes historical speech and current speech, and the text information includes historical text and current text. The model pre-training device 130 further includes: a judgment module 136 configured to determine, based on the fused encoded information, whether the historical speech and the current speech match, thereby obtaining a first judgment result, and to determine whether the historical text and the current text match, thereby obtaining a second judgment result. The pre-training module 135 is further configured to predict, based on the first judgment result and the second judgment result, the type of an input sample, the input sample including the speech information and the text information; construct a second loss function based on the predicted input sample type and the pre-labeled input sample type, and pre-train the first machine learning model based on the second loss function to obtain a pre-trained model. The model pre-training device of the embodiment shown in FIG13 can be configured to implement the technical solutions of the above-described method embodiments. The implementation principles and technical effects thereof are similar and are not further described here. FIG14 is a schematic structural diagram of a machine learning model training device provided in an embodiment of the present disclosure. The machine learning model training device provided in the embodiments of the present disclosure can execute the processing flow provided in the machine learning model training method embodiment. As shown in FIG14 , the machine learning model training device 140 includes: an acquisition module 141 configured to acquire sample data for a target task, wherein the sample data includes data to be processed and the annotation results of the data to be processed, and the data to be processed includes sample speech and sample text corresponding to the sample speech; an input module 142 configured to input the data to be processed into a second machine learning model so that the second machine learning model outputs a processed result, wherein the second machine learning model is a pre-trained model obtained according to the pre-training method described above; and a training module 143 configured to train the second machine learning model based on the processed results output by the second machine learning model and the annotation results of the data to be processed, thereby obtaining a trained machine learning model. Optionally, the target task includes at least one of the following: a sentiment analysis task, an emotion recognition task, an intent analysis task, or a key information collection task. The model training device in the embodiment shown in FIG14 can be configured to execute the technical solution of the above-mentioned method embodiment. Its implementation principles and technical effects are similar and will not be further described here. FIG15 is a schematic diagram of the structure of an intelligent dialogue device provided in an embodiment of the present disclosure.The intelligent dialogue device provided in the embodiments of the present disclosure can execute the processing flow provided in the embodiments of the intelligent dialogue method. As shown in Figure 15 , the intelligent dialogue device 150 includes: an acquisition module 151, configured to acquire historical dialogues between a target user and an intelligent dialogue system, as well as text information corresponding to the historical dialogues; an input module 152, configured to input the historical dialogues and text information into a trained machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, or key information. The machine learning model is trained according to the above-described method; and a dialogue module 153, configured to conduct an intelligent dialogue with the target user based on at least one of the target user's emotion, mood, intention, or key information. The intelligent dialogue device in the embodiment shown in Figure 15 can be configured to execute the technical solutions of the above-described method embodiments. The implementation principles and technical effects are similar and will not be further described here. The above describes the internal functions and structure of the model pre-training device, which can be implemented as an electronic device. Figure 16 is a schematic structural diagram of an electronic device embodiment provided in the embodiments of the present disclosure. As shown in Figure 16 , the electronic device includes a memory 161 and a processor 162. Memory 161 is configured to store programs. In addition to the aforementioned programs, memory 161 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method configured to operate on the electronic device, contact data, phone book data, messages, images, videos, etc. Memory 161 can be implemented using any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.The processor 162 is coupled to the memory 161 and executes a program stored in the memory 161, so as to be configured to: obtain voice information and text information corresponding to the voice information; encode the text information and the voice information separately to obtain text encoding information and voice encoding information; fuse the text encoding information and the voice encoding information to obtain fused encoding information; predict the time information corresponding to each text unit in the text information in the voice information based on the encoding information corresponding to the text information in the fused encoding information; and pre-train a first machine learning model based on the predicted time information and the position annotation of the voice unit corresponding to each text unit in the text information in the voice information to obtain a pre-trained model. Furthermore, as shown in FIG16 , the electronic device may further include other components such as a communication component 163, a power supply component 164, an audio component 165, and a display 166. FIG16 schematically illustrates only some components and does not imply that the electronic device includes only the components shown in FIG16 . The communication component 163 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 163 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 163 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), Bluetooth (BT), or other technologies. The power component 164 provides power to the various components of the electronic device. The power component 164 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device. The audio component 165 is configured to output and / or input audio signals. For example, the audio component 165 includes a microphone (M1C) that is configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signal may be further stored in the memory 161 or transmitted via the communication component 163. In some embodiments, the audio component 165 further includes a speaker configured to output audio signals. The display 166 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP).If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In addition, embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the model pre-training, training, and intelligent dialogue methods described in the above embodiments. It should be noted that, in this document, relational terms such as "first" and "second" are used to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further limitation, elements defined by the phrase "comprising a..." do not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements. The foregoing description is a specific embodiment of the present disclosure, intended to enable those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments described herein but is intended to have the widest scope consistent with the principles and novel features disclosed herein. Industrial Applicability: The machine learning model training, dialogue methods, apparatuses, devices, and media provided in the embodiments of the present disclosure are applied in machine learning scenarios. Voice information and text information corresponding to the voice information are acquired, and the text information and voice information are encoded, respectively, to obtain encoded text information and encoded voice information. Furthermore, the text encoding information and the speech encoding information are fused to obtain fused encoding information. Based on the fused encoding information, time information corresponding to each text unit in the text information in the speech information is predicted. Then, based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the text information in the speech information, a first machine learning model is pre-trained to obtain a pre-trained model.This allows the pre-trained model to align speech information and text information in time sequence. For example, the alignment between finer-grained text units in text information and finer-grained speech units in speech information can improve the pre-trained model's understanding of speech and text information, thereby improving the pre-trained model's accuracy in understanding user speech.

Claims

Claims 1. A machine learning model training method, the method comprising: Acquire voice information and text information corresponding to the voice information; Encode the text information and the voice information respectively to obtain text encoding information and voice encoding information; The text encoding information and the speech encoding information are fused to obtain fused encoding information; time information corresponding to each text unit in the text information in the speech information is predicted based on the encoding information corresponding to the text information in the fused encoding information; a first machine learning model is pre-trained based on the predicted time information and the position of the speech unit corresponding to each text unit in the text information in the speech information to obtain a pre-trained model.

2. The method according to claim 1, wherein: The text information and the voice information are respectively encoded to obtain text encoding information and voice encoding information, including: encoding at least one encoding unit respectively included in each text unit in the text information by a text encoder to obtain text encoding information, wherein the text encoding information includes a representation vector of each encoding unit; encoding each voice segment in the voice information by a voice encoder to obtain voice encoding information, wherein the voice encoding information includes a representation vector of each voice segment.

3. The method according to claim 2, wherein: The fused encoded information includes fused representation vectors corresponding to the respective encoding units and fused representation vectors corresponding to the respective voice segments; based on the fused encoded information, predicting the time information corresponding to each text unit in the text information in the voice information, including: if the text unit includes one encoding unit, predicting the start time and end time corresponding to the text unit in the voice information based on the fused representation vector corresponding to the encoding unit; if the text unit includes multiple encoding units, predicting the start time corresponding to the text unit in the voice information based on the fused representation vector corresponding to the first encoding unit among the multiple encoding units, and predicting the end time corresponding to the text unit in the voice information based on the fused representation vector corresponding to the last encoding unit among the multiple encoding units.

4. The method according to claim 3, wherein: Pre-training the first machine learning model according to the predicted time information and the position annotation of the speech unit corresponding to each text unit in the text information in the speech information, including: constructing a first loss function according to the start time and end time corresponding to the text unit in the speech information and the start position and end position of the speech unit corresponding to the text unit in the speech information that are pre-annotated; pre-training the first machine learning model according to the first loss function.

5. The method according to claim 4, wherein: The first loss function includes a first difference and a second difference, the first difference is a difference obtained according to the start time and the start position, and the second difference is a difference obtained according to the end time and the end position.

6. The method according to claim 1, wherein: The speech information includes historical speech and current speech, and the text information includes historical text and current text; the method further includes: judging whether the historical speech and the current speech match according to the fused encoded information to obtain a first judgment result, and judging whether the historical text and the current text match to obtain a second judgment result; predicting the type of an input sample according to the first judgment result and the second judgment result, the input sample including the speech information and the text information: A second loss function is constructed according to the predicted type of the input sample and the pre-labeled type of the input sample, and the first machine learning model is pre-trained according to the second loss function to obtain a pre-trained model.

7. A machine learning model training method, the method comprising: Obtain sample data of a target task, wherein the sample data includes data to be processed and annotated results of the data to be processed, wherein the data to be processed includes sample speech and sample text corresponding to the sample speech; input the data to be processed into a second machine learning model so that the second machine learning model outputs a processed result, wherein the second machine learning model is a pre-trained model obtained according to the pre-training method according to any one of claims 1 to 6; train the second machine learning model according to the processed results output by the second machine learning model and the annotated results of the data to be processed to obtain a trained machine learning model.

8. The method according to claim 7, wherein: The target task includes at least one of the following: sentiment analysis task, emotion recognition task, intention analysis task, and key information collection task.

9. An intelligent dialogue method, the method comprising: Acquire historical conversations between the target user and the intelligent dialogue system, as well as text information corresponding to the historical conversations; The historical conversation and the text information are input into a trained machine learning model, so that the machine learning model outputs at least one of the emotion, mood, intention, and key information of the target user, and the machine learning model is trained according to the method according to claim 7; and an intelligent conversation is conducted with the target user based on at least one of the emotion, mood, intention, and key information of the target user.

10. A machine learning model training device, comprising: An acquisition module, configured to acquire voice information and text information corresponding to the voice information; an encoding module, configured to encode the text information and the voice information respectively to obtain text encoding information and voice encoding information; a fusion processing module, configured to perform fusion processing on the text encoding information and the speech encoding information to obtain fused encoding information; A prediction module is configured to predict the time information corresponding to each text unit in the text information in the voice information based on the encoding information corresponding to the text information in the fused encoding information; a pre-training module is configured to pre-train the first machine learning model based on the predicted time information and the position annotation of the voice unit corresponding to each text unit in the text information in the voice information.

11. An electronic device, comprising: Memory; processor; And a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images