Machine learning model training, dialogue methods, devices, equipment and media

Abstract

This disclosure relates to a machine learning model training method, dialogue method, apparatus, device, and medium. The disclosure acquires speech information and corresponding text information, and encodes the text and speech information separately to obtain text-encoded information and speech-encoded information. The text-encoded information and speech-encoded information are fused to obtain fused encoded information. Based on the fused encoded information, the temporal information corresponding to each text unit in the text information is predicted in the speech information. Based on the predicted temporal information and the positional annotation of the speech unit corresponding to each text unit in the text information in the speech information, a first machine learning model is pre-trained to obtain a pre-trained model. This allows the pre-trained model to align speech and text information temporally, improving the pre-trained model's understanding of speech and text information and increasing the accuracy of the pre-trained model's understanding of user speech.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a machine learning model training, dialogue method, apparatus, device, and medium. Background Technology

[0002] Currently, intelligent customer service can engage in dialogue with users in some business scenarios, providing services such as business consultation and processing. Specifically, intelligent customer service can be implemented through machine learning models deployed on computing devices such as computers or computer clusters.

[0003] However, current machine learning models are not very accurate in understanding user speech. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a machine learning model training, dialogue method, apparatus, device, and medium to improve the understanding of speech and text information by a pre-trained model, thereby enhancing the accuracy of the pre-trained model's understanding of user speech.

[0005] In a first aspect, embodiments of this disclosure provide a machine learning model training method, including:

[0006] Acquire voice information and the corresponding text information;

[0007] The text information and the speech information are encoded respectively to obtain text encoded information and speech encoded information;

[0008] The text encoding information and the speech encoding information are fused to obtain fused encoding information;

[0009] Based on the encoding information corresponding to the text information in the fused encoding information, predict the time information corresponding to each text unit in the text information in the speech information;

[0010] The first machine learning model is pre-trained based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information.

[0011] Secondly, embodiments of this disclosure provide a machine learning model training method, the method comprising:

[0012] Obtain sample data for the target task. The sample data includes data to be processed and the annotation results of the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech.

[0013] The data to be processed is input into a second machine learning model, which outputs the processed result. The second machine learning model is a pre-trained model obtained according to the training method described in the first aspect.

[0014] Based on the processed results output by the second machine learning model and the labeled results of the data to be processed, the second machine learning model is trained to obtain the trained machine learning model.

[0015] Thirdly, embodiments of this disclosure provide an intelligent dialogue method, the method comprising:

[0016] Obtain historical dialogues between the target user and the intelligent dialogue system, as well as the text information corresponding to the historical dialogues;

[0017] The historical dialogue and the text information are input into the trained machine learning model, so that the machine learning model outputs at least one of the target user's sentiment, emotion, intention, and key information, and the machine learning model is trained according to the method described in the second aspect;

[0018] Intelligent dialogue is conducted with the target user based on at least one of the target user's emotions, feelings, intentions, and key information.

[0019] Fourthly, embodiments of this disclosure provide a machine learning model training apparatus, comprising:

[0020] The acquisition module is used to acquire voice information and the text information corresponding to the voice information;

[0021] An encoding module is used to encode the text information and the speech information respectively to obtain text encoded information and speech encoded information;

[0022] The fusion processing module is used to fuse the text encoding information and the speech encoding information to obtain fused encoding information;

[0023] The prediction module is used to predict the time information corresponding to each text unit in the text information in the speech information based on the encoding information corresponding to the text information in the fused encoding information;

[0024] The pre-training module is used to pre-train the first machine learning model based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information.

[0025] Fifthly, embodiments of this disclosure provide an electronic device, including:

[0026] Memory;

[0027] Processor; and

[0028] Computer programs;

[0029] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first, second, or third aspect.

[0030] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in the first, second, or third aspects.

[0031] The machine learning model training, dialogue method, apparatus, device, and medium provided in this disclosure acquire speech information and corresponding text information, and encode the text information and speech information respectively to obtain text-encoded information and speech-encoded information. Further, the text-encoded information and speech-encoded information are fused to obtain fused encoded information. Based on the fused encoded information, the time information corresponding to each text unit in the text information in the speech information is predicted. Then, based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the text information in the speech information, a first machine learning model is pre-trained to obtain a pre-trained model. This allows the pre-trained model to align speech information and text information temporally, for example, aligning finer-grained text units in the text information with finer-grained speech units in the speech information, thereby improving the pre-trained model's understanding of speech and text information and thus improving the accuracy of the pre-trained model's understanding of user speech. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0033] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart of the model pre-training method provided in the embodiments of this disclosure;

[0035] Figure 2 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0036] Figure 3 A schematic diagram of a machine learning model provided in an embodiment of this disclosure;

[0037] Figure 4 A schematic diagram illustrating a user's dialogue with an intelligent dialogue system provided in an embodiment of this disclosure;

[0038] Figure 5 A schematic diagram of timing prediction provided for another embodiment of this disclosure;

[0039] Figure 6 A schematic diagram of a machine learning model provided in another embodiment of this disclosure;

[0040] Figure 7 A schematic diagram illustrating the construction of a first loss function provided in another embodiment of this disclosure;

[0041] Figure 8 A schematic diagram illustrating various types of sample data provided in another embodiment of this disclosure;

[0042] Figure 9 A flowchart of a model pre-training method provided in another embodiment of this disclosure;

[0043] Figure 10 A schematic diagram of a machine learning model provided in another embodiment of this disclosure;

[0044] Figure 11 A flowchart of a model training method provided in another embodiment of this disclosure;

[0045] Figure 12 A flowchart of an intelligent dialogue method provided in another embodiment of this disclosure;

[0046] Figure 13 This is a schematic diagram of the structure of the model pre-training device provided in the embodiments of this disclosure;

[0047] Figure 14 This is a schematic diagram of the structure of the model training device provided in the embodiments of this disclosure;

[0048] Figure 15 A schematic diagram of the structure of the intelligent dialogue device provided in the embodiments of this disclosure;

[0049] Figure 16 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0050] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0051] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0052] It should be noted that the voice information (including but not limited to user voice collected by terminal devices) and text information (including but not limited to text information corresponding to user voice) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0053] In addition, the machine learning model training and dialogue methods provided in this application involve the following explanations of terms, detailed below:

[0054] Joint speech and semantic modeling: The system takes both speech and corresponding text transcription as input, and under the guidance of tasks such as speech-text alignment, it achieves a joint improvement in speech and text understanding.

[0055] Pre-training: An important method that allows models to learn unsupervised on large-scale corpora, which can significantly improve the results of downstream tasks.

[0056] Temporal sequence: refers to the characteristic that the input sequence has a temporal order. Both speech and text have temporal sequence.

[0057] Navigation: The user calls the voice robot.

[0058] Outbound calling: A voice robot makes phone calls to users.

[0059] Typically, intelligent customer service can engage in dialogue with users in certain business scenarios to provide services such as business consultation and processing. Specifically, intelligent customer service can be implemented using machine learning models deployed on computing devices such as computers or computer clusters. However, current machine learning models do not have high accuracy in understanding user speech. To address this issue, this disclosure provides a method for training a machine learning model, which will be described below with reference to specific embodiments.

[0060] Figure 1This is a flowchart illustrating a machine learning model training method provided in this embodiment. Specifically, this machine learning model training method can be a pre-training method. This method can be executed by a machine learning model training device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. The server can specifically be a cloud server, and the machine learning model training method can be executed in the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, each with computing, storage, and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide the service by providing a service interface, which users call to use the corresponding service. Service interfaces include Software Development Kits (SDKs), Application Programming Interfaces (APIs), etc. Furthermore, the machine learning model training method described in this embodiment is applicable to... Figure 2 The application scenarios shown are as follows. Figure 2 As shown, this application scenario includes a terminal 21 and a server 22. The server 22 can pre-train a first machine learning model using the machine learning model training method described in this embodiment. After pre-training, the first machine learning model becomes a pre-trained model. Furthermore, the pre-trained model can be stored on the server 22, or the server 22 can deploy the pre-trained model to the terminal 21 or other servers. The following is a combination of... Figure 2 This method will be described in detail, such as Figure 1 As shown, the specific steps of this method are as follows:

[0061] S101. Obtain voice information and the text information corresponding to the voice information.

[0062] For example, server 22 can be equipped with an intelligent dialogue system, such as... Figure 2The terminal 21 shown can collect user voice and send it to the server 22, where the intelligent dialogue system can respond with voice. Furthermore, the server 22 feeds back the voice response from the intelligent dialogue system to the terminal 21, and so on, allowing the user of terminal 21 to engage in voice dialogue with the intelligent dialogue system. During the dialogue, the server 22 can acquire voice information, which can be the voice conversation between the user and the intelligent dialogue system. Furthermore, the server 22 can also convert the voice conversation into corresponding text information using Automatic Speech Recognition (ASR), i.e., the text information is a text transcription of the voice conversation. Alternatively, the voice information acquired by the server 22 can be the user's voice, and the text information is a text transcription of that user's voice.

[0063] Specifically, the voice information and the corresponding text information can be used as input to the first machine learning model. In this embodiment, the text information may include at least one sentence. The voice information may include the voice of each sentence in the at least one sentence, or the voice of a portion of the at least one sentence.

[0064] S102. Encode the text information and the voice information respectively to obtain text encoded information and voice encoded information.

[0065] In this embodiment, the first machine learning model may include a text encoder and a speech encoder, wherein the text encoder can be used to encode text information to obtain text-encoded information, and the speech encoder can be used to encode speech information to obtain speech-encoded information.

[0066] In one feasible implementation, a text encoder can encode each sentence in the text information separately to obtain text encoding information, which includes a representation vector for each sentence. Similarly, a speech encoder can encode the speech of each sentence in the speech information separately to obtain speech encoding information, which includes a representation vector for the speech of each sentence.

[0067] In another feasible implementation, the text information and the speech information are encoded separately to obtain text-encoded information and speech-encoded information, including: encoding at least one encoding unit included in each text unit of the text information by a text encoder to obtain text-encoded information, wherein the text-encoded information includes the representation vector of each encoding unit; and encoding each speech segment in the speech information by a speech encoder to obtain speech-encoded information, wherein the speech-encoded information includes the representation vector of each speech segment.

[0068] For example, each sentence in this text information is composed of text units, which can be characters, words, phrases, sub-words, or single words. For instance, a sentence like "They won five rounds" can be segmented into five words when using words as text units: "he," "they," "won," "won," and "five rounds." Each text unit can also include at least one encoding unit. For example, "he" includes one character, so "he" includes one encoding unit, namely "he" itself. "Five rounds" includes two characters; if each character is used as an encoding unit, then "five rounds" includes two encoding units. Therefore, the sentence "They won five rounds" includes six encoding units: "he," "they," "won," "won," "five," and "rounds." Therefore, when encoding the sentence "They won five rounds" using a text encoder, these six encoding units can be encoded separately. After encoding, each encoding unit corresponds to a representation vector.

[0069] For example, consider the sentence "They won five games successively." This sentence contains five words, each of which is a text unit. Since the first four words are relatively short, each of them can be used as a separate encoding unit. However, the last word, "successfully," is longer, so it can be divided into two encoding units: "success" and "actively." Therefore, when encoding the sentence "They won five games successively" using a text encoder, six encoding units can be encoded: "They," "won," "five," "games," "success," and "actively." After encoding, each encoding unit corresponds to a representation vector.

[0070] Since the text information includes a sentence that is not limited to one, each sentence may include multiple text units, and each text unit may also include at least one encoding unit, when encoding the text information by the text encoder, each encoding unit in the text information can be encoded by the text encoder to obtain text encoding information, which includes the representation vector of each encoding unit.

[0071] Furthermore, the speech of each sentence in the speech information can be a 10-second audio signal. Before speech encoding, the speech of each sentence can be divided into multiple speech segments, for example, each speech segment can be 1 second long. Further, each speech segment in the speech information is encoded separately by a speech encoder to obtain speech encoding information, which includes the representation vector of each speech segment.

[0072] S103. The text encoding information and the speech encoding information are fused to obtain the fused encoding information.

[0073] In this embodiment, the first machine learning model may further include a modality fusion layer. After obtaining the text encoding information and speech encoding information as described above, the text encoding information and speech encoding information can be concatenated, and the concatenated result can be used as the input of the modality fusion layer. The modality fusion layer then performs fusion processing on the text encoding information and speech encoding information to obtain fused encoding information, which can also be denoted as a joint representation vector. Specifically, the input and output of the modality fusion layer are in one-to-one correspondence. For example, if the input of the modality fusion layer includes representation vectors of m encoding units and representation vectors of n speech segments (i.e., the input of the modality fusion layer includes m+n representation vectors), then the output of the modality fusion layer also includes m+n representation vectors, and the m+n representation vectors in the input and m+n representation vectors in the output are in one-to-one correspondence. For each representation vector output by the modality fusion layer, the representation vector simultaneously contains text components and speech components. Therefore, each representation vector output by the modality fusion layer can be denoted as a fused representation vector.

[0074] S104. Based on the encoding information corresponding to the text information in the fused encoding information, predict the time information corresponding to each text unit in the text information in the speech information.

[0075] For example, text information includes m coding units, and speech information includes n speech segments. The input of the modality fusion layer includes the representation vectors of the m coding units and the representation vectors of the n speech segments. Since the input and output of the modality fusion layer are in one-to-one correspondence, the first m representation vectors output by the modality fusion layer are the coding information corresponding to the text information, and the last n representation vectors output by the modality fusion layer are the coding information corresponding to the speech information. Since each representation vector output by the modality fusion layer contains both text and speech components, the first machine learning model can predict the time information corresponding to each text unit in the text information in the speech information based on the first m representation vectors output by the modality fusion layer. For example, the time information corresponding to each text unit in the speech information includes the start and end times of the text unit in the speech information. It can be understood that text units such as "he," "they," "win," "it," and "five rounds" are text-type information, and each text unit corresponds to a speech word, which is speech-type information. The speech word corresponding to a text unit can be the pronunciation of the text unit by the user or the intelligent dialogue system. Therefore, the start time of the text unit in the speech information is specifically the start time of the speech word corresponding to the text unit in the speech information (e.g., the start time of pronunciation), and the end time of the text unit in the speech information is specifically the end time of the speech word corresponding to the text unit in the speech information (e.g., the end time of pronunciation).

[0076] S105. Based on the predicted time information and the position label of the speech unit corresponding to each text unit in the speech information, the first machine learning model is pre-trained to obtain a pre-trained model.

[0077] Specifically, in this embodiment, the position of the speech unit corresponding to each text unit in the text information can be pre-labeled in the speech information. Further, based on the time information corresponding to each text unit in the text information predicted by the first machine learning model in the speech information, and the pre-labeled position of the speech unit corresponding to each text unit in the text information in the speech information, a first loss function is constructed, and then the first machine learning model is pre-trained based on the first loss function.

[0078] For example, the text information in this embodiment is "They won five rounds", and the voice information is the voice of "They won five rounds". In this embodiment, the positions of the voice units corresponding to each text unit in the text information can be pre-labeled in the voice information. For example, the text information includes 5 text units: "he", "they", "won", "five", and "rounds". The voice information includes the voice words corresponding to "he", "they", "won", "five", and "rounds" respectively, which are recorded as voice units. For example, the voice word corresponding to "he" is the pronunciation of "he". In this embodiment, the start position (such as start time or start moment) and end position (such as end time or end moment) of the voice word corresponding to "he" in the voice information can be pre-labeled. Similarly, the start positions and end positions of the voice words corresponding to "they", "won", "five", and "rounds" in the voice information can be pre-labeled respectively. Further, according to the start times and end times of "he", "they", "won", "five", and "rounds" in the voice information predicted by the first machine learning model, as well as the start positions and end positions of "he", "they", "won", "five", and "rounds" in the voice information pre-labeled respectively, a first loss function is constructed.

[0079] Specifically, according to the start times of "he", "they", "won", "five", and "rounds" in the voice information predicted by the first machine learning model, and the start positions of the voice words corresponding to "he", "they", "won", "five", and "rounds" pre-labeled respectively in the current voice, a first difference is calculated. According to the end times of "he", "they", "won", "five", and "rounds" in the voice information predicted by the first machine learning model, and the end positions of the voice words corresponding to "he", "they", "won", "five", and "rounds" pre-labeled respectively in the current voice, a second difference is calculated. Further, a first loss function is constructed according to the first difference and the second difference. And the first machine learning model is pre-trained according to the first loss function to obtain a pre-trained model.

[0080] This embodiment of the disclosure acquires speech information and corresponding text information, and encodes the text information and speech information respectively to obtain text-encoded information and speech-encoded information. Further, the text-encoded information and speech-encoded information are fused to obtain fused encoded information. Based on the fused encoded information, the time information corresponding to each text unit in the text information in the speech information is predicted. Then, based on the predicted time information and the positional annotation of the speech unit corresponding to each text unit in the text information in the speech information, a first machine learning model is pre-trained to obtain a pre-trained model. This allows the pre-trained model to align speech information and text information temporally, for example, aligning finer-grained text units in the text information with finer-grained speech units in the speech information, thereby improving the pre-trained model's understanding of speech and text information and thus improving the accuracy of the pre-trained model's understanding of user speech.

[0081] like Figure 3 The diagram shows the structure of the first machine learning model. Specifically, this first machine learning model includes a text encoder, a speech encoder, and a modality fusion layer. The following section will discuss this further. Figure 3 The pre-training method is described in detail.

[0082] Specifically, the input to the first machine learning model includes text information and speech information; that is, the input to the first machine learning model is multimodal information. Optionally, the speech information includes historical speech and current speech, and the text information includes historical text and current text. In other words, for each modality of information, both historical and current information are included. For example, when the speech information is a voice dialogue between a user and an intelligent dialogue system, the dialogue content continuously increases over time. Therefore, during the dialogue, some speech can be considered historical speech, and some can be considered current speech. Similarly, since the text information is a transcribed version of the voice dialogue, some sentences in the text information can be considered historical text, and some sentences can be considered current text.

[0083] like Figure 4 The diagram illustrates a dialogue between a user and an intelligent dialogue system. Assume... Figure 4 The dialogue shown is a text transcription. Each sentence spoken by the user can be transcribed into one round of text, and each sentence spoken by the intelligent dialogue system can also be transcribed into one round of text. Therefore, as... Figure 4The dialogue shown consists of 8 rounds of text, with "AAAAAA", "BBBBBB", "CCCC", "DDDD", "EEEEEE", "FFFFFF", "GGGGG", and "HHHHH" each representing one round of text. In this embodiment, the first 7 rounds of text can be recorded as historical text, and the last round, "HHHHH", can be recorded as the current text. Similarly, since each round of text corresponds to a speech, and each speech is recorded as one round of speech, there are 8 rounds of speech. The first 7 rounds of speech are recorded as historical speech, and the last round, "HHHHH", is recorded as the current speech. Furthermore, the maximum length of each round of speech can be 10 seconds.

[0084] Since text and speech information are two different modalities, the storage space occupied by text and speech of the same number of rounds may be different. For example, speech of the same number of rounds may occupy more storage space. Therefore, text and speech of the same number of rounds can be used as input to the first machine learning model, or text of the first number of rounds and speech of the second number of rounds can be used as input to the first machine learning model, where the first number of rounds is greater than the second number of rounds.

[0085] For example, in this embodiment, 8 rounds of text and 2 rounds of speech can be selected as such. Figure 3 The input to the first machine learning model is shown. Specifically, the 8 rounds of text include 7 rounds of historical text and 1 round of current text. The 2 rounds of speech include 1 round of historical speech and 1 round of current speech. Furthermore, the 1 round of current text and 1 round of current speech are aligned; for example, the 1 round of current text is "HHHHH", and the 1 round of current speech is the speech corresponding to "HHHHH". Therefore, as... Figure 3 The historical text shown can be the historical text from the 7 rounds mentioned here, such as... Figure 3 The current text shown can be the current text for the first round here. For example... Figure 3 The historical voice recording shown can be the first round of historical voice recordings here, such as... Figure 3 The current speech shown can be the current speech in one round here. That is, seven rounds of historical text and one round of current text constitute text information, and one round of historical speech and one round of current speech constitute speech information. Further, the text information is encoded by a text encoder to obtain text-encoded information; the speech information is encoded by a speech encoder to obtain speech-encoded information.

[0086] like Figure 3 As shown, to distinguish between the 7 rounds of historical text and the 1 round of current text, a preset symbol can be added at the beginning of the 7 rounds of historical text. <s> Add a preset symbol between the 7 rounds of historical text and the 1 round of current text.< / s>Alternatively, preset symbols can be added between each pair of adjacent text rounds to separate each round of text. Similarly, a method similar to... <s> and< / s> The preset symbols can be used to separate historical speech from current speech, or to separate each round of speech.

[0087] Furthermore, through a text encoder... <s> The seven rounds of historical texts mentioned above< / s> The current text of the first round, as described above, is encoded. The historical speech and the current speech of the first round, as described above, are encoded using a speech encoder.

[0088] For example, given the seven rounds of historical text and one round of current text as described above, each round of text can be a sentence. Each sentence consists of text units, and each text unit can also include at least one encoded unit. Therefore, it can be like this... Figure 3 The historical and current texts shown are each divided into several encoding units; for example, box 31 represents any one encoding unit. Simultaneously, preset symbols... <s> It can be used as an encoding unit with preset symbols.< / s> It can be used as a coding unit. For example... Figure 3 As shown, after encoding each encoding unit separately using a text encoder, text encoding information is obtained, which includes the representation vector of each encoding unit. For example, this text encoding information includes, for instance, the representation vector of each encoding unit. Figure 3 The representation vectors within box 32 shown, each small box within box 32 (e.g., box 33) represents the representation vector of a coding unit.

[0089] Additionally, in this embodiment, as Figure 3 The historical and current audio signals shown are both 10-second audio signals. Before encoding, the historical audio can be divided into multiple audio segments, and the current audio can also be divided into multiple audio segments. For example, each audio segment could be 1 second long. Figure 3 As shown, each speech segment in the historical speech is encoded by a speech encoder, and each speech segment in the current speech is encoded by a speech encoder to obtain speech encoding information, which includes the representation vector of each speech segment. For example, the speech encoding information includes, for example,... Figure 3 The representation vectors within box 34 are shown below. Each small box within box 34 (e.g., box 35) represents a representation vector of a speech segment.

[0090] like Figure 3As shown, the text encoding information represented by box 32 and the speech encoding information represented by box 34 are concatenated, and the concatenated result is used as the input to the modality fusion layer. This allows the modality fusion layer to fuse the text encoding information and the speech encoding information, obtaining fused encoding information, which can also be denoted as a joint representation vector. For example, the fused encoding information includes, for instance,... Figure 3 The representation vectors within box 36 are shown below. Each smaller box within box 36 (e.g., box 37) represents a representation vector output by the modality fusion layer. For example, box 33 represents a preset character. <s>The representation vector, box 37 represents the first representation vector output by the modality fusion layer. This first representation vector is the fusion result of text encoding information and speech encoding information, that is, this first representation vector is the preset character. <s> The representation vector, the representation vector of each encoded unit in the historical text, and the preset character< / s> The first representation vector is a fusion of the representation vectors of the current text, the representation vectors of each coding unit in the current text, the representation vectors of each speech segment in the historical speech, and the representation vectors of each speech segment in the current speech. Therefore, the first representation vector contains not only text components but also speech components. Similarly, the other representation vectors output by the modality fusion layer also contain both text and speech components.

[0091] Furthermore, from Figure 3 A portion of the representation vectors within each representation vector in box 36 is extracted, for example, the representation vectors output by the modality fusion layer corresponding to the current text are extracted, i.e., the representation vectors within box 38. Further, based on the representation vectors within box 38, the temporal information corresponding to each text unit in the current text within the current speech is predicted. For example, the current speech is 10 seconds long. After normalizing the length of the current speech, the start position of the current speech can be recorded as 0, and the end position as 1. For example... Figure 3 As shown, the start time of each text unit in the current text is predicted in the current speech through fully connected layer 1, and the end time of each text unit in the current text is predicted in the current speech through fully connected layer 2. For example... Figure 5 As shown, assuming the current text is "They won five rounds", based on the representation vector corresponding to the current text output by the modality fusion layer, the start time (e.g., 0.05) and end time (e.g., 0.16) of "he" in the current speech can be predicted, the start time (e.g., 0.18) and end time (e.g., 0.32) of "they" in the current speech can be predicted, and so on, the start time (e.g., 0.067) and end time (e.g., 0.97) of "five rounds" in the current speech can be predicted.

[0092] For example, the current text and the current speech are samples respectively. Therefore, the positions of the speech units corresponding to each text unit in the current text in the speech information can be pre-annotated. For example, the current speech includes the speech words corresponding to "他 (tā)", "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" respectively. The speech words are denoted as speech units. For example, the speech word corresponding to "他 (tā)" is the pronunciation of "他 (tā)". In this embodiment, the start position (such as start time or start moment) and the end position (such as end time or end moment) of the speech word corresponding to "他 (tā)" in the current speech can be pre-annotated. Similarly, the start positions and the end positions of the speech words corresponding to "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" in the current speech can be pre-annotated. Further, according to the start times of "他 (tā)", "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" predicted by the fully connected layer 1 in the current speech respectively, and the start positions of the speech words corresponding to "他 (tā)", "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" in the current speech pre-annotated respectively, a first difference is constructed. According to the end times of "他 (tā)", "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" predicted by the fully connected layer 2 in the current speech respectively, and the end positions of the speech words corresponding to "他 (tā)", "们 (men)", "赢 (yíng)", "了 (le)", and "五把 (wǔ bǎ)" in the current speech pre-annotated respectively, a second difference is constructed. Further, a first loss function is constructed according to the first difference and the second difference, and the first machine learning model as shown in Figure 3 is pre-trained through the first loss function to obtain a pre-trained model.

[0093] In the above embodiment, the encoded information after the fusion processing includes the fusion representation vectors corresponding to the respective encoding units and the fusion representation vectors corresponding to the respective speech segments; according to the encoded information after the fusion processing, predicting the time information corresponding to each text unit in the text information in the speech information includes: if the text unit includes one encoding unit, predicting the start time and the end time corresponding to the text unit in the speech information according to the fusion representation vector corresponding to the encoding unit; if the text unit includes multiple encoding units, predicting the start time corresponding to the text unit in the speech information according to the fusion representation vector corresponding to the first encoding unit among the multiple encoding units, and predicting the end time corresponding to the text unit in the speech information according to the fusion representation vector corresponding to the last encoding unit among the multiple encoding units.

[0094] As Figure 6 shown, the respective fusion representation vectors output by the modality fusion layer include a preset character <s> The corresponding fusion representation vector, the fusion representation vector corresponding to each coding unit in the historical text, and the preset character< / s> The corresponding fused representation vectors, the fused representation vectors corresponding to each coding unit in the current text, and the fused representation vectors corresponding to each speech segment. Specifically, if there are 8 rounds of historical text, 8 rounds of historical speech, and the 8 rounds of historical text and 8 rounds of historical speech are pairwise aligned, then according to the fused representation vectors corresponding to each coding unit in the historical text, the time information corresponding to each text unit in the historical text in the historical speech can be predicted. For example, according to the fused representation vectors corresponding to each coding unit in the first round of historical text, predict the start time and end time corresponding to each text unit in the first round of historical text in the first round of historical speech, and so on, predict the start time and end time corresponding to each text unit in the eighth round of historical text in the eighth round of historical speech. In addition, since the current text and the current speech are aligned, the start time and end time corresponding to each text unit in the current text in the current speech can be predicted according to the fused representation vectors corresponding to each coding unit in the current text.

[0095] For example, the current text is "They won five games", and "he", "they", "won", "five", "games" are coding units respectively. The fused representation vectors output by the modality fusion layer include the fused representation vectors corresponding to "he", "they", "won", "five", "games" respectively. For example Figure 6 as shown, A represents the fused representation vector corresponding to "he", and so on, F represents the fused representation vector corresponding to "games". Since "he", "they", "won" are text units respectively, and each text unit includes one coding unit. For example, "he" is a text unit, and this text unit includes one coding unit, which is "he", that is, "he" is both a text unit and a coding unit included in this text unit. Therefore, according to the fused representation vector A corresponding to "he", the start time and end time corresponding to "he" in the current speech can be predicted. For example Figure 6 as shown, the fully connected layer 1 includes parameter W start , the fully connected layer 2 includes parameter W end , the start time corresponding to "he" in the current speech predicted by the fully connected layer 1 is denoted as W start A, and the end time corresponding to "he" in the current speech predicted by the fully connected layer 2 is denoted as W end A. Similarly, the fully connected layer 1 can also predict the start times corresponding to "they", "won" in the current speech respectively, and the fully connected layer 2 can also predict the end times corresponding to "they", "won" in the current speech respectively.

[0096] Furthermore, since "five" is a text unit, but this text unit includes multiple coding units, for example, "five" is the first coding unit in this text unit, and "handle" is the last coding unit in this text unit, the start time of "five" in the current speech can be predicted based on the fusion representation vector corresponding to the first coding unit, and the end time of "five" in the current speech can be predicted based on the fusion representation vector corresponding to the last coding unit. For example, if the fusion representation vector corresponding to "five" is denoted as E, and the fusion representation vector corresponding to "handle" is denoted as F, then the start time of "five" in the current speech predicted by the fully connected layer 1 is denoted as W. start E, and the end time of the "five strokes" predicted by the fully connected layer 2 in the current speech is denoted as W. end F. For example Figure 5 The figure shows the start time of each text unit predicted by fully connected layer 1 and the end time of each text unit predicted by fully connected layer 2.

[0097] Optionally, the first machine learning model is pre-trained based on the predicted time information and the positional annotations of the speech units corresponding to each text unit in the speech information. This includes: constructing a first loss function based on the start and end times corresponding to the text units in the speech information and the pre-annotated start and end positions of the speech units corresponding to the text units in the speech information; and pre-training the first machine learning model based on the first loss function. Optionally, the first loss function includes a first difference and a second difference, wherein the first difference is the difference obtained based on the start time and the start position, and the second difference is the difference obtained based on the end time and the end position.

[0098] For example Figure 7 As shown, the duration of the current audio is 10 seconds, meaning it starts at second 0 and ends at second 10. This embodiment can pre-mark the start and end positions of the audio units corresponding to each text unit in the current text, such as "They won five rounds." For example, the audio unit corresponding to "he" starts at second 0.6 and ends at second 1.7, and so on. The audio unit corresponding to "five rounds" starts at second 6.9 and ends at second 9.9.

[0099] Since the start time predicted by fully connected layer 1 and the end time predicted by fully connected layer 2, as described above, are both values ​​between 0 and 1, this embodiment can normalize the duration of the current speech to match these values ​​on an order of magnitude. For example, the duration of the current speech can be normalized to 1 second, that is, the start position of the current speech can be normalized to second 0, and the end position of the current speech can be normalized to second 1. This ensures that the start and end positions of the speech units corresponding to each text unit in "They won five rounds" are normalized in the current speech. For example, after normalization, the speech unit corresponding to "he" has a start position of second 0.06 and an end position of second 0.17, and so on, the speech unit corresponding to "five rounds" has a start position of second 0.69 and an end position of second 0.99. In other words, during the normalization process, the speech unit is uniformly divided by the duration of the current speech (e.g., 10 seconds).

[0100] like Figure 7 As shown, for the same text unit in the current text, such as "he", the fully connected layer 1 predicts that the start time of "he" in the current speech is 0.05 seconds, while the pre-labeled speech unit corresponding to "he" in the current speech starts at 0.06 seconds. There is a certain difference between the start time and the start position, which is denoted as the first difference. Simultaneously, the fully connected layer 2 predicts that the end time of "he" in the current speech is 0.16 seconds, while the pre-labeled speech unit corresponding to "he" ends at 0.17 seconds. There is also a certain difference between the end time and the end position, which is denoted as the second difference. Furthermore, similar first and second differences exist for other text units in the current text. This embodiment is an illustrative example of predicting the start and end times of each text unit in the current text. In other embodiments, the time information corresponding to each text unit in historical text in historical speech can also be predicted. For example, based on the fusion representation vectors corresponding to each coding unit in the first round of historical text, the start and end times of each text unit in the first round of historical text are predicted in the first round of historical speech. Similarly, the start and end times of each text unit in the eighth round of historical text are predicted in the eighth round of historical speech. Therefore, for each text unit in each round of historical text, there also exist similar first and second differences.

[0101] Therefore, this embodiment can construct a first loss function based on the first difference and second difference corresponding to each text unit in the multi-round text (including historical text and current text). Further, based on this first loss function, the loss function can be applied to... Figure 3 The first machine learning model shown is pre-trained. For example, the first loss function... This can be expressed as the following formula:

[0102]

[0103] The first loss function uses Mean Squared Error (MSE), which is the squared difference between the predicted result (e.g., the predicted start or end time) and the true location (e.g., the pre-labeled start or end position). `i` represents the i-th round of text, `m` represents the number of rounds or text elements, `j` represents the j-th text unit in the i-th round, such as the j-th word, and `n` represents the number of words included in the i-th round of text. W start W represents the parameters included in the fully connected layer 1. end This indicates the parameters included in the fully connected layer 2. This represents the fused representation vector corresponding to the first encoded unit in the j-th text unit of the i-th round of text. This represents the fused representation vector corresponding to the last encoded unit in the j-th text unit of the i-th round of text. If the j-th text unit includes one encoded unit, then the first and last encoded units here are the same encoded unit, i.e. and Same. If the j-th text unit includes multiple encoded units, then the first and last encoded units here are different, i.e. and Different. Understandably, during the pre-training process, W... start and W end These are the parameters to be trained. This represents the start time of the j-th text unit in the i-th round of text predicted by the fully connected layer 1 in the i-th round of speech. s represents the end time of the j-th text unit in the i-th round of text predicted by the fully connected layer 2 in the i-th round of speech. ij This represents the starting position of the j-th text unit in the i-th round of speech before normalization, e ij L represents the end position of the j-th text unit in the i-th round of speech before normalization, as pre-annotated. a L represents the duration of the i-th round of speech, or the maximum duration across multiple rounds of speech. a Used for the normalization process described above.

[0104] Furthermore, according to the first loss function, for example... Figure 3 or Figure 6 When the first machine learning model shown is pre-trained, it can be used for, for example Figure 3 or Figure 6 The parameters of each training structure shown, such as the text encoder, speech encoder, modality fusion layer, fully connected layer 1, and fully connected layer 2, are iteratively updated until the number of iterations reaches the preset number or the parameters converge.

[0105] For example, based on the similarity or difference between the predicted results output by the first machine learning model and the pre-labeled true locations, adjustments are made as follows: Figure 3 or Figure 6 The parameters in each training structure shown make the predicted result output by the first machine learning model gradually similar to the pre-labeled true location during subsequent iterative training. For example, the parameters can be adjusted based on the gradient data between the predicted result output by the first machine learning model and the pre-labeled true location. Here, the gradient is essentially a vector representing the directional derivative of a function at a given point, where the directional derivative reaches its maximum value along that direction; that is, the function changes most rapidly and has the largest rate of change along that direction (the direction of the gradient) at that point. It is based on this principle that the direction of parameter adjustment can be guided, making the predicted result output by the first machine learning model closer to the pre-labeled true location.

[0106] As described above, the sample data used when pre-training the first machine learning model includes text information and speech information. For example, the text information includes 7 rounds of historical text and 1 round of current text, and the speech information includes 1 round of historical speech and 1 round of current speech. The 7 rounds of historical text and 1 round of current text are 8 consecutive sentences in the same dialogue, and similarly, the 1 round of historical speech and 1 round of current speech are 2 consecutive speech segments in the same dialogue. That is, there is coherent semantics between the 7 rounds of historical text and 1 round of current text, and there is coherent semantics between the 1 round of historical speech and 1 round of current speech. In other embodiments, three other types of sample data can be derived from this sample data. For example, by replacing the 1 round of current text in the sample data with a sentence from another dialogue, the first type of derived sample data is obtained, which results in the semantics between the 7 rounds of historical text and 1 round of current text no longer being coherent, but the semantics between the 1 round of historical speech and 1 round of current speech are preserved. For example, replacing the current voice in one round of the sample data with a voice from another dialogue yields a second type of derived sample data. This results in semantic incoherence between the current voice and the history voice, but semantic coherence between the seven-round history text and the current text remains. As another example, replacing the current text in one round of the sample data with a sentence from another dialogue, and replacing the current voice in one round of the sample data with a voice from another dialogue, yields a third type of derived sample data. This results in semantic incoherence between the seven-round history text and the current text, and also between the history voice and the current voice. If the original sample data is denoted as... Figure 8 The sample data shown has a label of 0. The other three types of sample data derived from this original sample data are, in order: Figure 8 The sample data shown includes Label 1, Label 2, and Label 3. Among them, Representing historical text, Indicates the current text. Indicates historical speech. Indicates the current voice. This indicates the current text after being replaced with other sentences. This indicates the current voice after it has been replaced with another voice.

[0107] During the pre-training process of the first machine learning model, the pre-training method also includes, for example: Figure 9 The following steps are shown:

[0108] S901. Based on the fused encoding information, determine whether the historical speech and the current speech match to obtain a first determination result, and determine whether the historical text and the current text match to obtain a second determination result.

[0109] This embodiment can use sample data with Label 0, Label 1, Label 2, or Label 3 as input to the first machine learning model. That is, as... Figure 10 The current text shown may be or Similarly, such as Figure 10 The current voice shown may be or After processing by the text encoder, speech encoder, and modality fusion layer in the first machine learning model, the modality fusion layer outputs multiple fused representation vectors. In this embodiment, the first fused representation vector output by the modality fusion layer can be input into the fully connected layer 3. The fully connected layer 3 is used to determine whether the historical speech and the current speech in the input match, that is, whether the semantics between the historical speech and the current speech are coherent, to obtain a first judgment result, and to determine whether the historical text and the current text in the input match, that is, whether the semantics between the historical text and the current text are coherent, to obtain a second judgment result.

[0110] Understandably, if the current text is In this case, matching the historical text with the current text means that the speech between the historical text and the current text is coherent, and the historical text and the current text have a clear contextual relationship in terms of semantic content. In this situation, the historical text and the current text belong to the same context within the same dialogue. If the current text is... This means the historical text and the current text do not match, i.e., they are not within the same context of the dialogue. Similarly, if the current speech is... Matching historical and current speech means that the speech is coherent between them, and that the historical and current speech have a clear contextual relationship in terms of semantic content. In this case, the historical and current speech belong to the same context within the same dialogue. If the current speech is... If the historical speech and the current speech do not match, it means that the historical speech and the current speech are not in the same context of the dialogue.

[0111] Therefore, the first judgment result, as described above, has two possible outcomes: either the historical speech matches the current speech, or they do not match. Similarly, the second judgment result also has two possible outcomes: either the historical text matches the current text, or they do not match.

[0112] S902. Based on the first judgment result and the second judgment result, predict the type of the input sample, wherein the input sample includes the voice information and the text information.

[0113] For example, fully connected layer 3 can predict the type of the input sample based on the first and second judgment results. For instance, if the first judgment result indicates a match between historical speech and current speech, and the second judgment result indicates a match between historical text and current text, then fully connected layer 3 predicts the input sample to be a sample with Label 0. If the first judgment result indicates a match between historical speech and current speech, and the second judgment result indicates a mismatch between historical text and current text, then fully connected layer 3 predicts the input sample to be a sample with Label 1. If the first judgment result indicates a mismatch between historical speech and current speech, and the second judgment result indicates a match between historical text and current text, then fully connected layer 3 predicts the input sample to be a sample with Label 2. If the first judgment result indicates a mismatch between historical speech and current speech, and the second judgment result indicates a mismatch between historical text and current text, then fully connected layer 3 predicts the input sample to be a sample with Label 3. In other words, fully connected layer 3 can predict which type of sample—Label 0, Label 1, Label 2, or Label 3—the input sample can be based on the first and second judgment results.

[0114] In one possible implementation, the fully connected layer 3 can directly output the type of the input sample it predicts, for example, Label=0, Label=1, Label=2, or Label=3.

[0115] In another possible implementation, the fully connected layer 3 can be a four-class classification task, meaning the output of the fully connected layer 3 can be four probability values, which represent the probabilities of the input sample being of type 0, 1, 2, and 3, respectively. Furthermore, the type corresponding to the highest probability value among these four values ​​is taken as the type of the input sample predicted by the fully connected layer 3.

[0116] S903. Based on the predicted type of the input sample and the pre-labeled type of the input sample, a second loss function is constructed, and the first machine learning model is pre-trained based on the second loss function. This embodiment can also pre-label whether historical speech and current speech in the input sample actually constitute contextual information, and pre-label whether historical text and current text in the input sample actually constitute contextual information. That is, this embodiment can pre-label the actual type of the input sample, i.e., pre-label which type of sample is among Label 0, Label 1, Label 2, or Label 3.

[0117] Furthermore, in this embodiment, a second loss function can be constructed based on the type of the input sample predicted by the fully connected layer 3 and the actual type of the pre-labeled input sample. This second loss function can be a classification cross-entropy loss function. Further, based on this second loss function, the following... Figure 10 The first machine learning model shown is pre-trained.

[0118] This embodiment uses the fused encoded information to determine whether the historical speech and the current speech match, obtaining a first determination result, and to determine whether the historical text and the current text match, obtaining a second determination result. This allows the first machine learning model to predict the type of the input sample based on the first and second determination results. Furthermore, the type of the input sample is pre-labeled based on whether the historical speech and the current speech actually constitute contextual information, and whether the historical text and the current text actually constitute contextual information. Based on the pre-labeled input sample type and the type of the input sample predicted by the first machine learning model, a second loss function is constructed, and the first machine learning model is pre-trained using the second loss function to obtain a pre-trained model. This allows the pre-trained model to recognize contextual information, which helps improve the accuracy of the pre-trained model in recognizing user intent, sentiment analysis, etc.

[0119] It is understood that in this embodiment, the first machine learning model can be pre-trained based on either the first loss function or the second loss function alone, or it can be pre-trained jointly based on the first loss function and the second loss function. Furthermore, in addition to the temporal prediction task (e.g., predicting the start and end times of text units in speech) and cross-modal dialogue response selection task (e.g., predicting the type of input samples) mentioned above, the first machine learning model can also be pre-trained through other tasks. For example, it can also be pre-trained through a masking prediction task. For example, in this embodiment, some words in historical text or current text can be masked, and / or some speech segments in historical speech or current speech can be masked. Then, the masked samples are input into the first machine learning model, and further, the masked words and / or speech segments are predicted based on the fusion representation vector output by the modality fusion layer. Thus, a third loss function is constructed based on the predicted masked words and / or speech segments and the actual masked words and / or speech segments, and the first machine learning model is pre-trained based on this third loss function. This allows us to utilize different modal information, such as text and speech, to model the interaction between the two modalities.

[0120] Understandably, after pre-training the first machine learning model, a pre-trained model can be obtained. This pre-trained model can be used as the basis for the model. Further training or fine-tuning of the pre-trained model can be carried out using sample data from downstream tasks to obtain the trained machine learning model, which can then perform downstream tasks.

[0121] Figure 11 This is a flowchart illustrating a machine learning model training method according to another embodiment of the present disclosure. This machine learning model training method can be a method for training or fine-tuning a pre-trained model. In this embodiment, the specific steps of the method are as follows:

[0122] S1101. Obtain sample data for the target task. The sample data includes data to be processed and the annotation results of the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech.

[0123] For example, the pre-trained model can be stored in server 22, and server 22 can further train or fine-tune the pre-trained model. Specifically, server 22 can obtain sample data for the target task, which can be a downstream task as described above. The sample data includes data to be processed and the annotation results of the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech. The annotation results of the data to be processed are pre-annotated user sentiment categories, user emotion categories, user intent, or key information in the sample speech or sample text based on the sample speech and / or sample text.

[0124] Optionally, the target task includes at least one of the following: sentiment analysis task, emotion recognition task, intent analysis task, and key information collection task.

[0125] For example, when the target task is sentiment analysis, the sample data acquired by server 22 includes sample speech, the corresponding sample text, and the pre-labeled user sentiment category corresponding to the sample speech or sample text. Specifically, the sample text includes historical text and current text, and the sample speech includes historical speech and current speech.

[0126] S1102. Input the data to be processed into the second machine learning model, so that the second machine learning model outputs the processed result. The second machine learning model is a pre-trained model obtained according to the pre-training method described above.

[0127] For example, the historical text, current text, historical speech, and current speech are input into a second machine learning model, i.e., a pre-trained model. The text encoder, speech encoder, and modality fusion layer in this pre-trained model process the input and predict the user sentiment category corresponding to the sample speech or text based on the fusion representation vector output by the modality fusion layer (e.g., the first fusion representation vector). This user sentiment category can then be used as the processing result output by the pre-trained model.

[0128] S1103. Based on the processed result output by the second machine learning model and the labeling result of the data to be processed, the second machine learning model is trained to obtain the trained machine learning model.

[0129] For example, the pre-trained model can be trained or fine-tuned based on the user sentiment category corresponding to the sample speech or sample text predicted by the pre-trained model and the user sentiment category corresponding to the sample speech or sample text pre-labeled, thereby obtaining the trained machine learning model.

[0130] Similarly, sample data from emotion recognition, intent analysis, or key information gathering tasks can be used to train or fine-tune the pre-trained model. It is understood that this embodiment can train or fine-tune the pre-trained model using emotion analysis, emotion recognition, intent analysis, or key information gathering tasks individually. Alternatively, the pre-trained model can be trained or fine-tuned using a combination of these tasks. Furthermore, the training or fine-tuning of the pre-trained model is not limited to the tasks described above; other tasks can also be used.

[0131] For example, the sample audio represents a conversation between a user and an intelligent dialogue system. During this conversation, the user and the intelligent dialogue system may discuss some key information. For instance, in a scenario where a user is booking a flight, the intelligent dialogue system might ask the user which date and time they need the flight, and what type of flight they need (e.g., discounted or non-discounted). The date, time, and type of flight are all key pieces of information. Therefore, this embodiment can use the key information predicted by the pre-trained model and the pre-annotated key information to train or fine-tune the pre-trained model.

[0132] It is understandable that when training or fine-tuning the pre-trained model for sentiment analysis tasks, emotion recognition tasks, intent analysis tasks, and / or key information collection tasks, new fully connected layers can be deployed in the pre-trained model, without the need for fully connected layer 1, fully connected layer 2, fully connected layer 3, etc. as described above.

[0133] This embodiment, for the first time in a dialogue scenario, performs joint modeling based on speech and semantics. Based on the temporal characteristics of speech, a first machine learning model is pre-trained through a temporal prediction task, and a cross-modal dialogue response selection task is introduced for further pre-training. This allows the pre-trained model to not only align speech and text information temporally but also recognize contextual information. Experiments have shown that the solution provided in this embodiment offers a 5-20 percentage point improvement over existing speech and semantic modeling methods, enabling more accurate understanding of user speech and enhancing user experience. Therefore, in practice, the solution described in this embodiment not only learns finer-grained and more accurate speech-text alignment but also captures more speech information, better helping the system understand user intent and improving user experience.

[0134] Furthermore, the method described in this embodiment outperforms both pure text and pure speech models, with a greater improvement over the speech model, indicating that our model effectively facilitates joint speech and semantic modeling. Moreover, our method represents a significant improvement over existing state-of-the-art (SOTA) methods for joint speech and semantic modeling. Even with non-pre-trained methods, we outperform each state-of-the-art (SOTA) method by more than 1 percentage point on all tasks, achieving a remarkable 3.1 percentage point improvement on the Multimodal Dataset (MOSI).

[0135] Furthermore, on publicly available datasets, our approach achieves a 10.1% improvement in accuracy compared to plain text methods, and even achieves a 5-6% improvement compared to state-of-the-art multimodal methods.

[0136] Figure 12 This is a flowchart illustrating an intelligent dialogue method according to another embodiment of this disclosure. The method can be executed by a cloud server, for example, a cloud server deployed with a trained or fine-tuned machine learning model as described above. The cloud server can implement the intelligent dialogue method described in this embodiment using the trained or fine-tuned machine learning model. In this embodiment, the specific steps of the method are as follows:

[0137] S1201. Obtain the historical dialogue between the target user and the intelligent dialogue system, as well as the text information corresponding to the historical dialogue.

[0138] For example, the cloud server may be pre-deployed with an intelligent dialogue system, which can obtain the historical dialogue between the target user and the intelligent dialogue system, as well as the text information corresponding to the historical dialogue.

[0139] S1202. Input the historical dialogue and the text information into the trained machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, and key information. The machine learning model is trained according to the method described above.

[0140] For example, the cloud server can input the historical conversation and the text information into a machine learning model that has been trained or fine-tuned, so that the machine learning model outputs at least one of the target user's emotions, feelings, intentions, and key information.

[0141] S1203. Engage in intelligent dialogue with the target user based on at least one of the target user's emotions, feelings, intentions, and key information.

[0142] For example, the cloud server can engage in intelligent dialogue with the target user based on at least one of the target user's emotions, feelings, intentions, and key information. For instance, if the last sentence in a historical dialogue is a question from the target user, the cloud server can generate a response based on at least one of the target user's emotions, feelings, intentions, and key information, and send the response back to the target user's terminal. If the last sentence in a historical dialogue is a response from the intelligent dialogue system, the cloud server can generate a question based on at least one of the target user's emotions, feelings, intentions, and key information, and send the question back to the target user's terminal, thus allowing the target user and the intelligent dialogue system in the cloud server to continue their dialogue.

[0143] This embodiment acquires historical dialogues between the target user and the intelligent dialogue system, along with the corresponding text information. These historical dialogues and text information are then input into a trained machine learning model, causing the model to output at least one of the target user's sentiment, emotion, intent, and key information. This improves the accuracy of the machine learning model's understanding of the target user. Furthermore, based on at least one of the target user's sentiment, emotion, intent, and key information, the model engages in intelligent dialogue with the target user. This allows the machine learning model to conduct precise dialogue with the target user based on a precise understanding of them.

[0144] Figure 13 This is a schematic diagram of a machine learning model training apparatus provided in an embodiment of this disclosure. The machine learning model training apparatus provided in this embodiment can execute the processing flow provided in the machine learning model training method embodiment, such as… Figure 13 As shown, the machine learning model training device 130 includes:

[0145] The acquisition module 131 is used to acquire voice information and text information corresponding to the voice information;

[0146] Encoding module 132 is used to encode the text information and the voice information respectively to obtain text encoded information and voice encoded information;

[0147] The fusion processing module 133 is used to perform fusion processing on the text encoding information and the speech encoding information to obtain fused encoding information;

[0148] Prediction module 134 is used to predict the time information corresponding to each text unit in the text information in the speech information based on the encoding information corresponding to the text information in the fused encoding information;

[0149] The pre-training module 135 is used to pre-train the first machine learning model based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information.

[0150] Optionally, the encoding module 132 encodes the text information and the speech information respectively to obtain text-encoded information and speech-encoded information, specifically for:

[0151] The text information is encoded by encoding at least one encoding unit included in each text unit by a text encoder to obtain text encoding information, which includes the representation vector of each encoding unit.

[0152] Each speech segment in the speech information is encoded by a speech encoder to obtain speech encoding information, which includes a representation vector for each speech segment.

[0153] Optionally, the fused encoding information includes the fused representation vector corresponding to each encoding unit and the fused representation vector corresponding to each speech segment.

[0154] When the prediction module 134 predicts the time information corresponding to each text unit in the text information in the speech information based on the fused encoded information, it is specifically used for:

[0155] If the text unit includes an encoding unit, then the start time and end time of the text unit in the speech information are predicted based on the fusion representation vector corresponding to the encoding unit.

[0156] If the text unit includes multiple coding units, then the start time of the text unit in the speech information is predicted based on the fusion representation vector corresponding to the first coding unit among the multiple coding units, and the end time of the text unit in the speech information is predicted based on the fusion representation vector corresponding to the last coding unit among the multiple coding units.

[0157] Optionally, when the pre-training module 135 pre-trains the first machine learning model based on the predicted time information and the position annotations of the speech units corresponding to each text unit in the speech information, it is specifically used for:

[0158] A first loss function is constructed based on the start and end times of the text unit in the speech information, and the start and end positions of the speech units corresponding to the text units in the speech information that are pre-labeled.

[0159] The first machine learning model is pre-trained based on the first loss function.

[0160] Optionally, the first loss function includes a first difference and a second difference, wherein the first difference is a difference obtained based on the start time and the start position, and the second difference is a difference obtained based on the end time and the end position.

[0161] Optionally, the speech information includes historical speech and current speech, and the text information includes historical text and current text; the model pre-training device 130 further includes: a judgment module 136, used to judge whether the historical speech and the current speech match based on the fused encoded information to obtain a first judgment result, and to judge whether the historical text and the current text match to obtain a second judgment result; the pre-training module 135 is further used to predict the type of input sample based on the first judgment result and the second judgment result, the input sample including the speech information and the text information; construct a second loss function based on the predicted type of input sample and the pre-labeled type of input sample, and pre-train the first machine learning model based on the second loss function to obtain a pre-trained model.

[0162] Figure 13 The model pre-training device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0163] Figure 14 This is a schematic diagram of a machine learning model training apparatus provided in an embodiment of this disclosure. The machine learning model training apparatus provided in this embodiment can execute the processing flow provided in the machine learning model training method embodiment, such as… Figure 14 As shown, the machine learning model training device 140 includes:

[0164] The acquisition module 141 is used to acquire sample data of the target task. The sample data includes data to be processed and the annotation results of the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech.

[0165] The input module 142 is used to input the data to be processed into the second machine learning model, so that the second machine learning model outputs the processed result. The second machine learning model is a pre-trained model obtained according to the pre-training method described above.

[0166] The training module 143 is used to train the second machine learning model based on the processed result output by the second machine learning model and the labeling result of the data to be processed, so as to obtain the trained machine learning model.

[0167] Optionally, the target task includes at least one of the following:

[0168] Sentiment analysis tasks, emotion recognition tasks, intent analysis tasks, and key information collection tasks.

[0169] Figure 14 The model training apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0170] Figure 15 This is a schematic diagram of the intelligent dialogue device provided in an embodiment of the present disclosure. The intelligent dialogue device provided in this embodiment of the present disclosure can execute the processing flow provided in the intelligent dialogue method embodiment, such as... Figure 15 As shown, the intelligent dialogue device 150 includes:

[0171] The acquisition module 151 is used to acquire the historical dialogue between the target user and the intelligent dialogue system, as well as the text information corresponding to the historical dialogue;

[0172] The input module 152 is used to input the historical dialogue and the text information into the trained machine learning model, so that the machine learning model outputs at least one of the target user's emotion, mood, intention, and key information. The machine learning model is trained according to the method described above.

[0173] The dialogue module 153 is used to conduct intelligent dialogue with the target user based on at least one of the target user's emotions, feelings, intentions, and key information.

[0174] Figure 15 The intelligent dialogue device shown in the embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0175] The above describes the internal functions and structure of the model pre-training device, which can be implemented as an electronic device. Figure 16 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. (See attached diagram.) Figure 16 As shown, the electronic device includes a memory 161 and a processor 162.

[0176] Memory 161 is used to store programs. In addition to the programs described above, memory 161 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0177] The memory 161 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0178] Processor 162 is coupled to memory 161 and executes the program stored in memory 161 for:

[0179] Acquire voice information and the corresponding text information;

[0180] The text information and the speech information are encoded respectively to obtain text encoded information and speech encoded information;

[0181] The text encoding information and the speech encoding information are fused to obtain fused encoding information;

[0182] Based on the encoding information corresponding to the text information in the fused encoding information, predict the time information corresponding to each text unit in the text information in the speech information;

[0183] Based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information, the first machine learning model is pre-trained to obtain a pre-trained model.

[0184] Furthermore, such as Figure 16 As shown, the electronic device may also include other components such as a communication component 163, a power supply component 164, an audio component 165, and a display 166. Figure 16 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 16 The components shown.

[0185] Communication component 163 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 163 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 163 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0186] Power supply component 164 provides power to various components of an electronic device. Power supply component 164 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0187] Audio component 165 is configured to output and / or input audio signals. For example, audio component 165 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 161 or transmitted via communication component 163. In some embodiments, audio component 165 also includes a speaker for outputting audio signals.

[0188] Display 166 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0189] In addition, this disclosure also provides a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the model pre-training, training, and intelligent dialogue methods described in the above embodiments.

[0190] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / s>

Claims

1. A method for training a machine learning model, wherein, The method includes: Acquire voice information and the corresponding text information; The text information and the speech information are encoded respectively to obtain text encoded information and speech encoded information; The text encoding information and the speech encoding information are fused to obtain fused encoding information; Based on the encoded information corresponding to the text information in the fused encoded information, predict the time information corresponding to each text unit in the text information in the speech information, including: if the text unit includes one encoded unit, predict the start time and end time of the text unit in the speech information based on the fusion representation vector corresponding to the encoded unit; if the text unit includes multiple encoded units, predict the start time of the text unit in the speech information based on the fusion representation vector corresponding to the first encoded unit among the multiple encoded units, and predict the end time of the text unit in the speech information based on the fusion representation vector corresponding to the last encoded unit among the multiple encoded units, wherein the fused encoded information includes: the fusion representation vector corresponding to each encoded unit included in each text unit, and the fusion representation vector corresponding to each speech segment in the speech information; Based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information, the first machine learning model is pre-trained to obtain a pre-trained model.

2. The method according to claim 1, wherein, The text information and the speech information are encoded separately to obtain text-encoded information and speech-encoded information, including: The text information is encoded by encoding at least one encoding unit included in each text unit by a text encoder to obtain text encoding information, which includes the representation vector of each encoding unit. Each speech segment in the speech information is encoded by a speech encoder to obtain speech encoding information, which includes a representation vector for each speech segment.

3. The method according to claim 1, wherein, Based on the predicted time information and the position annotations of the speech units corresponding to each text unit in the speech information, the first machine learning model is pre-trained, including: A first loss function is constructed based on the start and end times of the text unit in the speech information, and the start and end positions of the speech units corresponding to the text units in the speech information that are pre-labeled. The first machine learning model is pre-trained based on the first loss function.

4. The method according to claim 3, wherein, The first loss function includes a first difference and a second difference, wherein the first difference is the difference obtained based on the start time and the start position, and the second difference is the difference obtained based on the end time and the end position.

5. The method according to claim 1, wherein, The voice information includes historical voice and current voice, and the text information includes historical text and current text; The method further includes: Based on the fused encoding information, it is determined whether the historical speech and the current speech match to obtain a first determination result, and it is determined whether the historical text and the current text match to obtain a second determination result; Based on the first judgment result and the second judgment result, the type of the input sample is predicted, wherein the input sample includes the voice information and the text information; Based on the predicted input sample type and the pre-labeled input sample type, a second loss function is constructed, and the first machine learning model is pre-trained based on the second loss function to obtain a pre-trained model.

6. A method for training a machine learning model, wherein, The method includes: Obtain sample data for the target task. The sample data includes data to be processed and the annotation results of the data to be processed. The data to be processed includes sample speech and sample text corresponding to the sample speech. The data to be processed is input into a second machine learning model, so that the second machine learning model outputs the processed result. The second machine learning model is a pre-trained model obtained according to the machine learning model training method as described in any one of claims 1-5. Based on the processed results output by the second machine learning model and the labeled results of the data to be processed, the second machine learning model is trained to obtain the trained machine learning model.

7. The method according to claim 6, wherein, The target task includes at least one of the following: Sentiment analysis tasks, emotion recognition tasks, intent analysis tasks, and key information collection tasks.

8. An intelligent dialogue method, wherein, The method includes: Obtain historical dialogues between the target user and the intelligent dialogue system, as well as the text information corresponding to the historical dialogues; The historical dialogue and the text information are input into the trained machine learning model, so that the machine learning model outputs at least one of the target user's sentiment, emotion, intention, and key information, and the machine learning model is trained according to the method described in claim 6; Intelligent dialogue is conducted with the target user based on at least one of the target user's emotions, feelings, intentions, and key information.

9. A machine learning model training device, wherein, include: The acquisition module is used to acquire voice information and the text information corresponding to the voice information; An encoding module is used to encode the text information and the speech information respectively to obtain text encoded information and speech encoded information; The fusion processing module is used to fuse the text encoding information and the speech encoding information to obtain fused encoding information; The prediction module is configured to predict the time information corresponding to each text unit in the text information in the speech information based on the encoding information corresponding to the text information in the fused encoding information. This includes: if the text unit includes one encoding unit, predicting the start and end times of the text unit in the speech information based on the fusion representation vector corresponding to the encoding unit; if the text unit includes multiple encoding units, predicting the start time of the text unit in the speech information based on the fusion representation vector corresponding to the first encoding unit among the multiple encoding units, and predicting the end time of the text unit in the speech information based on the fusion representation vector corresponding to the last encoding unit among the multiple encoding units. The fused encoding information includes: the fusion representation vectors corresponding to each encoding unit included in each text unit, and the fusion representation vectors corresponding to each speech segment in the speech information. The pre-training module is used to pre-train the first machine learning model based on the predicted time information and the position annotation of the speech unit corresponding to each text unit in the speech information.

10. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images