Label generation method and device, computer readable storage medium and computer equipment

Abstract

The application discloses a label generation method and device, a computer readable storage medium and a computer device. The method comprises the following steps: obtaining dialogue audio data of a target object and a preset scale, wherein the preset scale comprises prior knowledge under each scene dimension; performing identification processing on the dialogue audio data to obtain each scene dimension, dialogue text data and answer text data under each scene dimension; determining a first classification result of the target object in a time sequence coding dimension according to the dialogue text data, the answer text data and the prior knowledge under each scene dimension, and time sequence information of each scene dimension in the dialogue audio data; determining a second classification result of the target object in a preset scale dimension according to the dialogue text data, the answer text data and the prior knowledge under each scene dimension; generating a target classification label of the target object according to the first classification result and the second classification result, and improving the accuracy of the target classification label.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a tag generation method, apparatus, computer-readable storage medium, and computer equipment. Background Technology

[0002] In the current era of explosive growth in the data industry, collecting data from various dimensions such as users' social attributes, consumption habits, and preferences, and then characterizing the features of users or products, and analyzing and statistically mining the potential value information from these features, can be seen as the foundation for enterprises to apply big data and a prerequisite for targeted advertising and personalized recommendations.

[0003] Furthermore, with the accelerating pace of modern life and increasingly fierce social competition, generating user classification tags, such as depression level tags, has become a demand.

[0004] Currently, most user classification tags are generated based on scale assessments, such as using scales to ask questions and rate responses to ultimately provide classification tags, or using analogical reasoning, which employs sequence learning from a large amount of labeled data to classify tags based on simple interactive dialogue voice information. Scale-based assessments are a human-based method, and the determined classification tags are influenced by subjective human factors. Experienced and inexperienced individuals may produce vastly different classification tags, affecting the accuracy of the tags. While analogical reasoning is easier to implement, it suffers from the problem of failing to capture key information in long dialogues with many turns, also impacting the accuracy of the classification tags. Summary of the Invention

[0005] This application provides a label generation method, apparatus, computer-readable storage medium, and computer device, which can improve the accuracy of generating target classification labels for target objects.

[0006] This application provides a label generation method, including:

[0007] The dialogue audio data corresponding to the target object is obtained, and the dialogue audio data is processed to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

[0008] Obtain a preset scale corresponding to the target object. The preset scale includes prior knowledge for each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data.

[0009] Based on the dialogue text data, the response text data, and the prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data, the first classification result of the target object in the temporal coding dimension is determined.

[0010] Based on the dialogue text data, the response text data, and the prior knowledge in each scene dimension, determine the second classification result of the target object in the preset scale dimension;

[0011] Based on the first classification result and the second classification result, a target classification label is generated for the target object.

[0012] This application also provides a label generation method, including:

[0013] Obtain the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, and obtain the training preset scale of the training target object. The training preset scale includes training prior knowledge under each scene dimension in multiple scene dimensions. The training prior knowledge is predetermined based on the training audio data.

[0014] The training dialogue audio data is processed to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension.

[0015] Obtain the initial classification label generation model;

[0016] Based on the training dialogue text data, the training response text data, and the training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data, the initial classification label generation model is used to determine the first classification result of the training target object in the temporal encoding dimension.

[0017] Based on the training dialogue text data, the training response text data, and the training prior knowledge in each scene dimension, the initial classification label generation model is used to determine the training scene score of the training target object in each scene dimension;

[0018] The overall loss value is determined based on the training first classification result, the first label result, the training scene score, and the training prior knowledge;

[0019] The training parameters in the initial classification label generation model are updated according to the overall loss value until the overall loss value meets the preset conditions to obtain the trained classification label generation model.

[0020] This application embodiment also provides a label generation apparatus, including:

[0021] The acquisition module is used to acquire the dialogue audio data corresponding to the target object;

[0022] The recognition module is used to perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

[0023] The acquisition module is also used to acquire a preset scale corresponding to the target object. The preset scale includes prior knowledge for each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data.

[0024] The first determining module is used to determine the first classification result of the target object in the temporal coding dimension based on the dialogue text data, the response text data and the prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data.

[0025] The second determining module is used to determine the second classification result of the target object on the preset scale dimension based on the dialogue text data, the response text data and the prior knowledge in each scene dimension;

[0026] The generation module is used to generate target classification labels for the target object based on the first classification result and the second classification result.

[0027] This application embodiment also provides a label generation apparatus, including:

[0028] The training acquisition module is used to acquire the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, and to acquire the training preset scale of the training target object. The training preset scale includes training prior knowledge under each scene dimension in multiple scene dimensions. The training prior knowledge is predetermined based on the training audio data.

[0029] The training recognition module is used to recognize and process the training dialogue audio data to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension.

[0030] The training acquisition module is used to acquire the initial classification label generation model;

[0031] The first training determination module is used to determine the first classification result of the target object in the temporal encoding dimension by using the initial classification label generation model based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data;

[0032] The second training determination module is used to determine the training scene score of the training target object in each scene dimension based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension, using the initial classification label generation model.

[0033] The loss value determination module is used to determine the overall loss value based on the training first classification result, the first label result, the training scene score, and the training prior knowledge.

[0034] An update module is used to update the training parameters in the initial classification label generation model according to the overall loss value until the overall loss value meets a preset condition, so as to obtain the trained classification label generation model.

[0035] This application also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps in the tag generation method as described in any of the above embodiments.

[0036] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the steps in the tag generation method described in any of the above embodiments by calling the computer program stored in the memory.

[0037] The tag generation method, apparatus, computer-readable storage medium, and computer device provided in this application, when generating target classification tags for target objects, consider not only the first classification in the time-encoding dimension but also the second classification result in the preset scale dimension, thereby improving the accuracy of generating target classification tags for target objects. Furthermore, both the first classification result in the time-encoding dimension and the second classification result in the preset scale dimension consider prior knowledge from the preset scale, integrating prior knowledge into the determination process of the first and second classification results, thus improving the accuracy of both the first and second classification results in the time-encoding dimension and the second classification result in the preset scale dimension, further enhancing the accuracy of generating target classification tags for target objects. In addition, when generating the first classification result in the time-encoding dimension, the temporal information of each scene dimension in the dialogue audio data is considered, i.e., the contextual information of the dialogue audio data is considered, integrating global information from the entire dialogue audio data, improving the accuracy of the first classification result, and further enhancing the accuracy of generating target classification tags for target objects. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the label generation method provided in the embodiments of this application.

[0040] Figure 2 This is a schematic diagram of the data preprocessing process provided in the embodiments of this application.

[0041] Figure 3 This is a schematic diagram of the text information provided in the embodiments of this application.

[0042] Figure 4 This is a schematic diagram of the classification label generation model provided in the embodiments of this application.

[0043] Figure 5 This is a schematic diagram of a multi-granularity knowledge reasoning module provided in an embodiment of this application.

[0044] Figure 6 This is a schematic diagram of a sub-process of the label generation method provided in an embodiment of this application.

[0045] Figure 7 This is a schematic diagram of a sub-process of the label generation method provided in an embodiment of this application.

[0046] Figure 8This is a schematic diagram of another sub-process of the label generation method provided in the embodiments of this application.

[0047] Figure 9 This is another schematic diagram of the label generation method provided in the embodiments of this application.

[0048] Figure 10 This is a schematic diagram of the label generation device provided in an embodiment of this application.

[0049] Figure 11 This is another schematic diagram of the label generation device provided in an embodiment of this application.

[0050] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a label generation method, apparatus, computer-readable storage medium, and computer device. Specifically, the label generation method of this application can be executed by a computer device, and the label generation apparatus of this application is integrated into the computer device. The computer device can be a terminal or a server, etc. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, personal computer (PC), robot, or other terminal device. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services and cloud databases.

[0053] Figure 1 This is a flowchart illustrating a label generation method provided in an embodiment of this application. The label generation method is applied in a computer device and includes the following steps.

[0054] 101. Obtain the dialogue audio data corresponding to the target object, and perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

[0055] The dialogue audio data includes at least two different role objects, such as a first role object and a second role object. For example, the first role object is a doctor, and the second role object is a patient. The corresponding target classification label could be a depression level classification label or a disease level classification label, etc., and the second role object is the target object. This embodiment of the application is used as an example for illustration. In some other embodiments, the first role object is a teacher, and the second role object is a student, etc., and the second role object is the target object. The corresponding target classification label could be a level label, etc., from a knowledge mastery level label.

[0056] The dialogue audio data corresponding to the target audience can be understood as dialogue audio data that includes the audio data of the target audience. For example, when a doctor inquires about a patient's condition, the dialogue audio data between the doctor and the patient includes the patient's audio data, which corresponds to the dialogue audio data of the target audience. The audio data of the target audience includes the content of multiple rounds of interactive communication between the doctor and the user. For example, based on scales such as the Hamilton Anxiety Scale (HAMD) or the PHQ9 scale, multiple rounds of questions and answers are used to obtain the content of multiple rounds of interactive communication.

[0057] Acquire the dialogue audio data of the target object and perform recognition processing on the dialogue audio data.

[0058] like Figure 2 As shown, the dialogue audio data undergoes role separation, role classification, speech recognition, and scene dimension recognition processing to obtain the corresponding text information. This text information includes multiple scene dimensions involved in the dialogue audio data, dialogue text data for each scene dimension, and response text data for each scene dimension. Specifically, the dialogue text data for each scene dimension refers to the text data corresponding to the dialogue between at least two different role objects, and the response text data for each scene dimension refers to the text data corresponding to the target object's response to other role objects. It is important to note that the response text data is specific to the target object. This step considers the impact of the target object's (e.g., the patient's) response text data on the final target classification label.

[0059] In one embodiment, the steps for identifying and processing the dialogue audio data include: performing role separation and role classification processing on the dialogue audio data to obtain at least two different role objects, one of which is a target object; performing speech recognition processing on the dialogue audio data to obtain dialogue text data corresponding to the at least two different role objects and the response text data of the target object; and performing scene dimension recognition on the dialogue text data corresponding to the at least two different role objects to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

[0060] Furthermore, the dialogue audio data can be processed separately for role separation, speech recognition, and scene dimension recognition.

[0061] Specifically, the role separation engine determines the speaker's role object in the dialogue audio data. For example, the role separation engine obtains the sound source information, the sound source location and voiceprint features corresponding to the sound source information in the dialogue audio data, and determines the role object corresponding to the sound source location based on the voiceprint features. In this way, the first role object and the second role object are separated.

[0062] Speech recognition processing involves converting dialogue audio data into dialogue text data. For specific details, please refer to existing speech recognition processing methods, which will not be elaborated upon here.

[0063] The role classification process primarily distinguishes between the separated primary and secondary role objects, such as differentiating between doctors and patients. Rule-based matching can be used to differentiate between the separated primary and secondary role objects. For example, this includes: first, constructing a database of commonly used doctor-related question patterns using a scale such as the HAMD scale, and calculating the edit distance between these patterns and the input text, such as dialogue text data, to determine if the person is a doctor; second, constructing a database of keywords used by doctors in their inquiries, such as "have you," "how are you," "do you think," etc., and determining the primary and secondary role objects by statistically analyzing the frequency of these keywords in the entire dialogue audio data. Because the dialogue audio data is based on the HAMD scale, the accuracy of the role classification process is relatively high.

[0064] The scene dimension identification process primarily involves segmenting the entire dialogue text data into 17 scene dimension segments of the HAMD scale to meet subsequent processing requirements. Scene dimension identification can be performed using neural network models or deep learning models. For example, a Bi-directional Long Short-Term Memory (BiLSTM) neural network model can be used for scene dimension identification. Each time step of the BiLSTM model takes a single question-answer pair as input and outputs the scene dimension category to which the current question-answer pair belongs. Because the scene dimensions of the HAMD scale are quite distinct, the actual accuracy of scene dimension identification is relatively high.

[0065] After role separation, speech recognition, role classification, and scene-level recognition (or data preprocessing), the output is... Figure 2 The text information in the text.

[0066] 102. Obtain the preset scale corresponding to the target object. The preset scale includes prior knowledge for each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data.

[0067] The pre-defined scale is a scale designed for the target population, such as the HAMD scale. It is important to note that the target population in this step is the same as the target population in the steps described above.

[0068] Because each scenario dimension of the HAMD scale has its own separate scenario score and scoring criteria, and these scenario scores and scoring criteria are summarized and refined by a large number of psychiatric experts in practice, they possess rich prior knowledge. That is, the pre-defined scale includes prior knowledge for each scenario dimension, which may include scenario scores and corresponding options for the 17 scenario dimensions of the pre-determined HAMD scale.

[0069] This prior knowledge can be determined based on dialogue audio data. For example, the content of the dialogue audio data can be used to determine the scene scores and corresponding options for the 17 scene dimensions of the HAMD scale. For instance, when a doctor is interviewing a patient, they will ask questions for each of the 17 scene dimensions of the HAMD scale, and determine the scene scores and corresponding options for the 17 scene dimensions of the HAMD scale based on the patient's answers and performance.

[0070] The preset scale is presented in the form of a data table, which can obtain prior knowledge for each scene dimension, namely the scene score and the corresponding option content for each scene score.

[0071] Table 1 shows the scoring criteria and corresponding scenario scoring results for some scenario dimensions of the HAMD scale. The "item" refers to the scenario dimension. The HAMD scale has 17 scenario dimensions, but Table 1 only shows two. The scenario score for the first scenario dimension is 2, corresponding to the option "expressing oneself spontaneously in conversation." The scenario score for the second scenario dimension is 1, corresponding to the option "blaming oneself, feeling that one has burdened others."

[0072] Table 1. Scoring criteria and corresponding scenario scoring results for some scenario dimensions of the HAMD scale.

[0073]

[0074]

[0075] After the above steps, the following can be obtained: Figure 3 The text information shown is as follows. Here, `scence_sequence` represents the sequence of scene dimensions, with "0" under `scence_sequence` indicating the 0th scene dimension. The text content in `sent` represents the dialogue text data, "depressive mood" in `scence` represents the specific scene dimension name, `score` represents the scene rating (the 0th scene dimension has a rating of 2), `score_query` contains the options for the scene rating, and the text content in `answer` represents the response text data.

[0076] from Figure 3 As can be seen, each scenario dimension includes dialogue text data, response text data, and prior knowledge. Figure 3 The text information shown serves as the basis for subsequent processing. A series of processes are performed based on this text information to generate target classification labels for the target objects.

[0077] In one embodiment, this application provides a classification label generation model, which can be represented by MT-MGKR. The classification label generation model can determine a first classification result of the target object in the time-series encoding dimension and a second classification result in the preset scale dimension, and generate a target classification label for the target object based on the first and second classification results. Figure 4 The diagram shown is a schematic representation of the classification label generation model provided in an embodiment of this application. In the following text, the classification label generation model is used to perform a series of processes on the obtained text information to generate target classification labels for the target object.

[0078] To incorporate prior knowledge from a pre-defined scale, embodiments of this application design a multi-granularity knowledge reasoning module. Figure 5This is a schematic diagram of the multi-granularity knowledge inference module in the classification label generation model provided in this application embodiment. The multi-granularity knowledge inference module can be represented by MGKI (Multi-granularity knowledge inference). When encoding features of the corresponding dialogue audio data, this module emphasizes the impact of the target object's response text data on the final target classification label. Secondly, to better integrate prior knowledge, a Cross-Attention module is designed to facilitate the interaction between dialogue text data and prior knowledge, strengthening the representation of input information by prior knowledge. The following will combine... Figure 4 and Figure 5 To understand the classification label generation model provided in the embodiments of this application.

[0079] 103. Based on the dialogue text data, response text data, and prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data, determine the first classification result of the target object in the temporal coding dimension.

[0080] The temporal encoding dimension can be understood as an information representation dimension that incorporates temporal information. Temporal information is integrated during feature encoding to determine the result based on this information representation.

[0081] Understandably, based on the dialogue text data, response text data, and prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data, a classification label generation model is used to determine the first classification result of the target object in the temporal encoding dimension.

[0082] Specifically, based on the dialogue text data, response text data, and prior knowledge for each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data, a classification label generation model is used to determine the first classification result of the target object in the temporal encoding dimension. The dialogue text data, response text data, and prior knowledge for each scene dimension, along with the temporal information of each scene dimension in the dialogue audio data, are input into the classification label generation model for processing, thus determining the first classification result of the target object in the temporal encoding dimension.

[0083] In one embodiment, step 103 further includes: determining semantic fusion features for each scene dimension based on dialogue text data, response text data, and prior knowledge for each scene dimension; and performing temporal encoding processing on the semantic fusion features for each scene dimension based on the temporal information of each scene dimension in the dialogue audio data to determine the first classification result of the target object in the temporal encoding dimension. The semantic fusion features are obtained by semantic processing of the fusion features obtained from the dialogue text data, response text data, and prior knowledge for each scene dimension. The semantic processing can utilize a self-attention mechanism to process the fusion features, such as self-attention processing.

[0084] Furthermore, to incorporate prior knowledge of the scale, the prior knowledge is fused with the input information. Correspondingly, the step of determining the semantic fusion features for each scene dimension based on the dialogue text data, response text data, and prior knowledge for each scene dimension includes: performing feature encoding processing on the dialogue text data, response text data, and prior knowledge for each scene dimension to obtain dialogue text features, response text features, and prior knowledge features; fusing the prior knowledge features, dialogue text features, and response text features to obtain the fusion features for each scene dimension; and using an attention mechanism to perform semantic processing on the fusion features to obtain the semantic fusion features for each scene dimension.

[0085] There are various ways to fuse prior knowledge features, dialogue text features, and response text features, as long as the prior knowledge features can be integrated into the text features to strengthen the representation of prior knowledge for input information such as dialogue text data and response text data.

[0086] For example, prior knowledge features and dialogue text features under each scene dimension can be interactively processed, and the result of the interaction processing can be fused with the response text features to obtain the fused features under each scene dimension. Alternatively, prior knowledge features and response text features can be interactively processed, and the result of the interaction processing can be fused with the dialogue text features to obtain the fused features under each scene dimension. Or, prior knowledge features and dialogue text features can be interactively processed, and prior knowledge features and response text features can be interactively processed a second time, and the result of the first interaction processing and the result of the second interaction processing can be fused to obtain the fused features under each scene dimension.

[0087] In one embodiment, specifically, as Figure 6 As shown, the steps in step 103 above include the following steps.

[0088] 201. The dialogue text data, response text data and prior knowledge under each scene dimension are processed by feature encoding to obtain dialogue text features, response text features and prior knowledge features.

[0089] After data preprocessing, the dialogue text data can be represented as follows: Where i is the scene dimension index, for example, HAMD17 has a maximum of 17 scene dimensions, and j is the text index under each scene dimension, with a maximum text length of L. To enhance the target object's response, the response text data of the target object, such as the patient, is represented separately under each scene dimension. Given the dialogue video data, after data preprocessing, the corresponding response text data is represented as follows: Here, i and j are the same as i and j in the dialogue text data.

[0090] Figure 4 In the first scene dimension, Scene01_QA corresponds to the dialogue text data, and Scene01_Answer corresponds to the answer text data in the first scene dimension.

[0091] Correspondingly, Figure 5 The contextual question-and-answer format refers to dialogue text data within a specific contextual dimension. Here, [CLS] is the start identifier, and [SEP] corresponds to the separator, etc. Currently, this refers to the dialogue text data within the first contextual dimension, including:

[0092] [CLS] Q: How have you felt since last week? A: I've been feeling rather down and depressed. Q: Has something unpleasant happened? A: Probably not, it's just been a period of consistently feeling down. Q: Can your family and friends tell that you're not in a good mood? A: Yes, they can tell. [SEP]

[0093] Figure 5 The patient responses mentioned refer to the textual data of responses within a specific scenario dimension. Currently, we are referring to the textual data of responses within the first scenario dimension, including:

[0094] [CLS] It's rather low and depressed. [SEP] Maybe not, it might be a period of sustained low and depressed. [SEP] You can see that. [SEP]

[0095] In one embodiment, the step of performing feature encoding processing on the dialogue text data, response text data, and prior knowledge under each scene dimension to obtain dialogue text features, response text features, and prior knowledge features includes: performing a first feature encoding processing on the dialogue text data and response text data under each scene dimension to obtain dialogue text features and response text features; and performing a second feature encoding processing on the prior knowledge under each scene dimension to obtain prior knowledge features.

[0096] The first feature encoding process can be text encoding, such as using the BERT (Bidirectional Encoder Representations from Transformer) model to perform the first feature encoding process on the dialogue text data and response text data in each scene dimension to obtain the dialogue text features and response text features, as shown in formulas (1) and (2).

[0097] H = Sequence(BERT(X)) (1)

[0098] H A =Pool(BERT(A)) (2)

[0099] in, H A ={v i ,0<=i<17,v i ∈R d}, where H is the sequence encoding representation of the dialogue text data, i.e., the dialogue text feature, H A The sequence encoding representation of the response text data of the target object, i.e., response text features.

[0100] The second feature encoding process can be a prior knowledge mapping process. For example, it involves counting the number of options in the rating criteria for each scene dimension and selecting the maximum number as the dimension of prior knowledge for each scene dimension. For instance, for the HAMD scale, the maximum number of options across multiple scene dimensions is 5, while some scene dimensions have 3 options. Therefore, the dimension of prior knowledge for each scene dimension is determined to be 5, and the options corresponding to the scene rating options in the prior knowledge are mapped to the determined dimension of prior knowledge. Finally, the obtained prior knowledge features can be represented as follows: Where i is the scene dimension index, and there are a maximum of 17 scene dimensions in HAMD17. j is the index of multiple knowledge features in the prior knowledge features under each scene dimension, with a maximum length of 5. If a single scene dimension has less than 5 knowledge features, PAD / other identifiers are used to represent it.

[0101] Thus, we obtain the dialogue text features, response text features, and prior knowledge features for each scene dimension. It's important to note that the dialogue text data, response text data, and prior knowledge for each scene dimension can also be processed using other feature encoding methods to obtain the corresponding dialogue text features, response text features, and prior knowledge features.

[0102] After obtaining the prior knowledge features, dialogue text features, and response text features, in order to fully learn the prior knowledge features of the preset scale, the prior knowledge features, dialogue text features, and response text features are processed in a scenario-customized multi-granularity knowledge reasoning module to obtain the semantic fusion features and scenario scores for each scenario dimension. This can be represented by the following formula (3).

[0103] s i ,m i =MGKR i (h i ,v i ), 0 <= i < 17 (3)

[0104] After processing by the multi-granularity knowledge reasoning module, the output is represented as S={s i ,0<=i<17,s i ∈R 1}, M={m i ,0<=i<17,m i ∈R d}, where S is the set of scene scores for each scene dimension, s i Let M be the scene score in the i-th scene dimension, and M be the set of vector representations of semantic fusion features in each scene dimension. i This is a vector representation of the semantic fusion features for each scene dimension. Here, "scene score" has the same meaning as "scene rating." However, since the scene rating in this step is obtained through multi-granularity knowledge reasoning, it is referred to as "scene score" to distinguish it from the scene rating in the prior knowledge described above.

[0105] like Figure 4 As shown, S_scence01, S_scence02, S_scence03, ..., S_scence17 are the scene scores for the first scene dimension, the second scene dimension, the third scene dimension, ..., the 17th scene dimension, respectively, and H01, H02, H03, ..., H17 are the semantic fusion features for the first scene dimension, the second scene dimension, the third scene dimension, ..., the 17th scene dimension, respectively.

[0106] Steps 202 to 203 below further describe how the multi-granularity knowledge reasoning module obtains semantic fusion features for each scene dimension.

[0107] 202. Prior knowledge features, dialogue text features, and response text features are fused to obtain fused features for each scene dimension.

[0108] To incorporate prior knowledge features from a predefined scale, this step involves fusing these features with other dialogue and response text features. The above description outlines multiple implementation methods for fusing prior knowledge features, dialogue text features, and response text features to obtain fused features for each scene dimension. This application embodiment uses one such implementation method as an example for illustration.

[0109] Correspondingly, the above-mentioned step of fusing prior knowledge features, dialogue text features, and response text features to obtain fused features for each scene dimension includes: performing interactive processing on prior knowledge features and dialogue text features to obtain text knowledge interactive features for each scene dimension; and performing a first fusion processing on text knowledge interactive features for each scene dimension and response text features to obtain fused features for each scene dimension. In this way, prior knowledge features are used to fully represent dialogue text data, strengthening the representation of dialogue text data by prior knowledge.

[0110] Interactive processing refers to a processing method that interactively integrates prior knowledge features with dialogue text features, such as... Figure 5 As shown, interactive processing can be implemented through the Cross Attention module.

[0111] In one embodiment, such as Figure 7 As shown, the steps described above for interacting prior knowledge features with dialogue text features to obtain text knowledge interaction features for each scene dimension include the following steps. Alternatively, it can be understood that the Cross Attention module implements... Figure 7 The content shown.

[0112] 301. Determine the relationship weights between each knowledge feature in each scene dimension and each text feature in the dialogue text features.

[0113] To fully represent input information using prior knowledge features, we first need to determine the relationship weights between each knowledge feature in the prior knowledge features and each text feature in the input dialogue text features for each scene dimension. For example, there can be 5 knowledge features in the prior knowledge features for each scene dimension, and L text features in the dialogue text features, which can be understood as one text corresponding to one text feature. A non-linear function is then used to determine the relationship weights between each knowledge feature in the prior knowledge features and each text feature in the input dialogue text features for each scene dimension.

[0114] Formula (4) can be used to determine the relationship weight between each knowledge feature and text feature in each scene dimension.

[0115]

[0116] Where W1∈R d*2d b1∈R d W2∈R 1*d For learnable model parameters, Let t1 be the relationship weight or weight representation between the t1-th knowledge feature and the t2-th text feature in the i-th scene dimension, reflecting the correlation information between the t1-th knowledge feature and the t2-th text feature.

[0117] 302. Based on the relation weights, determine the weight matrix between prior knowledge features and dialogue text features in each scene dimension.

[0118] To focus on the information representation of text by prior knowledge, it is necessary to first calculate the weight matrix between prior knowledge features and input information such as dialogue text features.

[0119] For example, for each scene dimension, based on the relationship weight between the t1th knowledge feature and the t2th text feature, the sum of the relationship weights between the t1th knowledge feature and all text features in that scene dimension is determined. Based on the relationship weight between the t1th knowledge feature and the t2th text feature, and the sum of the relationship weights corresponding to the t1th knowledge feature, the weight values ​​corresponding to the t1th knowledge feature and the t2th text feature are determined. In this way, the weight values ​​of each knowledge feature and each text feature can be obtained. Based on the weight values ​​of each knowledge feature and each text feature, the weight matrix between the prior knowledge feature and the corresponding dialogue text feature in that scene dimension is determined.

[0120] In the i-th scene dimension, the weight values ​​of the t1-th knowledge feature and the t2-th text feature can be determined by formula (5).

[0121]

[0122] Wherein, the weight matrix between prior knowledge features and dialogue text features in the i-th scene dimension can be represented as:

[0123] 303. The dialogue text features are processed using a weight matrix to obtain the text knowledge interaction features that fuse prior knowledge with the dialogue text features.

[0124] The text features of the dialogue are multiplied using a weight matrix to obtain text knowledge interaction features, which incorporate prior knowledge.

[0125] For example, text knowledge interaction features can be obtained through the following formula (6).

[0126]

[0127] Where C represents the obtained text knowledge interaction features, and H represents the dialogue text features after processing by the BERT model.

[0128] After obtaining the text knowledge interaction features for each scene dimension, the target object's response text features are fused. The target object's response text features are finer-grained features, generally containing key judgment information. Therefore, the text knowledge interaction features and response text features for each scene dimension are first fused together to obtain the fused features for each scene dimension.

[0129] The first fusion process can be an additive process. First, the text knowledge interaction features and dialogue text features are processed to the same dimension. For example, if the text knowledge interaction features are 5*L dimensions and the dialogue text features are L dimensions, then the dialogue text features are copied multiple times to obtain 5*L dimensions. Then, the first text knowledge interaction feature is added to the first response text feature to obtain the first value of the fused feature. This process is repeated to obtain the fused feature for each scene dimension. Figure 5 In the diagram, the first fusion process is represented as a circle with a plus sign in the middle.

[0130] For example, the text knowledge interaction features and answer text features under each scene dimension can be first fused using the following formula (7) to obtain fused features.

[0131] G = element_size sum(C,H) A (7)

[0132] Wherein, G is the fused feature obtained after incorporating the target object's response text information.

[0133] It should be noted that the fusion feature in the embodiments of this application can also be called a multi-granularity fusion feature, which means that the fusion feature integrates information of multiple granularities, such as prior knowledge information, target object's answer text data and dialogue text data, etc.

[0134] In one embodiment, after obtaining the target classification label of the target object, a certain scientific explanation is required. However, current analogical reasoning or scale-based methods do not provide explanatory content related to the target classification label of the target object, and interpretability is often a weakness of current deep learning models. Therefore, this embodiment provides an explanation of the rationality of the target classification label of the target object based on the weight matrix.

[0135] 304. Based on the weight matrix, generate the first explanation of the target category label for the target object.

[0136] Specifically, based on the weight values ​​of the weight matrix, explanatory content for the final target classification label is further generated, explaining why the target classification label was obtained. The weight matrix represents the weight between prior knowledge features and input information such as dialogue text features. A larger weight value indicates a stronger correlation between the corresponding knowledge feature and the text feature. Therefore, knowledge features and text features with larger weight values ​​in the weight matrix are selected, and the first explanatory content for the target object's classification label is generated based on these features.

[0137] 203. An attention mechanism is used to perform semantic processing on the fused features to obtain semantic fusion features for each scene dimension.

[0138] After obtaining the fusion features for each scene dimension, the self-attention mechanism is used to perform semantic processing on the fusion features to obtain the semantic fusion features for each scene dimension. Figure 4 H01, H02, H03, ..., H17 are semantic fusion features for each scene dimension.

[0139] Correspondingly, the semantic fusion features for each scene dimension can be obtained according to formula (8).

[0140] M = Self-Attention(G) (8)

[0141] Here, M is a set of vector representations of semantic fusion features in each scene dimension. For details, please refer to the explanation above.

[0142] All of the above were learned representations within a single scene dimension. However, to understand the entire dialogue audio data, learning representations only within a single scene dimension is clearly insufficient.

[0143] In order to understand the global information in the dialogue audio data, step 204 performs temporal encoding on the semantic fusion features under each scene dimension to represent the semantic fusion features under each scene dimension in a concatenated manner.

[0144] 204. Based on the temporal information of each scene dimension in the dialogue audio data, the semantic fusion features under each scene dimension are temporally encoded to determine the first classification result of the target object in the temporal encoding dimension.

[0145] The temporal information for each scene dimension can be the chronological order of each scene dimension's appearance, or the temporal information of each scene dimension's appearance in the dialogue audio data, or other information that can characterize temporal sequence. In the embodiments of this application, one or more neural network models or deep learning models that can express temporal information can be used to perform temporal encoding processing on the semantic fusion features under each scene dimension to obtain the first classification result of the target object in the temporal encoding dimension.

[0146] In one embodiment, step 204 above includes: performing temporal concatenation encoding on the semantic fusion features under each scene dimension based on the temporal information of each scene dimension in the dialogue audio data to obtain global semantic representation information including context information; and performing semantic classification processing on the global semantic representation information to obtain the first classification result of the target object in the temporal encoding dimension.

[0147] The temporal concatenation coding process can be implemented using a bidirectional long short-term memory (BiLSTM) neural network model. In other embodiments, other neural networks that can combine morning and afternoon information can also be used.

[0148] Based on the temporal information of each scene dimension in the dialogue audio data, the semantic fusion features of each scene dimension are input into a bidirectional long short-term memory (LSTM) neural network model. A forward LSM neural network is used for forward processing to obtain a first output vector in the forward direction, and a backward LSM neural network is used for backward processing to obtain a second output vector in the backward direction. The first and second output vectors are then fused together to obtain global semantic representation information including contextual information. This second fusion process can be a concatenation process, combining the first and second output vectors to obtain the global semantic representation information.

[0149] For example, forward processing can be implemented using formula (9) to obtain the first output vector, backward processing can be implemented using formula (10) to obtain the second output vector, and second fusion processing can be implemented using formula (11) to obtain global semantic representation information.

[0150]

[0151]

[0152] M = [M F M b (11)

[0153] Among them, M F ∈R d M is the first output vector in the forward direction. b ∈R d Let M ∈ R be the second output vector in the backward direction. 1r*2d To take into account the representation information of the preceding and following context, that is, the global semantic representation information including context information, it can also be understood as the global information including the dialogue audio data.

[0154] After obtaining the global semantic representation information, semantic classification processing is performed on it to obtain the first classification result of the target object in the temporal encoding dimension. For example, after semantic processing of the global semantic representation information using a self-attention mechanism, the processing result is input into a multilayer perceptron (MLP) for perceptual processing. The perceptual processing result is then input into a softmax layer for normalization processing, thus obtaining the first classification result of the target object in the temporal encoding dimension. In other words, after the global semantic representation information passes through Self-Attention, MLP, and Softmax, the classification label and probability of the target object from a global perspective can be obtained.

[0155] For example, formula (12) can be used to represent the semantic classification process.

[0156] S_e2e_level=Softmax(MLP(Self-Attention(M))) (12)

[0157] Where S_e2e_level represents the first classification result in the end-to-end comparison. After processing with Softmax, multiple classification labels and their probabilities are obtained, denoted as follows: For example, depression levels are divided into four levels: no depression, mild depression, moderate depression, and severe depression. Therefore, there are four corresponding classification labels and their corresponding probabilities. The classification label with the highest probability is taken as the first classification result. That is, the first classification result includes the classification label with the highest probability and its corresponding probability.

[0158] The above describes the process of determining the first classification result of the target object in the temporal coding dimension.

[0159] 104. Based on the dialogue text data, response text data and prior knowledge in each scenario dimension, determine the second classification result of the target object in the preset scale dimension.

[0160] Among them, the preset scale dimension can be understood as the statistical processing dimension of multiple scenario dimensions based on the preset scale, and the result is determined from multiple scenario dimensions of the preset scale based on statistical methods (statistical processing).

[0161] Understandably, based on the dialogue text data, response text data, and prior knowledge in each scenario dimension, a classification label generation model is used to determine the second classification result of the target object in the preset scale dimension.

[0162] Specifically, based on the dialogue text data, response text data, and prior knowledge for each scenario dimension, a classification label generation model is used to determine the second classification result of the target object on a preset scale dimension. In other words, the dialogue text data, response text data, and prior knowledge for each scenario dimension are input into the classification label generation model, which then processes the data to determine the second classification result of the target object on the preset scale dimension.

[0163] In one embodiment, step 104 further includes: determining a scene score for each scene dimension based on the dialogue text data, response text data, and prior knowledge for each scene dimension; and determining the first classification result of the target object on a preset scale dimension based on the scene score for each scene dimension. The scene score for each scene dimension can be 0, 1, 2, 3, 4, etc. The scene score for each scene dimension is calculated and differs from the scene rating in the prior knowledge mentioned above.

[0164] Furthermore, the step of determining the scene score for each scene dimension based on the dialogue text data, response text data, and prior knowledge for each scene dimension includes: performing feature encoding processing on the dialogue text data, response text data, and prior knowledge for each scene dimension to obtain dialogue text features, response text features, and prior knowledge features; fusing the prior knowledge features, dialogue text features, and response text features to obtain fused features for each scene dimension; and normalizing the fused features to obtain the scene score for each scene dimension.

[0165] In one embodiment, specifically, as Figure 8 As shown, the steps in step 104 above include the following steps.

[0166] 401. The dialogue text data, response text data and prior knowledge under each scene dimension are processed by feature encoding to obtain dialogue text features, response text features and prior knowledge features.

[0167] 402. Prior knowledge features, dialogue text features, and response text features are fused to obtain fused features for each scene dimension.

[0168] Please refer to the corresponding descriptions in the text above for steps 401 and 402, which will not be repeated here.

[0169] 403. Normalize the fused features to obtain the scene score for each scene dimension.

[0170] The normalization process can be achieved through Softmax processing or other processes that can achieve similar functionality. The fused features are normalized to obtain the scene score and probability value for each scene dimension. For example, formula (13) can be used to obtain the scene score and probability value for each scene dimension. For each scene dimension, there are 5 scene scores and 5 corresponding probability values. The scene score with the highest probability value is taken as the final scene score for that scene dimension.

[0171] S = softmax(G) (13)

[0172] Please refer to the corresponding sections above for the meanings of S and G.

[0173] like Figure 4 As shown, S_scence01, S_scence02, S_scence03, ..., S_scence17 are the scene scores for the first scene dimension, the second scene dimension, the third scene dimension, ..., the 17th scene dimension, respectively.

[0174] 404. Based on the scene score under each scene dimension, determine the first classification result of the target object on the preset scale dimension.

[0175] After obtaining the scene score and probability value for each scene dimension, the scene scores for each scene dimension are summed to obtain the total score of a pre-defined scale, such as the HAMD scale. Classification labels are obtained based on the segmentation method defined by the specialist and the total score. The probability value of each scene score under each scene dimension yields the probability corresponding to the classification label; for example, summing the probability values ​​of each scene score under each scene dimension gives the corresponding probability. The score assessment results already provide the quantitative level / classification label for the pre-defined scale dimensions. For example, a total score of 20 corresponds to a quantitative level / classification label such as moderate depression. The first classification result includes the classification label, such as moderate depression, and its corresponding probability.

[0176] Table 2 shows an example of a segmentation method based on specialist physician experience for the score assessment section of the HAMD scale.

[0177] Table 2. Examples of segmentation methods for the HAMD scale.

[0178] Category Tag Level Score range Encoded ID No depression Score<=7 0 Mild depression 7<Score<=17 1 moderate depression 17<Score<=24 2 Major depression 24<Score 3

[0179] The implementation process of this step can be represented by formula (14).

[0180] S_scence_level = HAMD score evaluation (S) (14)

[0181] In one embodiment, this application provides an explanation of the reasonableness of the target classification label for the target object based on the scene score. Further, it includes the following steps.

[0182] 405. Based on the scene score under each scene dimension, generate the second explanation of the target object's target classification label.

[0183] Among them, the target category label of the target object is closely related to the scene score under each scene dimension. For example, if the target category label is severe depression, then there must be several scene dimensions with high scene scores. The scene dimension with a high scene score, such as greater than 3, is determined as the target dimension. Therefore, the second interpretation content of the target category label of the target object is generated based on the target dimension.

[0184] It's important to note that the first explanatory content for generating the target category label based on the weight matrix, and the second explanatory content for generating the target category label based on the scene score under each scene dimension, can be steps performed after obtaining the target category label. After obtaining the first and second explanatory contents, the final explanatory content is generated based on them, to conform to reading habits.

[0185] In practice, steps 103 and 104 can be performed in accordance with the execution method of the classification label generation model shown in the attached figure. That is, there is no clear distinction between steps 103 and 104. In the overall execution process, the fusion features under each scene dimension are obtained first, and then subsequent processing is performed according to the fusion features to obtain the first classification result of the target object in the temporal coding dimension and the second classification result of the target object in the preset scale dimension.

[0186] 105. Based on the first classification result and the second classification result, generate the target classification label for the target object.

[0187] Among them, such as Figure 4 As shown, the Decision_Level Fusion module is used to fuse the first and second classification results to generate the target classification label for the target object. The Decision_Level Fusion module employs three strategies to fuse the first and second classification results: using the first classification result as the final classification result (e.g., taking S_e2e_level as the final classification result); using the second classification result as the final classification result (e.g., taking S_scence_level as the final classification result); and weighting the probabilities of the classification labels in the second classification result and adding them to the corresponding global probabilities in the first classification result, then reordering the probabilities and outputting the quantitative level / classification label with the highest probability as the final target classification label.

[0188] In the above embodiments, when generating target classification labels for target objects, not only the first classification in the time encoding dimension is considered, but also the second classification result in the preset scale dimension, which improves the accuracy of generating target classification labels for target objects. Moreover, both the first classification result in the time encoding dimension and the second classification result in the preset scale dimension consider prior knowledge in the preset scale, integrating prior knowledge into the process of determining the first and second classification results, thereby improving the accuracy of the first classification result in the time encoding dimension and the second classification result in the preset scale dimension, and further improving the accuracy of generating target classification labels for target objects. In addition, when generating the first classification result in the time encoding dimension, the temporal information of each scene dimension in the dialogue audio data is considered, that is, the contextual information of the dialogue audio data is considered, and the global information in the entire dialogue audio data is integrated, improving the accuracy of the first classification result, and further improving the accuracy of generating target classification labels for target objects. Furthermore, in this embodiment, the explanatory content of the target classification labels can be further generated based on the weight matrix and scene score, improving the interpretability of the target classification labels.

[0189] The classification label generation model in this application embodiment is based on a multi-granularity knowledge reasoning module, which can effectively encode and understand dialogue text data, such as doctor-patient dialogue in the field of depression. At the same time, it integrates prior knowledge and fine-grained response text data modeling to enhance the expressive power of the classification label generation model. Through two different classification methods, namely time-series encoding dimension and preset scale dimension, it effectively improves the classification label generation model's ability to understand global and local information.

[0190] Figure 9 This is another flowchart illustrating the label generation method provided in this application embodiment. The label generation method mainly describes the process of generating a classification label generation model, specifically including the following steps.

[0191] 501. Obtain the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, as well as obtain the training preset scale of the training target object. This training preset scale includes training prior knowledge for each scene dimension in multiple scene dimensions, which is predetermined based on the training audio data. Among them, the first label result is the pre-determined classification result in the temporal coding dimension.

[0192] 502. The training dialogue audio data is processed to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension.

[0193] 503, Obtain the initial classification label generation model.

[0194] 504. Based on the training dialogue text data, training response text data, and training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data, the model is generated using the initial classification labels to determine the first classification result of the training target object in the temporal encoding dimension.

[0195] 505. Based on the training dialogue text data, training response text data and training prior knowledge in each scene dimension, the model is generated using the initial classification labels to determine the training scene score of the training target object in each scene dimension.

[0196] Steps 501 to 505 above are consistent with the process of generating a model using classification labels described earlier, except that the word "training" is added to the name for distinction. The implementation process is identical; please refer to the descriptions of the corresponding steps above for details, which will not be repeated here. It should be noted that in step 505, during the training phase, it is only necessary to obtain the training scene score of the training target object in each scene dimension.

[0197] It should be noted that if you need to further extend steps 501 to 505 above, you can refer to the process of generating a model using classification labels above. The difference is that you need to add the word "training" to some of the names in the further extension.

[0198] 506. The overall loss value is determined based on the training first classification result, the first label result, the training scenario score, and the training prior knowledge.

[0199] In this embodiment, the classification label generation model is a multi-task model, which can obtain two different results in two different dimensions, such as the temporal encoding dimension and the preset scale dimension. Therefore, a multi-task joint training method is used during training to obtain the classification label generation model after training.

[0200] In one embodiment, step 506 above includes: determining a first loss value based on the training first classification result and the first label result; determining a second loss value based on the training scene score and training prior knowledge under each scene dimension; and determining an overall loss value based on the first loss value and the second loss value.

[0201] Considering the imbalance in the difficulty and quantity of samples in the field of depression, a joint loss function is adopted for optimization, using a sample balancing loss function such as Focal Loss and a gradient balancing loss function such as GHM Loss. The Focal Loss function reduces the weight of easily classified samples (i.e., samples with high confidence) and increases the weight of difficult samples, forcing the model to focus primarily on those difficult samples. In the medical field, especially in psychiatric consultation scenarios, due to the lack of information, there are many difficult samples, and numerous cases of mislabeling and confusion exist. In such cases, the Focal Loss function would affect the stability and optimization direction of the model. Therefore, the GHM Loss function is introduced to balance the model's attention to difficult samples. The GHM Loss function dynamically balances the weight of difficult samples; samples that are too difficult will not receive more attention because they are often mislabeled or incorrectly labeled. In summary, the final loss value is obtained by weighting and balancing the loss values ​​calculated by the Focal Loss function and the GHM Loss function.

[0202] For example, the loss value can be determined using formula (15).

[0203] loss = L FL +γ*L GHM (15)

[0204] Where γ is a hyperparameter used to balance L GHM The proportion, loss is the loss value, L FL The loss value L is calculated using the Focal Loss function. GHM The loss value is calculated using the GHM Loss function.

[0205] As mentioned above, the embodiments of this application involve multi-task training, and therefore involve loss values ​​for multiple tasks, such as a first loss value and a second loss value.

[0206] The step of determining the first loss value based on the training results of the first classification and the first label includes: calculating the first loss value corresponding to the first task using the method corresponding to formula (15) based on the training results of the first classification and the second label. For example, the calculated first loss value can be expressed as loss. e2e To express.

[0207] The step of determining the second loss value based on the training scene score and prior knowledge for each scene dimension includes: determining the scene loss value for each scene dimension based on the training scene score and prior knowledge for the corresponding scene dimension; and determining the second loss value based on the scene loss value for each scene dimension. For example, based on the training scene score and prior knowledge for each scene dimension, the scene loss value for each scene dimension is calculated using the method corresponding to formula (15), and the sum of the scene loss values ​​for each scene dimension is determined as the second loss value corresponding to the second task. For example, the calculated second loss value can be used as a loss function. scence To express.

[0208] The step of determining the overall loss value based on the first loss value and the second loss value includes: weighted summing of the first loss value and the second loss value to obtain the overall loss value. This overall loss value is the final loss value of the initial classification label generation model.

[0209] For example, the loss value can be determined using formula (16).

[0210] Loss = α * loss scence +β*loss e2e (16)

[0211] Here, Loss represents the overall loss value, and α and β are hyperparameters used to balance the weight ratios of the first and second tasks.

[0212] 507. Update the training parameters in the initial classification label generation model based on the overall loss value until the overall loss value meets a preset condition to obtain the trained classification label generation model. The preset condition can be a condition such as the overall loss value converging.

[0213] The trained classification label generation model is the same as the classification label generation model described in the above embodiment.

[0214] In one scenario, after obtaining the classification label generation model, the model is used to process the dialogue text data, the response text data, and the preset scale corresponding to the target object in each of the multiple scene dimensions involved in the target object, to generate the target classification label for the target object.

[0215] In one scenario, after obtaining the classification label generation model, it is also possible to acquire multiple scene dimensions involved in the dialogue audio data corresponding to the target object, dialogue text data under each scene dimension, response text data under each scene dimension, and a preset scale corresponding to the target object. The preset scale includes prior knowledge under each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data. The classification label generation model is then used to process the response text data, dialogue text data, and prior knowledge under each scene dimension to generate the target classification label for the target object.

[0216] It is important to note that the classification label generation model in the above embodiments does not include a data preprocessing process for the dialogue audio data of the target object. In some embodiments, however, the data preprocessing process for the dialogue audio data of the target object can be integrated into the classification label generation model. Correspondingly, when using the classification label generation model, after obtaining the dialogue audio data of the target object, the dialogue audio data is input into the classification label generation model for data preprocessing. This preprocessing yields multiple scene dimensions involved in the dialogue audio data, dialogue text data for each scene dimension, and response text data for each scene dimension, before further processing. During the training phase, after obtaining the classification label generation model, it is used to process the dialogue audio data of the target object and a preset scale to obtain the target classification label for the target object.

[0217] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0218] To facilitate better implementation of the label generation method of this application, this application also provides a label generation apparatus, which corresponds to the label generation method using a classification label generation model. Please refer to... Figure 10 , Figure 10 This is a schematic diagram of the label generation device provided in an embodiment of this application. The label generation device may include an acquisition module 601, an identification module 602, a first determination module 603, a second determination module 604, and a generation module 605.

[0219] The acquisition module 601 is used to acquire the dialogue audio data corresponding to the target object.

[0220] The recognition module 602 is used to perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

[0221] The acquisition module 601 is further configured to acquire a preset scale corresponding to the target object, wherein the preset scale includes prior knowledge for each scene dimension, and the prior knowledge is predetermined based on the dialogue audio data.

[0222] The first determining module 603 is used to determine the first classification result of the target object in the temporal coding dimension based on the dialogue text data, the response text data and the prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the dialogue audio data.

[0223] The second determining module 604 is used to determine the second classification result of the target object on the preset scale dimension based on the dialogue text data, the response text data and the prior knowledge in each scene dimension.

[0224] The generation module 605 is used to generate a target classification label for the target object based on the first classification result and the second classification result.

[0225] This application also provides a label generation apparatus, which corresponds to a label generation method based on a trained classification label generation model. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of the tag generation device provided in an embodiment of this application. The tag generation device may include a training acquisition module 701, a training recognition module 702, a first training determination module 703, a second training determination module 704, a loss value determination module 705, and an update module 706.

[0226] The training acquisition module 701 is used to acquire the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, and to acquire the training preset scale of the training target object. The training preset scale includes training prior knowledge under each of the multiple scene dimensions. The training prior knowledge is predetermined based on the training audio data.

[0227] The training recognition module 702 is used to perform recognition processing on the training dialogue audio data to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension.

[0228] The training module 701 is used to obtain the initial classification label generation model.

[0229] The first training determination module 703 is used to determine the first classification result of the target object in the temporal encoding dimension based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data, using the initial classification label generation model.

[0230] The second training determination module 704 is used to determine the training scene score of the training target object in each scene dimension by using the initial classification label generation model based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension.

[0231] The loss value determination module 705 is used to determine the overall loss value based on the training first classification result, the first label result, the training scene score, and the training prior knowledge.

[0232] The update module 706 is used to update the training parameters in the initial classification label generation model according to the overall loss value until the overall loss value meets the preset conditions, so as to obtain the trained classification label generation model.

[0233] In one embodiment, the tag generation device may further include a processing module, which is used to process the dialogue text data, the response text data and the preset scale corresponding to the target object in each of the multiple scene dimensions involved in the target object using the classification tag generation model, so as to generate the target classification tag of the target object.

[0234] In one embodiment, the training acquisition module is further configured to acquire multiple scene dimensions involved in the dialogue audio data corresponding to the target object, dialogue text data under each scene dimension, response text data under each scene dimension, and a preset scale corresponding to the target object. The preset scale includes prior knowledge under each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data. The processing module is configured to use the classification label generation model to process the response text data, dialogue text data, and prior knowledge under each scene dimension to generate target classification labels for the target object.

[0235] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0236] Accordingly, embodiments of this application also provide a computer device, which can be a terminal or a server. For example... Figure 12 As shown, Figure 12This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 800 includes a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, and a computer program stored on the memory 802 and executable on the processor. The processor 801 and the memory 802 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0237] The processor 801 is the control center of the computer device 800. It connects various parts of the computer device 800 through various interfaces and lines. By running or loading software programs (computer programs) and / or modules stored in the memory 802, and calling data stored in the memory 802, it performs various functions of the computer device 800 and processes data, thereby monitoring the computer device 800 as a whole.

[0238] In this embodiment, the processor 801 in the computer device 800 loads the instructions corresponding to one or more application processes / computer programs into the memory 802 according to the following steps, and the processor 801 runs the application / computer program s stored in the memory 802 to realize various functions:

[0239] Acquire dialogue audio data corresponding to the target object, and perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension; acquire a preset scale corresponding to the target object, the preset scale including prior knowledge under each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data; determine a first classification result of the target object in the temporal encoding dimension based on the dialogue text data, response text data, and prior knowledge under each scene dimension, as well as the temporal sequence information of each scene dimension in the dialogue audio data; determine a second classification result of the target object in the preset scale dimension based on the dialogue text data, response text data, and prior knowledge under each scene dimension; generate a target classification label for the target object based on the first classification result and the second classification result; or...

[0240] The process involves: acquiring training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal encoding dimension; acquiring a training preset scale for the training target object, wherein the training preset scale includes training prior knowledge for each scene dimension, and the training prior knowledge is predetermined based on the training audio data; performing recognition processing on the training dialogue audio data to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data for each scene dimension, and training response text data for each scene dimension; acquiring an initial classification label generation model; and based on the training dialogue text data, the training response text data, and the training prior knowledge for each scene dimension, and the training target object's training dialogue audio data, training response text data, and training prior knowledge, the training target object's training dialogue audio data is processed. The training dialog audio data is used to generate a first classification result for the training target object in the temporal encoding dimension by using the initial classification label generation model to determine the temporal encoding dimension of the temporal information. Based on the training dialog text data, the training response text data, and the training prior knowledge in each scene dimension, the initial classification label generation model is used to determine the training scene score for the training target object in each scene dimension. The overall loss value is determined based on the first classification result, the first label result, the training scene score, and the training prior knowledge. The training parameters in the initial classification label generation model are updated based on the overall loss value until the overall loss value meets a preset condition, thus obtaining the trained classification label generation model.

[0241] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0242] Optional, such as Figure 12 As shown, the computer device 800 also includes: a touch screen display 803, a radio frequency circuit 804, an audio circuit 805, an input unit 806, and a power supply 807. The processor 801 is electrically connected to the touch screen display 803, the radio frequency circuit 804, the audio circuit 805, the input unit 806, and the power supply 807.

[0243] The touch display screen 803 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 803 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar devices. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the data to the processor 801 to determine the type of touch event. Subsequently, the processor 801 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and display panel can be integrated into the touch display screen 803 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 803 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 803 can also be used as part of the input unit 806 to achieve input functions.

[0244] In this embodiment, the touch display screen 803 is used to present a graphical user interface and receive operation commands generated by the user interacting with the graphical user interface.

[0245] The radio frequency circuit 804 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0246] Audio circuitry 805 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 805 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 805, converted back into audio data, and output to processor 801 for processing. The audio data is then transmitted via radio frequency circuitry 804 to, for example, another computer device, or output to memory 802 for further processing. Audio circuitry 805 may also include an earphone jack to facilitate communication between peripheral headphones and computer devices.

[0247] The input unit 806 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0248] Power supply 807 is used to supply power to various components of computer device 800. Optionally, power supply 807 can be logically connected to processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 807 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0249] although Figure 12 As not shown in the diagram, the computer device 800 may also include a camera, sensors, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0250] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0251] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0252] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute the steps in any of the tag generation methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0253] Acquire dialogue audio data corresponding to the target object, and perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension; acquire a preset scale corresponding to the target object, the preset scale including prior knowledge under each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data; determine a first classification result of the target object in the temporal encoding dimension based on the dialogue text data, response text data, and prior knowledge under each scene dimension, as well as the temporal sequence information of each scene dimension in the dialogue audio data; determine a second classification result of the target object in the preset scale dimension based on the dialogue text data, response text data, and prior knowledge under each scene dimension; generate a target classification label for the target object based on the first classification result and the second classification result; or...

[0254] The process involves: acquiring training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal encoding dimension; acquiring a training preset scale for the training target object, wherein the training preset scale includes training prior knowledge for each scene dimension, and the training prior knowledge is predetermined based on the training audio data; performing recognition processing on the training dialogue audio data to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data for each scene dimension, and training response text data for each scene dimension; acquiring an initial classification label generation model; and based on the training dialogue text data, the training response text data, and the training prior knowledge for each scene dimension, and the training target object's training dialogue audio data, training response text data, and training prior knowledge, the training target object's training dialogue audio data is processed. The training dialog audio data is used to generate a first classification result for the training target object in the temporal encoding dimension by using the initial classification label generation model to determine the temporal encoding dimension of the temporal information. Based on the training dialog text data, the training response text data, and the training prior knowledge in each scene dimension, the initial classification label generation model is used to determine the training scene score for the training target object in each scene dimension. The overall loss value is determined based on the first classification result, the first label result, the training scene score, and the training prior knowledge. The training parameters in the initial classification label generation model are updated based on the overall loss value until the overall loss value meets a preset condition, thus obtaining the trained classification label generation model.

[0255] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0256] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0257] Since the computer program stored in the storage medium can execute the steps in any of the label generation methods provided in the embodiments of this application, the beneficial effects that any of the label generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0258] The above provides a detailed description of a label generation method, apparatus, storage medium, and computer device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A label generation method, characterized in that, include: The dialogue audio data corresponding to the target object is obtained, and the dialogue audio data is processed to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension. Obtain a preset scale corresponding to the target object. The preset scale includes prior knowledge for each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data. Based on the dialogue text data, the response text data, and the prior knowledge in each scene dimension, determine the semantic fusion features in each scene dimension; Based on the temporal information of each scene dimension in the dialogue audio data, the semantic fusion features under each scene dimension are temporally encoded to determine the first classification result of the target object in the temporal encoding dimension. Based on the dialogue text data, the response text data, and the prior knowledge for each scene dimension, determine the scene score for each scene dimension; Based on the scene score under each scene dimension, determine the second classification result of the target object on the preset scale dimension; Based on the first classification result and the second classification result, a target classification label is generated for the target object.

2. The method according to claim 1, characterized in that, The step of determining the semantic fusion features for each scene dimension based on the dialogue text data, the response text data, and the prior knowledge for each scene dimension includes: The dialogue text data, the response text data, and the prior knowledge under each scene dimension are respectively subjected to feature encoding processing to obtain dialogue text features, response text features, and prior knowledge features; The prior knowledge features, the dialogue text features, and the response text features are fused together to obtain fused features for each scene dimension; The fused features are semantically processed using an attention mechanism to obtain semantic fusion features for each scene dimension.

3. The method according to claim 2, characterized in that, The step of fusing the prior knowledge features, the dialogue text features, and the response text features to obtain fused features for each scene dimension includes: The prior knowledge features and the dialogue text features are interactively processed to obtain text knowledge interaction features for each scene dimension. The text knowledge interaction features and the answer text features under each scene dimension are subjected to a first fusion process to obtain the fused features under each scene dimension.

4. The method according to claim 3, characterized in that, The prior knowledge features for each scene dimension include multiple knowledge features. The step of interacting with the prior knowledge features and the dialogue text features to obtain the text knowledge interaction features for each scene dimension includes: Determine the relationship weights between each knowledge feature in each scene dimension and each text feature in the dialogue text features; Based on the relation weights, determine the weight matrix between the prior knowledge features and the dialogue text features in each scene dimension; The dialogue text features are processed using the weight matrix to obtain text knowledge interaction features that fuse the dialogue text features with the prior knowledge.

5. The method according to claim 1, characterized in that, The step of performing temporal encoding processing on the semantic fusion features under each scene dimension based on the temporal information of each scene dimension in the dialogue audio data, to determine the first classification result of the target object in the temporal encoding dimension, includes: Based on the temporal information of each scene dimension in the dialogue audio data, the semantic fusion features under each scene dimension are subjected to temporal concatenation encoding to obtain global semantic representation information including context information. The global semantic representation information is subjected to semantic classification processing to obtain the first classification result of the target object in the temporal coding dimension.

6. The method according to claim 1, characterized in that, The step of determining the scene score for each scene dimension based on the dialogue text data, the response text data, and the prior knowledge for each scene dimension includes: The dialogue text data, the response text data, and the prior knowledge under each scene dimension are respectively subjected to feature encoding processing to obtain dialogue text features, response text features, and prior knowledge features; The prior knowledge features, the dialogue text features, and the response text features are fused together to obtain fused features for each scene dimension; The fused features are normalized to obtain the scene score for each scene dimension.

7. The method according to claim 1, characterized in that, The step of recognizing and processing the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension includes: The dialogue audio data is subjected to role separation and role classification processing to obtain at least two different role objects, among which the target object is included; The dialogue audio data is processed by speech recognition to obtain dialogue text data corresponding to at least two different role objects and the response text data of the target object; Scene dimension recognition is performed on the dialogue text data corresponding to at least two different role objects to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension.

8. A label generation method, characterized in that, include: Obtain the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, and obtain the training preset scale of the training target object. The training preset scale includes training prior knowledge under each scene dimension in multiple scene dimensions. The training prior knowledge is predetermined based on the training audio data. The training dialogue audio data is processed to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension. Obtain the initial classification label generation model; Based on the training dialogue text data, the training response text data, and the training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data, the initial classification label generation model is used to determine the first classification result of the training target object in the temporal encoding dimension. Based on the training dialogue text data, the training response text data, and the training prior knowledge in each scene dimension, the initial classification label generation model is used to determine the training scene score of the training target object in each scene dimension; The overall loss value is determined based on the training first classification result, the first label result, the training scene score, and the training prior knowledge; The training parameters in the initial classification label generation model are updated according to the overall loss value until the overall loss value meets the preset conditions to obtain the trained classification label generation model.

9. The method according to claim 8, characterized in that, The step of determining the overall loss value based on the training first classification result, the first label result, the training scene score, and the training prior knowledge includes: A first loss value is determined based on the training first classification result and the first label result; The second loss value is determined based on the training scenario score and the training prior knowledge for each scenario dimension. The total loss value is determined based on the first loss value and the second loss value.

10. The method according to claim 8, characterized in that, The step of determining the second loss value based on the training scene score and the training prior knowledge for each scene dimension includes: The scene loss value for each scene dimension is determined based on the training scene score for each scene dimension and the training prior knowledge for the corresponding scene dimension. The second loss value is determined based on the scene loss value under each scene dimension.

11. A label generating apparatus, characterized in that, include: The acquisition module is used to acquire the dialogue audio data corresponding to the target object; The recognition module is used to perform recognition processing on the dialogue audio data to obtain multiple scene dimensions involved in the dialogue audio data, dialogue text data under each scene dimension, and response text data under each scene dimension. The acquisition module is also used to acquire a preset scale corresponding to the target object. The preset scale includes prior knowledge for each scene dimension, wherein the prior knowledge is predetermined based on the dialogue audio data. The first determining module is used to determine the semantic fusion features of each scene dimension based on the dialogue text data, the response text data and the prior knowledge of each scene dimension; and to perform temporal encoding processing on the semantic fusion features of each scene dimension based on the temporal information of each scene dimension in the dialogue audio data, so as to determine the first classification result of the target object in the temporal encoding dimension. The second determining module is used to determine the scene score for each scene dimension based on the dialogue text data, the response text data, and the prior knowledge for each scene dimension; and to determine the second classification result of the target object on the preset scale dimension based on the scene score for each scene dimension. The generation module is used to generate target classification labels for the target object based on the first classification result and the second classification result.

12. A label generating apparatus, characterized in that, include: The training acquisition module is used to acquire the training dialogue audio data corresponding to the training target object and the first label result of the training target object in the temporal coding dimension, and to acquire the training preset scale of the training target object. The training preset scale includes training prior knowledge under each scene dimension in multiple scene dimensions. The training prior knowledge is predetermined based on the training audio data. The training recognition module is used to recognize and process the training dialogue audio data to obtain multiple scene dimensions involved in the training dialogue audio data, training dialogue text data under each scene dimension, and training response text data under each scene dimension. The training acquisition module is used to acquire the initial classification label generation model; The first training determination module is used to determine the first classification result of the target object in the temporal encoding dimension by using the initial classification label generation model based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension, as well as the temporal information of each scene dimension in the training dialogue audio data; The second training determination module is used to determine the training scene score of the training target object in each scene dimension based on the training dialogue text data, the training response text data and the training prior knowledge in each scene dimension, using the initial classification label generation model. The loss value determination module is used to determine the overall loss value based on the training first classification result, the first label result, the training scene score, and the training prior knowledge. An update module is used to update the training parameters in the initial classification label generation model according to the overall loss value until the overall loss value meets a preset condition, so as to obtain the trained classification label generation model.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the tag generation method as described in any one of claims 1-10.

14. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the steps of the label generation method as described in any one of claims 1-10 by calling the computer program stored in the memory.

Images