Group chat-oriented dialogue structure analysis model training method, analysis method and device

Abstract

The application provides a group chat-oriented dialogue structure analysis model training method, analysis method and device. After obtaining the text representation of each message by using a BERT model, the attention representation of a target message speaker is mined by combining the defined relationship encoding between each speaker through a GRU. According to the reply relationship of the previous message, the corresponding thread is constructed, the target message is connected to each thread, and the content semantic representation of each thread is obtained by inputting the GRU. The discourse structure representation of the target message and the previous message is obtained by a multilayer perceptron. The true reply object of the target message is jointly evaluated, analyzed and judged by combining the attention representation, the content semantic representation and the discourse structure representation, and the accuracy of dialogue structure analysis is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of information communication technology, and in particular to a conversation structure analysis model training method, analysis method and device for group chat. Background Art

[0002] As instant messaging apps like QQ, WeChat, and DingTalk become essential communication tools, group chat, as a primary function, is increasingly used. Analyzing the content of massive group chat messages and capturing meaningful information is crucial. For example, analyzing discussions of hot social topics within group chats can help prevent emergencies. Identifying illegal activities such as premeditated crimes, pornography, gambling, and drug trafficking can help alert the police and mitigate social harm. Deepening group chat understanding requires analyzing the conversation structure. This process transforms disorganized, time-based streaming data into structured data with reply relationships, facilitating further analysis. However, the prevalence of short messages, the use of pronouns, and non-standardized language present significant challenges for this technology.

[0003] Traditional statistical learning-based research mainly relies on constructing language features. By extracting specific features and combining them with the representation of words in the message as the message representation, a simple feedforward neural network is used to capture the reply relationship between message pairs. However, this method does not consider the contextual environment. Existing dialogue structure discovery technologies mostly focus only on the semantics of the message text content level, ignoring the user-level semantic information in group chat scenarios. Summary of the Invention

[0004] In view of this, the embodiments of the present invention provide a conversation structure analysis model training method, analysis method and device for group chat to eliminate or improve one or more defects in the prior art and solve the problem of analyzing and identifying the real reply objects of the target message in group chat information.

[0005] The technical solutions of the present invention are as follows:

[0006] In one aspect, the present invention provides a method for training a conversation structure analysis model for group chat, comprising:

[0007] Obtaining a training sample set, the training sample set including multiple samples, each sample including a target message to be analyzed and multiple previous messages of the target message, and marking the target message and each previous message's true reply object message;

[0008] Obtaining an initial network model, wherein the processing of the initial network model includes:

[0009] Using the BERT (Bidirectional Encoder Representation from Transformers) model to initialize the text representation of each message in the sample;

[0010] The target message and a set number of preceding messages are used as a candidate set, a relationship is defined between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code, and the text representation of each message in the sample is connected with the corresponding relationship code to obtain an updated representation; the updated representation of each message in the sample is sequentially input into a gated recurrent unit, and the finally obtained hidden layer representation is used as the attention representation of the speaker of the target message, and the attention of the speaker of the target message to each message in the sample is calculated based on the attention representation;

[0011] Constructing multiple threads according to the reply relationship of each previous message in the sample, and connecting the target message to each thread to obtain a spliced ​​thread, and using the finally obtained hidden layer representation as the content semantic representation of the corresponding spliced ​​thread;

[0012] After respectively concatenating the text representation of the target message with the text representation of the previous message in the candidate set, the resultant text is input into a first multi-layer perceptron to obtain a discourse structure representation;

[0013] Concatenating the attention level, content semantic representation, and discourse structure representation corresponding to each preceding message in the sample and inputting them into a second multi-layer perceptron, normalizing the output results to obtain a matching score corresponding to each preceding message in the sample, and outputting a predicted reply object for the target message based on the matching score of each preceding message in the sample;

[0014] The initial network model is trained using the training sample set to obtain a target dialogue structure analysis model.

[0015] In some embodiments, the BERT model is pre-trained using a specified dataset.

[0016] In some embodiments, defining a relationship between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code includes:

[0017] For the i-th message in the candidate set, if the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,0];

[0018] If the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,1];

[0019] If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,0];

[0020] If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,1];

[0021] If the i-th message is the target message, the relationship between the i-th message and the target message is defined as [1, -1].

[0022] In some embodiments, the updated representation of each message in the sample is sequentially input into a gated recurrent unit, and the output hidden layer is used as the attention representation of the speaker of the target message. The attention degree of the speaker of the target message to each message in the sample is calculated based on the attention representation, including:

[0023] Denote the update of the i-th message as u′ i Input gated recurrent unit, the expression is:

[0024] u′ i =concat(u i ,r i );

[0025]

[0026] Among them, u i represents the text representation of the i-th message, r i Indicates the relationship code between the i-th message and the target message; h i-1 represents the hidden layer output of the i-1th node, h i represents the hidden layer output of the i-th node;

[0027] Calculate the similarity e between the i-th message and the target message i , the calculation formula is:

[0028]

[0029] in, represents the transpose of the text representation of the i-th message, W represents the weight coefficient, h t Represents the final output of the gated recurrent unit;

[0030] The similarity e i Perform normalization to get the target message speaker's attention to the i-th message attn i , the calculation formula is:

[0031] attn i =softmax(e i ).

[0032] In some embodiments, in the sample, the attention level, content semantic representation, and discourse structure representation corresponding to each above message in the sample are concatenated and input into a second multi-layer perceptron, and the output results are normalized to obtain a matching score corresponding to each above message in the sample, which is expressed as:

[0033] s i =softmax(MLP(concat(tho i ,disc i ,attn i )));

[0034] Among them, s i Represents the matching score, softmax is the normalized exponential function, MLP represents the operation of the second multi-layer perceptron, concat represents the connection operation, tho i Indicates the semantic representation of the content corresponding to the i-th message, disc i Represents the discourse structure representation corresponding to the i-th message, attn i Indicates the attention level corresponding to the i-th message.

[0035] In some embodiments, the initial network model is trained using the training sample set, including: training using a binary cross entropy loss function, where for the i-th message, the loss function is:

[0036]

[0037] Among them, s i Represents the matching score, when the i-th message is the parent message of the target message y i =1, when the i-th message is not the parent message of the target message y i =0.

[0038] In some embodiments, in the sample, if the target message is the beginning of a conversation, the real reply object of the target message is marked as the target message itself.

[0039] On the other hand, the present invention also provides a method for analyzing conversation structure in group chat, comprising:

[0040] Obtaining a target message to be analyzed and multiple group chat messages above the target message;

[0041] Each group chat message and the target message are sequentially input into the above-mentioned group chat-oriented dialogue structure analysis model training method, and the object message to which the target message actually replies is output.

[0042] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0043] On the other hand, a computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0044] The beneficial effects of the present invention are at least:

[0045] In the group chat-oriented conversation structure analysis model training method, analysis method, and device described in the present invention, after using the BERT model to obtain the textual representation of each message, the attention representation of the speaker of the target message is mined using a GRU, combined with the defined relationship encoding between speakers. Corresponding threads are constructed based on the reply relationships of the previous message, and the target message is connected to each thread and input into the GRU to obtain the content semantic representation of each thread. Finally, a multi-layer perceptron is used to obtain the discourse structure representation of the target message and the previous message. This combined attention representation, content semantic representation, and discourse structure representation are used to jointly evaluate and analyze the target message's true reply recipient, significantly improving the accuracy of conversation structure analysis.

[0046] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will in part become apparent to those skilled in the art upon examination of the following or may be learned by practice of the present invention. The objects and other advantages of the present invention may be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as in the accompanying drawings.

[0047] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0049] Figure 1 This is a logical diagram of the initial network model in the conversation structure analysis model training method for group chat described in one embodiment of the present invention.

[0050] Figure 2 This is an example of a sample in the conversation structure analysis model training method for group chat described in an embodiment of the present invention.

[0051] Figure 3 The figure is a schematic diagram of the structure of the initial network model in the conversation structure analysis model training method for group chat according to one embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0053] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0054] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0055] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0056] To effectively analyze the conversation structure in group chats and quickly and accurately identify the true recipients of responses to target messages, this paper provides a conversation structure analysis model training method, analysis method, and device for group chats. This method fully considers the semantics of user attention, content, and discourse structure in group chats, capturing and integrating these three semantics through algorithms such as gated recurrent units (GRUs) and multi-layer perceptrons (MLPs).

[0057] It should be noted in advance that the speaker of the message refers to the user who sends the corresponding message, and the reply recipient refers to a specific message mentioned above.

[0058] Specifically, the present invention provides a method for training a conversation structure analysis model for group chats, such as Figure 1 As shown, it includes steps S101 to S103:

[0059] Step S101: obtaining a training sample set, which includes multiple samples. Each sample contains a target message to be analyzed and multiple previous messages of the target message, and marks the target message and the object messages of each previous message that are actually replied to.

[0060] Step S102: Obtain the initial network model, such as Figure 3 As shown, the processing of the initial network model includes:

[0061] Step S1021: Use the BERT model to initialize the text representation of each message in the sample.

[0062] Step S1022: Take the target message and a set number of previous messages as a candidate set, define the relationship between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code, connect the text representation of each message in the sample with the corresponding relationship code to obtain an updated representation; input the updated representation of each message in the sample into the gated recurrent unit in turn, use the finally obtained hidden layer representation as the attention representation of the speaker of the target message, and calculate the attention of the speaker of the target message to each message in the sample based on the attention representation.

[0063] Step S1023: construct multiple threads according to the reply relationship of each previous message in the sample, and connect the target message to each thread to obtain a spliced ​​thread, and use the final hidden layer representation as the content semantic representation of the corresponding spliced ​​thread.

[0064] Step S1024: After connecting the text representation of the target message with the text representation of the previous message in the candidate set, the text representation is input into the first multi-layer perceptron to obtain the discourse structure representation.

[0065] Step S1025: The attention degree, content semantic representation and discourse structure representation corresponding to each previous message in the sample are connected and input into the second multi-layer perceptron, the output results are normalized to obtain the matching score corresponding to each previous message in the sample, and the predicted reply object of the target message is output according to the matching score of each previous message in the sample.

[0066] Step S103: Using the training sample set to train the initial network model to obtain the target dialogue structure analysis model.

[0067] In step S101, a training sample set is constructed based on an existing conversation database. The conversation database can include different topics. In some usage scenarios, a training sample set can also be constructed based on corpus data of specific topics, depending on the different recognition requirements. The constructed sample includes the target message to be analyzed and the previous messages of the target message. By marking the actual reply object of each message as a label, the structural relationship of the messages in a sample can be indicated. It should be noted that the target message is the analysis object in a sample, and the previous messages refer to the multiple consecutive messages preceding the target message.

[0068] In some embodiments, if the target message is the starter of a conversation, the target message's actual reply object is marked as the target message itself. Furthermore, the target message's actual reply object can be referred to as the target message's parent message. If the target message is the starter of a group conversation, the target message itself is considered the target message's parent message.

[0069] In step S102, an initial network model is constructed to realize the function of searching for the parent message of the target message.

[0070] To find the parent message of the target message, the BERT model is first used in step S1021 to initialize the text representation of each message in the sample to generate a deep bidirectional language representation, converting the text message into an expression form that can be recognized and processed by subsequent models. In some embodiments, the BERT model can be pre-trained using a specified dataset, for example, the BERT-base uncased pre-trained model provided by BERT can be used.

[0071] In step S1022, a candidate set is first constructed to analyze and evaluate the target message speaker's attentional representation. A time window parameter, K, is set, and the candidate set includes the K-1 preceding messages preceding the target message and the target message. Based on the relationship between the speakers of each message in the candidate set, a relationship code is defined between each preceding message and the target message. This is then used to capture the information the target message speaker focuses on.

[0072] In some embodiments, in step S1022, a relationship is defined between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code, including:

[0073] 1) For the i-th message in the candidate set, if the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,0].

[0074] 2) If the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,1].

[0075] 3) If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,0].

[0076] 4) If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,1].

[0077] 5) If the i-th message is the target message, the relationship between the i-th message and the target message is defined as [1, -1].

[0078] After defining the relationship encoding between the speaker of each preceding message and the speaker of the target message, the text representation of each message in the sample is concatenated with the corresponding relationship encoding to obtain an updated representation. This representation is then fed into a gated recurrent unit (GRU) for processing, with the output of the last hidden layer node serving as the target message's attention representation. Furthermore, the target message's attention to each message in the candidate set is calculated.

[0079] Specifically, in some embodiments, the updated representation of each message in the sample is sequentially input into a gated recurrent unit, and the output hidden layer is used as the attention representation of the speaker of the target message. The attention degree of the speaker of the target message to each message in the sample is calculated based on the attention representation, including:

[0080] Denote the update of the i-th message as u′ i Input gated recurrent unit, the expression is:

[0081] u′ i =concat(u i ,r I );

[0082]

[0083] Among them, u I represents the text representation of the i-th message, r i Indicates the relationship code between the i-th message and the target message; h i-1 represents the hidden layer output of the i-1th node, h i represents the hidden layer output of the i-th node;

[0084] Calculate the similarity e between the i-th message and the target message i , the calculation formula is:

[0085]

[0086] in, represents the transpose of the text representation of the i-th message, W represents the weight coefficient, h t Represents the final output of the gated recurrent unit;

[0087] Similarity e i Perform normalization to get the target message speaker's attention to the i-th message attn i , the calculation formula is:

[0088] attn i =softmax(ei ).

[0089] In step S1023, a thread is constructed based on the reply relationship of the above message. A thread is an ordered sequence consisting of one or more messages. In the sequence, except for the first message, the reply message of each other message is its previous message (the first message is a reply to itself). For example, for a group chat message A, B, C, D, E and F, where B replies to A, C replies to B, D replies to C, E replies to C, and F replies to E, three threads can be constructed, the first one is: A\B\C\D, the second one is: C\E\F, and the third one is: empty. The text representation of the target message is connected to each thread message respectively, and input into the mask gated recurrent unit respectively, and the final hidden layer representation is used as the content semantic representation of the thread. When the target message is connected to the thread to which it actually replies, there is a significant difference in the content semantic representation compared to when it is connected to other threads.

[0090] In step S1024, there are various discourse structure relationships between the target message and the previous messages in the candidate message set, such as comments, questions and answers, and continuations. In order to mine the discourse structure relationships between each previous message in the candidate set and the target message, the previous message and the target message are connected and processed through the first multi-layer perceptron to obtain the discourse structure representation.

[0091] In step S1025, for a specific previous message, the matching score with the target message is calculated based on its corresponding attention level, content semantic representation and discourse structure representation, and the final predicted reply object is determined and output.

[0092] In some embodiments, in step S1025, the attention level, content semantic representation, and discourse structure representation corresponding to each preceding message in the sample are concatenated and input into a second multi-layer perceptron. The output result is normalized to obtain a matching score corresponding to each preceding message in the sample, which is expressed as:

[0093] s i =softmax(MLP(concat(tho i ,disc i ,attn i )));

[0094] Among them, s i Represents the matching score, softmax is the normalized exponential function, MLP represents the operation of the second multi-layer perceptron, concat represents the connection operation, tho i Indicates the semantic representation of the content corresponding to the i-th message, disc i Represents the discourse structure representation corresponding to the i-th message, attn i Indicates the attention level corresponding to the i-th message.

[0095] In step S103, the initial network model is trained using the training sample set. Specifically, a binary cross entropy loss function may be used for training. For the i-th message, the loss function is:

[0096]

[0097] On the other hand, the present invention also provides a method for analyzing a conversation structure in a group chat, comprising steps S201 to S202:

[0098] Step S201: Obtain a target message to be analyzed and multiple group chat messages above the target message.

[0099] Step S202: input each group chat message and the target message into the above-mentioned group chat-oriented dialogue structure analysis model training method in sequence, and output the object message to which the target message actually replies.

[0100] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0101] On the other hand, a computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0102] The present invention will be described below in conjunction with specific embodiments:

[0103] This embodiment provides a multi-party conversation structure discovery method for group chat scenarios, including the following steps:

[0104] Step 1: Divide the original data and construct a training sample set. Each sample contains a target message and its previous message; identify each sample in sequence.

[0105] Step 2: Obtain the text representation of all message texts in each sample through BERT.

[0106] Step 3: Encode user relationships and obtain user attention semantic information representation and attention scores through GRU.

[0107] Step 4: Obtain content semantic representation through mask GRU.

[0108] Step 5: Obtain implicit discourse structure relationship representation through MLP.

[0109] Step 6: Fuse the representations obtained in steps 3, 4, and 5 and identify the dialogue structure.

[0110] Specifically, for clarity of the following description, the meaning of important symbols is first defined. The symbols and technical terms involved in this embodiment are as follows:

[0111] The target message represents the message of the current reply object to be identified. The parent message represents the real reply object of the target message. D represents a conversation. K represents the size of the sliding window. u t Indicates the target message. Indicates the speaker of the target message. Indicates the speaker of the target message's reply. Indicates u t The candidate set of reply messages. Indicates u t The candidate set of the thread to which the .

[0112] In order to better understand the above symbols, Figure 2 For example, when message 4 is the target message, its speaker The reply message is "scunizi", the reply message object is message 1, and the speaker of the reply object of the target message is The candidate set of reply messages for "Treeh416" and the target message It's messages 1, 2, 3, 4.

[0113] like Figure 1 As shown, the present invention provides a method for discovering a conversation structure based on multi-semantic fusion, comprising the following steps:

[0114] Step 1: Divide the original message stream data into sample sets that can be used for identification;

[0115] Given an original multi-party conversation dataset C = {u1,u2,…,u |C|},u i Represents the message in the data set. i Will be associated with a speaker a i , indicating a i Speak u i The dialogue structure discovery task is to find the t}, find the target message u t The reply message object. It is worth noting that u t It may be the beginning of a conversation, at which point it can be considered that u t The reply message object is itself. In order to simplify the problem, a time window parameter K is set up, so for a target message u t For example, given {u t-K+1 ,…,u t-1 ,u t} as the dialogue environment (also the reply message candidate set), in which the target message u is found t The real response object.

[0116] In addition, historical real labels need to be used during training, and historical predicted labels need to be used during testing to avoid the problem of data label leakage.

[0117] Step 2: Obtain the text representation of each message in the sample through the pre-trained model BERT. This embodiment uses the bert-base uncased pre-trained model officially provided by BERT.

[0118] Step 3: Encode the relationship between each message definition and the target message speaker in the sample after step 2, and obtain the user attention semantic information representation through GRU.

[0119] In the sliding conversation window, the speaker of the target message pays different attention to different messages in the candidate set. For example, the speaker of the target message is likely to pay more attention to the messages (and corresponding speakers) that reply to his earlier messages. Based on this, we propose a user (speaker) attention module to capture the speaker of the target message. The information of interest. This module mainly consists of two parts: target message speaker Encoder and target message speaker The former is mainly used to learn of the content, which will The interdependence with the messages in the candidate set aims to assign greater weights to message pairs that can better help complete the task.

[0120] Step 3.1: Encode user relationships.

[0121] Here we first define the relationship between the target speaker and other speakers, and then encode the relationship between the speakers, where the target speaker refers to the speaker of the target message. and other messages in the candidate set The relationship between can be defined as:

[0122]

[0123] It should be noted that this embodiment uses the code [1, -1] to define the relationship between the target message speaker and itself. Indicates the target message u t and u in the candidate set i A sequence of relations.

[0124] for Figure 2 , where the relationship between the messages is shown in Table 1:

[0125]

[0126] Table 1

[0127] Step 3.2 obtains the target speaker attention representation.

[0128] Connect the text representation obtained in the initial step 2 with the relational encoding to obtain a new message representation, namely:

[0129] u′ i =concat(u i ,r i );

[0130] The updated representation u′ of each message i Input into GRU in sequence, namely:

[0131]

[0132] H={h t-K+1 ,…,h t-1 ,h t};

[0133] Finally, the output hidden layer h t Targeted message speaker Attention representation.

[0134] Step 3.3: Get the target speaker's attention score for the candidate set messages.

[0135] Specifically, using target message speakers Characterization of h t As a query, u i (i={t-K+1,t-K+2,...,t}) is used as the key value, and the similarity of each pair is calculated, and then the softmax operation is performed to obtain For each candidate message u i Attention:

[0136]

[0137] attn i =softmax(e i );

[0138] Through the above method, the target message speaker can be obtained For each candidate message u i Attention:

[0139] Step 4: Obtain content semantic representation for the samples processed in step 2 through masked GRU. Intuitively, if the target message is a reply to message i, then after the target message participates in the thread with i as the tail node, the thread has more content consistency than other threads.

[0140] Step 4.1: Build the thread.

[0141] First, let's clarify the concept of thread. A thread is an ordered sequence of one or more messages. In this sequence, except for the first message, the reply message of each message is the previous message (the first message is a reply to itself). In order to explain the process of the model more clearly, Figure 2 Take Table 2 as an example, where the target message u t For u7, candidate message set For {u1,…,u6,u7}, the candidate thread set The candidate thread set is {th1,…,th6,th7}, and the specific content of the candidate thread set is shown in Table 2. Among them, th7 is an empty thread, so that the model can meet the situation where the target message u7 is the beginning of a new thread.

[0142]

[0143] Table 2

[0144] Step 4.2: Capture thread context semantics.

[0145] The target message is concatenated with each thread in the candidate thread set, with the target message as the last element in the thread set. The concatenated messages in the thread set pass through the GRU in sequence.

[0146]

[0147] The hidden layer representation h' obtained by the final node output t As the semantic representation of the content of this thread.

[0148] Perform the above calculations on each thread in the candidate set in turn, and finally obtain the representation of all threads is {tho1,…,tho2,tho7}, which will be used as the input of the matching module.

[0149] Step 5: Obtain implicit discourse structure relationship representation, i.e. discourse structure representation, for the samples in step 2 through MLP (multi-layer perceptron).

[0150] Target message u t and message u in the candidate message set iThere are many discourse structure relationships in a conversation, such as comments, questions and answers, and continuations. In the absence of manual discourse structure tags, this embodiment uses implicit discourse structure for representation instead of directly modeling explicit relationships.

[0151] Specifically, an MLP (Multi-layer Perceptron) is used to encode paired messages into latent message states:

[0152] disc i =MLP(concat(u i ,u t ));

[0153] Finally got the disc i As the target message u t and message u in the candidate message set i The same operation is performed on each message in the candidate message set, and finally the following is obtained:

[0154]

[0155] Step 6: Fuse the representations obtained in steps 3, 4, and 5 and identify the dialogue structure.

[0156] Step 6.1 For a given candidate message u i , splicing the representation of the thread to which it belongs tho i , and the target message u t The implicit discourse structure disc i 、Target message speaker to u i Attention score attn i , perform MLP and softmax operations on the concatenated representation to obtain the candidate matching score, namely:

[0157] s i =softmax(MLP(concat(tho i ,disc i ,attn i )));

[0158] Among them, s i Represents the matching score, softmax is the normalized exponential function, MLP represents the operation of the second multi-layer perceptron, concat represents the connection operation, tho i Indicates the semantic representation of the content corresponding to the i-th message, disc i Represents the discourse structure representation corresponding to the i-th message, attn i Indicates the attention level corresponding to the i-th message.

[0159] According to the candidate message ui The matching score determines whether it belongs to the target message u t The parent message.

[0160] Step 6.2: Use the binary cross entropy loss function to train, for each message u i For , its loss function is:

[0161]

[0162] When the candidate message u i is the target message u t The parent message time i =1, otherwise y i = 0. Under a well-trained model, the candidate message with the largest matching score is used as the predicted parent class of the target message.

[0163] In order to accurately evaluate the performance of the proposed model, experiments are conducted on two benchmark datasets and compared with the existing (state-of-the-art) model.

[0164] The first dataset is the Ubuntu IRC (Internet Relay Chat) public dataset, which comes from the log records of the Internet Relay Chat Log in Ubuntu's technical discussion platform.

[0165] The second dataset is a manually annotated dataset from WeChat and QQ groups, two commonly used multi-party conversation platforms. It contains messages from multiple groups over different time periods. The specific annotation strategy is consistent with that of the Ubuntu dataset. The proposed model is compared with the following baseline models:

[0166] (1) Previous: This algorithm selects the previous message as the parent message. This is the simplest strategy and does not consider any semantic information. It works best when the multi-party conversation is inactive.

[0167] (2) Feed-Forward (FF): This algorithm is provided by the provider of the Ubuntu IRC annotation dataset. It manually constructs 77 features and obtains a message vector representation by averaging the word Glove representations. The message representation and the 77 features are concatenated and passed through a two-layer feed-forward neural network.

[0168] (3) Masked Hierarchical Transformer (MHT): This is a two-stage Transformer model. The bottom stage is a fine-tuned BERT structure, and the top stage is a message-level encoding. By introducing the reply tag of the previous message into thread-level representation learning, the influence of irrelevant messages is avoided.

[0169] The accuracy, recall and F-1 value are selected as evaluation indicators.

[0170] Tables 3 and 4 show the experimental results of conversation reply relation recognition on the Ubuntu test set and the WeChat QQ group chat dataset, respectively.

[0171] Table 3 Experimental results of Ubuntu IRC dataset

[0172] Precision Recall F1 Previous 30.8 29.5 30.2 FF 36.3 39.7 38.0 MHT 53.9 51.7 52.8 BERT 33.6 31.9 32.7 eSIM 25.1 24.3 24.7 The present invention (K=16) 67.2 63.8 65.5

[0173] Table 4 Experimental results of WeChat & QQ dataset

[0174] Precision Recall F1 Previous 29.1 27.5 28.3 BERT 30.3 28.6 29.4 MHT 32.7 30.8 31.7 The present invention (K=10) 38.2 36.1 37.1

[0175] As can be seen from Tables 3 and 4, the model has a significant improvement over the comparison model. This is because other SOTA algorithms are based on message content-level semantics and do not consider the important role of users in group chats.

[0176] This paper presents a method for discovering conversation structure based on multi-semantic fusion. Unlike previous approaches that focused solely on the semantics of textual content information, this chapter incorporates user-level semantics and integrates them with content-level semantics to discover conversation structure. Specific semantics include clue content semantics, discourse structure semantics, and user attention semantics. Experimental results demonstrate that this technique achieves significant improvements in precision, recall, and F1 for conversation structure discovery tasks.

[0177] In summary, the conversation structure analysis model training method, analysis method, and device for group chat described in the present invention utilize a BERT model to obtain text representations for each message. This model then uses a GRU to extract the attention representation of the speaker of the target message, combining it with the defined relationship encoding between speakers. Corresponding threads are constructed based on the reply relationships of the preceding messages. The target message is then connected to each thread and fed into the GRU to obtain a content semantic representation of each thread. Finally, a multi-layer perceptron is used to obtain a discourse structure representation of the target message and the preceding message. This combined attention representation, content semantic representation, and discourse structure representation are used to jointly evaluate and analyze the target message's true reply recipient, significantly improving the accuracy of conversation structure analysis.

[0178] Those skilled in the art will appreciate that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether hardware or software is used depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment may be stored in a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried in a carrier wave. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. The code segments may be downloaded via a computer network such as the Internet or an intranet.

[0179] It should also be noted that the exemplary embodiments described herein describe methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the steps described above. In other words, the steps may be performed in the order described in the embodiments, or in a different order, or several steps may be performed simultaneously.

[0180] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0181] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A conversation structure analysis model training method for group chat, characterized by: include: Obtaining a training sample set, the training sample set including multiple samples, each sample including a target message to be analyzed and multiple previous messages of the target message, and marking the target message and each previous message's true reply object message; Obtaining an initial network model, wherein the processing of the initial network model includes: Initialize the text representation of each message in the sample using the BERT model; The target message and a set number of preceding messages are used as a candidate set, a relationship is defined between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code, and the text representation of each message in the sample is connected with the corresponding relationship code to obtain an updated representation; the updated representation of each message in the sample is sequentially input into a gated recurrent unit, and the finally obtained hidden layer representation is used as the attention representation of the speaker of the target message, and the attention of the speaker of the target message to each message in the sample is calculated based on the attention representation; Constructing multiple threads based on the reply relationships of the preceding messages in the sample, and connecting the target message to each thread to obtain a concatenated thread, inputting the text representation of each concatenated thread into a masked gated recurrent unit, and using the resulting hidden layer representation as the content semantic representation of the corresponding concatenated thread; After respectively concatenating the text representation of the target message with the text representation of the previous message in the candidate set, the resultant text is input into a first multi-layer perceptron to obtain a discourse structure representation; Concatenating the attention level, content semantic representation, and discourse structure representation corresponding to each preceding message in the sample and inputting them into a second multi-layer perceptron, normalizing the output results to obtain a matching score corresponding to each preceding message in the sample, and outputting a predicted reply object for the target message based on the matching score of each preceding message in the sample; The initial network model is trained using the training sample set to obtain a target dialogue structure analysis model.

2. The method for training a conversation structure analysis model for group chat according to claim 1, characterized in that: The BERT model is pre-trained using a specified dataset.

3. The method for training a conversation structure analysis model for group chat according to claim 1, wherein: Defining the relationship between the speaker of the target message and the speakers of all messages in the candidate set to obtain a relationship code includes: For the i-th message in the candidate set, if the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,0]; If the speaker of the i-th message is not the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [0,1]; If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is not the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,0]; If the speaker of the i-th message is the speaker of the target message, and the speaker of the i-th message reply object is the speaker of the target message reply object, then the relationship between the i-th message and the target message is defined as [1,1]; If the i-th message is the target message, the relationship between the i-th message and the target message is defined as [1, -1].

4. The method for training a conversation structure analysis model for group chat according to claim 3, wherein: Inputting the updated representation of each message in the sample into a gated recurrent unit in sequence, outputting a hidden layer as the attention representation of the speaker of the target message, and calculating the attention of the speaker of the target message to each message in the sample based on the attention representation, including: Denote the update of the i-th message as u′ i Input gated recurrent unit, the expression is: u′ i =concat(u i ,r i ); Among them, u i represents the text representation of the i-th message, r i Indicates the relationship code between the i-th message and the target message; h i-1 represents the hidden layer output of the i-1th node, h i represents the hidden layer output of the i-th node; Calculate the similarity e between the i-th message and the target message i , the calculation formula is: in, represents the transpose of the text representation of the i-th message, W represents the weight coefficient, h t Represents the final output of the gated recurrent unit; The similarity e i Perform normalization to get the target message speaker's attention to the i-th message attn i , the calculation formula is: attn i =softmax(e i )。 5. The method for training a conversation structure analysis model for group chat according to claim 4, characterized in that: In the sample, the attention degree, content semantic representation, and discourse structure representation corresponding to each above message in the sample are connected and input into the second multi-layer perceptron. The output results are normalized to obtain the matching score corresponding to each above message in the sample, which is expressed as: s i =softmax(MLP(concat(tho i ,disc i ,attn i ))); Among them, s i Represents the matching score, softmax is the normalized exponential function, MLP represents the operation of the second multi-layer perceptron, concat represents the connection operation, tho i Indicates the semantic representation of the content corresponding to the i-th message, disc i Represents the discourse structure representation corresponding to the i-th message, attn i Indicates the attention level corresponding to the i-th message.

6. The method for training a conversation structure analysis model for group chat according to claim 5, characterized in that: The initial network model is trained using the training sample set, including: training using a binary cross entropy loss function, where for the i-th message, the loss function is: Among them, s i Represents the matching score, when the i-th message is the parent message of the target message y i =1, when the i-th message is not the parent message of the target message y i =0.

7. The method for training a conversation structure analysis model for group chat according to claim 6, wherein: In the sample, if the target message is the beginning of a conversation, the real reply object of the target message is marked as the target message itself.

8. A method for analyzing conversation structure in group chat, characterized in that: include: Obtaining a target message to be analyzed and multiple group chat messages above the target message; Each group chat message and the target message are sequentially input into the conversation structure analysis model training method for group chat as described in any one of claims 1 to 7, and the object message to which the target message actually replies is output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images