Dialogue analysis method and device, glasses, equipment and readable storage medium

Abstract

According to the dialogue analysis method and device, the glasses, the equipment and the readable storage medium, the dialogue audio is collected in real time and converted into the dialogue text, the dialogue text is labeled on the basis of the speech function type of the speech round, deep information support can be provided for understanding of the dialogue type, and the dialogue analysis efficiency is improved. Key information and structural features in the dialogue are accurately identified by using a sequence detection rule of a preset dialogue type and a machine learning algorithm, so that feedback information is immediately generated and displayed, a dialogue guider quickly knows the real-time condition of the dialogue in the dialogue process, dialogue adjustment and dialogue process optimization are conveniently and quickly performed, and the dialogue efficiency is improved. And the dialogue efficiency and effect are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a conversation analysis method, apparatus, glasses, equipment and readable storage medium. Background Art

[0002] At present, traditional conversation analysis scenarios usually rely on recording and videotaping to record conversations, and after the entire conversation is over, a large language model is used to analyze the conversation content in detail to identify problems in the conversation process. However, this analysis method cannot meet the needs of conversation scenarios that require efficient feedback on the conversation situation and flexible adjustment of the conversation direction, such as classroom teaching, customer interviews, and hosting scenarios. Taking classroom teaching scenarios as an example, teachers need to wait until after class to obtain the conversation situation through classroom conversation analysis, and then trace back the teaching situation in the classroom for subsequent teaching adjustments. This misses the important opportunity to adjust teaching strategies and interaction methods in real time in the classroom, affecting the teaching effect.

[0003] Therefore, traditional conversation analysis methods have obvious limitations in scenarios that require efficient conversation feedback to flexibly adjust the conversation direction. There is an urgent need for a conversation analysis method that can achieve efficient feedback on the conversation situation and flexibly adjust the conversation direction. Summary of the invention

[0004] In view of this, in order to solve the above-mentioned technical problems, the present application provides a conversation analysis method, apparatus, glasses, equipment and readable storage medium.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a conversation analysis method is provided, the method comprising:

[0007] Converting the conversation audio received in real time into conversation text;

[0008] Using a first text network to annotate each turn in the dialogue text, obtaining an annotated text with annotated results; the annotated results are used to indicate the discourse function type of the turn;

[0009] Obtaining a sequence of annotation results according to the turn order and the annotation results in the annotated text;

[0010] Determine a sequence recognition result of the labeled result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes a dialogue type to which the labeled result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs;

[0011] According to the sequence recognition result, the annotated text is diagnosed using a second text network to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue situation to optimize the dialogue.

[0012] Optionally, the using the first text network to mark each turn in the dialogue text includes:

[0013] Obtain classification conditions for each preset annotation label; the preset annotation label corresponds one-to-one to the discourse function type of the turn in the dialogue context; construct a prompt text suitable for the prompt project according to the classification conditions; use the first text network and the prompt text to identify the annotation label belonging to each turn in the dialogue text and annotate it.

[0014] Optionally, the using the first text network to mark each turn in the dialogue text includes:

[0015] The text content in the turn is input into the first text network to obtain the annotation result output by the first text network; wherein, the output layer of the first text network is provided with a plurality of classification labels, and the classification labels correspond one-to-one to the discourse function types of the turn in the dialogue context; the annotation result is obtained according to the classification labels greater than a set threshold in the probability distribution of the plurality of classification labels generated by the first text network.

[0016] Optionally, determining the sequence recognition result of the labeled result sequence according to a sequence detection rule of a preset dialog type includes:

[0017] In the case where the marked result sequence satisfies any of the sequence detection rules, determining a preset dialogue type corresponding to the satisfied sequence detection rule as the dialogue type to which the marked result sequence belongs, and generating a first recognition result;

[0018] When the marked result sequence does not satisfy the sequence detection rules of all preset dialogue types, it is determined that the dialogue type indicates an unknown dialogue type, and the rule information that the marked result sequence does not satisfy is determined for each sequence detection rule to generate a second recognition result.

[0019] Optionally, the diagnosing the annotated text using a second text network according to the sequence recognition result includes:

[0020] When the sequence recognition result is the first recognition result, a dialogue target of a preset dialogue type corresponding to the satisfied sequence detection rule is obtained; and according to the dialogue target, the annotated text is diagnosed using the second text network.

[0021] Optionally, the diagnosing the annotated text using a second text network according to the sequence recognition result includes:

[0022] In the case where the sequence recognition result is the second recognition result, the rule information included in the second recognition result is obtained; and according to the rule information and the unknown dialogue type indicating the corresponding dialogue type, the annotated text is diagnosed using the second text network.

[0023] Optionally, the method further comprises:

[0024] Acquire video data collected simultaneously with the conversation audio; align the video data with the annotated text according to timestamps; determine the level of participation of the conversation person according to the video data and the annotated text; determine detection parameters matching the level of participation of the conversation person according to a preset corresponding relationship; the corresponding relationship includes a preset level of participation and detection parameters corresponding to each level of participation;

[0025] The diagnosing the annotated text using a second text network according to the sequence recognition result includes: diagnosing the annotated text using the second text network according to the sequence recognition result and the detection parameter.

[0026] Optionally, the method further comprises:

[0027] Outputting the feedback information in a structured mode; the feedback information may include at least one of dialogue participation status, interaction prompts, and dialogue direction suggestions;

[0028] According to a preset page layout, the feedback information is converted into a virtual page and projected and displayed in the user's field of vision.

[0029] According to a second aspect of an embodiment of the present application, a conversation analysis device is provided, the device comprising:

[0030] An audio conversion module, used for converting the conversation audio received in real time into conversation text;

[0031] a marking module, configured to mark each turn in the dialogue text using a first text network to obtain a marked text with marking results; the marking results are used to indicate the discourse function type of the turn;

[0032] A sequence acquisition module, used for obtaining a sequence of annotation results according to the turn order and the annotation results in the annotated text;

[0033] A sequence recognition module, used to determine a sequence recognition result of the labeled result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes the dialogue type to which the labeled result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs;

[0034] A feedback information generating module is used to diagnose the annotated text using a second text network according to the sequence recognition result to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue situation to optimize the dialogue.

[0035] Optionally, the marking module is specifically used for:

[0036] Obtain classification conditions for each preset annotation label; the preset annotation label corresponds one-to-one to the discourse function type of the turn in the dialogue context; construct a prompt text suitable for the prompt project according to the classification conditions; use the first text network and the prompt text to identify the annotation label belonging to each turn in the dialogue text and annotate it.

[0037] Optionally, the marking module is specifically used for:

[0038] The text content in the turn is input into the first text network to obtain the annotation result output by the first text network; wherein, the output layer of the first text network is provided with a plurality of classification labels, and the classification labels correspond one-to-one to the discourse function types of the turn in the dialogue context; the annotation result is obtained according to the classification labels greater than a set threshold in the probability distribution of the plurality of classification labels generated by the first text network.

[0039] Optionally, the sequence recognition module is specifically used to:

[0040] In the case where the marked result sequence satisfies any of the sequence detection rules, determining a preset dialogue type corresponding to the satisfied sequence detection rule as the dialogue type to which the marked result sequence belongs, and generating a first recognition result;

[0041] When the marked result sequence does not satisfy the sequence detection rules of all preset dialogue types, it is determined that the dialogue type indicates an unknown dialogue type, and the rule information that the marked result sequence does not satisfy is determined for each sequence detection rule to generate a second recognition result.

[0042] Optionally, the feedback information generating module is specifically used to:

[0043] When the sequence recognition result is the first recognition result, a dialogue target of a preset dialogue type corresponding to the satisfied sequence detection rule is obtained; and according to the dialogue target, the annotated text is diagnosed using the second text network.

[0044] Optionally, the feedback information generating module is specifically used to:

[0045] In the case where the sequence recognition result is the second recognition result, the rule information included in the second recognition result is obtained; and according to the rule information and the unknown dialogue type indicating the corresponding dialogue type, the annotated text is diagnosed using the second text network.

[0046] Optionally, the device further comprises:

[0047] Acquire video data collected simultaneously with the conversation audio; align the video data with the annotated text according to timestamps; determine the level of participation of the conversation person according to the video data and the annotated text; determine detection parameters matching the level of participation of the conversation person according to a preset corresponding relationship; the corresponding relationship includes a preset level of participation and detection parameters corresponding to each level of participation;

[0048] The feedback information generation module is specifically used to: diagnose the annotated text using the second text network according to the sequence recognition result and the detection parameter.

[0049] Optionally, the device further comprises:

[0050] The feedback information is output in a structured mode; the feedback information may include at least one of dialogue participation status, interaction prompts, and dialogue direction suggestions; according to a preset page layout, the feedback information is converted into a virtual page and projected and displayed in the user's field of vision.

[0051] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned conversation analysis method by calling the computer program.

[0052] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned conversation analysis method is implemented.

[0053] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0054] In the technical solution provided by the present application, by collecting conversation audio in real time and converting it into conversation text, and annotating the conversation text based on the speech function type of the turn, it is possible to provide in-depth information support for the understanding of the conversation type, and utilize the sequence detection rules and machine learning algorithms of the preset conversation type to accurately identify the key information and structural features in the conversation, thereby instantly generating and displaying feedback information, so that the conversation leader can quickly understand the real-time situation of the conversation during the conversation, facilitate rapid conversation adjustments, optimize the conversation process, improve conversation efficiency and effectiveness, promote effective communication and interaction between conversation participants, and improve the overall quality of the conversation and the satisfaction of the participants.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] Figure 1A It is a schematic diagram of the overall process of a conversation analysis method shown in an exemplary embodiment of the present application;

[0058] Figure 1B This is an example of a speech function type of a preset turn in a classroom teaching scenario shown in an exemplary embodiment of the present application;

[0059] Figure 1C This is an example of a preset dialogue type divided based on different teaching objectives in a classroom teaching scenario shown in an exemplary embodiment of the present application;

[0060] Figure 2 It is a flowchart of a step of obtaining a sequence recognition result of a labeling result sequence corresponding to a labeling text based on a sequence detection rule, shown in an exemplary embodiment of the present application;

[0061] Figure 3 is a flowchart of a second text network diagnosis step according to a sequence recognition result shown in an exemplary embodiment of the present application;

[0062] Figure 4 is another flowchart of the second text network diagnosis steps according to the sequence recognition result shown in an exemplary embodiment of the present application;

[0063] Figure 5 is a schematic diagram of a glasses structure to which a conversation analysis method is applied, shown as an exemplary embodiment of the present application;

[0064] Figure 6is a structural diagram of a conversation analysis device shown in an exemplary embodiment of the present application;

[0065] Figure 7 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0066] Next, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0067] It should be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In this application, the terms first, second, third, etc. may be used to describe various information, but these information should not be limited to these terms, which are only used to distinguish the same type of information from each other.

[0068] In the current conversation analysis scenario, traditional conversation recording and analysis methods mainly rely on recording and video recording to capture and save the conversation content in order to fully save the entire conversation process. After the entire conversation process is over, a large language model is used to analyze and interpret the recorded conversation content in detail to identify possible problems or deficiencies in the conversation process.

[0069] As various industries have higher requirements for the quality of conversational interactions, this traditional conversation analysis method has obvious limitations. That is, this conversation analysis method can achieve a comprehensive review and evaluation of the conversation content to a certain extent, but its inherent lag limits its application effect in conversation scenarios that require immediate feedback and adjustment.

[0070] Taking classroom teaching scenarios as an example, since traditional dialogue analysis methods require analysis and evaluation of dialogue content after class, teachers often can only get feedback on classroom dialogues after class, thus missing the important opportunity to adjust teaching strategies and interaction methods in real time in class. This affects teachers' precise control of the teaching process and restricts the improvement of teaching effectiveness. It cannot meet the needs of teachers in this scenario for efficient feedback on dialogue situations and for flexible adjustment of teaching strategies and interaction methods based on real-time interaction with students.

[0071] In addition, in other conversation scenarios that require immediate feedback and adjustment, such as customer interviews and hosting scenarios, traditional conversation analysis methods also have the same limitations. Therefore, there is an urgent need for a conversation analysis method that can achieve efficient feedback on the conversation situation and flexibly adjust the conversation direction.

[0072] In view of the above technical problems, this application proposes a conversation analysis method, which can be executed by a server, an intelligent terminal device such as a mobile phone or a computer, or a dedicated hardware and chip that can provide the computing power required for conversation analysis. This application does not limit this. The method can capture and analyze the content of the conversation in real time, and provide instant feedback and adjustment suggestions on the conversation situation, thereby effectively improving the quality and efficiency of the conversation.

[0073] This conversation analysis method can be widely used in various conversation scenarios that require instant feedback and flexible adjustment of the conversation situation, such as conference hosting scenarios, classroom teaching scenarios, psychological counseling scenarios, customer interview scenarios, etc. The conversation analysis method can be used to provide conversation leaders in conversation scenarios (such as hosts, teachers, psychological counselors, etc.) with instant assessment results of the specific situation of the current conversation content, prompt the shortcomings of the conversation content and make suggestions for conversation adjustments, so as to improve the quality of conversation interaction, optimize the conversation process, and promote effective communication and interaction between conversation participants.

[0074] See also Figure 1A The step flow chart shown exemplarily, the method may include at least the following steps:

[0075] S101, converting the conversation audio received in real time into conversation text;

[0076] The conversation audio received in real time refers to a digital audio signal captured and transmitted in real time during a conversation through a microphone or other audio input device. For example, in a classroom teaching scenario, the conversation audio is a classroom conversation captured in real time, including conversation interactions between teachers and students. When the method execution subject and the audio input device that collects the conversation audio are located in the same device, the conversation audio can be transmitted through a short-range fast transmission method such as Bluetooth Wi-Fi direct connection or wired connection; when the method execution subject and the audio input device are not located in the same device, if the conversation audio collected by the microphone is to be transmitted in real time to a cloud server for processing, the conversation audio can be transmitted through the Internet, local area network and other networks.

[0077] The dialogue text refers to the content that converts the voice information in the dialogue audio into readable text form, which can be stored in the form of turns. A turn is a basic unit in a dialogue, which refers to a continuous speech in a dialogue, spoken by one participant until it is interrupted or taken over by another participant. In the dialogue text, each turn can be clearly identified and is usually distinguished by line breaks, punctuation marks or other separators. The dialogue text may include speaker identification, speech content, timestamp information indicating the time point when the turn occurred, and other information such as sentiment tags, semantic role labels, etc.

[0078] In order to reduce background noise and improve the clarity of voice signals, the received conversation audio can be preprocessed before the voice content of the conversation audio is converted into conversation text, including noise reduction, gain control, echo cancellation, endpoint detection and other operations, so as to improve the accuracy of speech recognition.

[0079] Regarding converting the conversation audio into the conversation text, the pre-trained speech recognition model can be used to extract audio features from the digital audio signal and convert the extracted audio features into text content. The speech recognition model can be a neural network model based on deep learning, such as a recurrent neural network, a long short-term memory network or a transformer, etc. These models can learn the mapping relationship between the speech signal and the text content through training with a large amount of speech data.

[0080] S102, using a first text network to annotate each turn in the dialogue text to obtain an annotated text with annotated results; the annotated results are used to indicate the discourse function type of the turn;

[0081] A turn refers to a continuous speech by a speaker in a conversation until another speaker starts speaking. It is the basic unit of a conversation. Each turn contains a complete semantic unit, which can be a sentence or multiple sentences. For example, A says a paragraph and B responds with a sentence. Then A's paragraph belongs to one turn, and B's sentence belongs to another turn. For example, the conversation text example: Teacher: "Please describe the painting you saw yesterday.", Student: "The painting is a forest with many green trees.", then the teacher's speech belongs to one turn, and the student's speech belongs to another turn, that is, the conversation text example contains two turns.

[0082] The discourse function type is used to indicate the role or function of a turn in a conversation, such as asking questions, answering, stating, supplementing, etc., to facilitate understanding of the structure and context of the conversation and to use it for subsequent conversation analysis and processing. The discourse function type is related to the conversation context to which the received conversation audio belongs. Different conversation contexts have unique interaction purposes, social norms, and information flow patterns, which will affect the function of the turn. Therefore, in order to more accurately reflect the characteristics and needs of the conversation in the conversation context, the classification of discourse function types can be set for different conversation contexts.

[0083] For example, if the audio is collected from a classroom conversation in a classroom teaching scenario, the conversation context to which the conversation audio belongs is the classroom teaching scenario. Figure 1BAs shown, the discourse function types of the turn in this dialogue context may include but are not limited to: invitation to elaborate, elaboration, invitation to reason, explanation of reasons, invitation to coordinate ideas, approval, questioning, review and citation, extensive reference, other invitations, silence, etc. Different discourse function types cooperate with each other in classroom teaching scenarios to jointly construct rich classroom dialogues, promote the transfer of knowledge, the cultivation of thinking and the conduct of interaction. Similarly, in other dialogue scenarios, such as the host scenario, the discourse function types may include but are not limited to opening greetings, guiding topics, summarizing, inviting speeches, adjusting the atmosphere, etc., forming a unique discourse function type system according to the characteristics of the dialogue context to meet different interaction needs.

[0084] In text processing, annotation refers to the process of adding additional information (such as discourse function type or emotion label) to each unit (such as sentence or turn) in the text. The annotation result is the information obtained after annotation. In this embodiment, it represents the annotation information assigned to each turn in the dialogue text, which represents the discourse function type of the turn.

[0085] The first text network is used to identify and annotate the speech function type of each turn in the dialogue text, and the basis for the turn annotation is the speech function types of multiple turns preset for the dialogue context to which the collected dialogue audio belongs. The first text network may include a deep learning network, such as a convolutional neural network, a recurrent neural network, a long short-term memory network, a Transformer, etc., which can be used as the first text network after being pre-trained or fine-tuned according to the annotation task of the present application. In this embodiment, for the acquired dialogue text, each turn can be identified and segmented by detecting pauses, line breaks or other separators in the dialogue text, and then each turn is passed as input to the first text network, so that the first text network identifies the speech function type of each turn and outputs the annotation result representing the speech function type of the turn or the turn with the annotation result. By identifying and annotating the speech function type of each turn (i.e., a continuous speech of each speaker) in the dialogue text, a text with annotation information is finally generated.

[0086] In the process of annotating using the first text network dialogue round, in order to ensure the consistency and readability of the annotation results and improve the efficiency of subsequent analysis and processing of the annotation results, the discourse function type can be represented by coding, that is, a unique code is set for each discourse function type, so that when annotating in the dialogue round, the corresponding discourse function type can be quickly identified and the corresponding coding annotation can be added. For example, see Figure 1B The coding example in , sets a unique code for each discourse function type in the aforementioned example classroom teaching scenario.

[0087] For example, suppose there is a conversation text M1 as follows:

[0088] Teacher: "Please describe the painting you saw yesterday."

[0089] Student: “That picture is a forest with lots of green trees.”

[0090] Teacher: "Can you tell me why this painting made a deep impression on you?"

[0091] Student: “Because the forest looks very peaceful and makes me feel relaxed.”

[0092] The annotated text M2 after being annotated by the first text network can be represented as follows:

[0093] Teacher: "Please describe the painting you saw yesterday." [Invitation to Elaborate / ELI]

[0094] Student: "That picture is a forest with lots of green trees." [elaboration / EL]

[0095] Teacher: "Can you tell me why this painting made a deep impression on you?" [Invitation to Reasoning / REI]

[0096] Student: "Because the forest looks very peaceful and makes me feel relaxed." [Explain reason / RE]

[0097] It should be understood that, based on the fact that a turn represents a continuous speech of a speaker, including at least one sentence, a turn corresponds to one or more discourse function types. In the process of annotating using the first text network dialogue turn, when the turn corresponds to multiple discourse function types, annotations representing the discourse function types can be added to the same turn in sequence according to the order of the text content in the turn, and the annotation results representing the multiple discourse function types are used as the annotation results of the turn. That is, the same turn can include multiple annotations representing discourse function types, each of which corresponds to a specific text content part in the turn.

[0098] For example, taking the discourse function types in the aforementioned classroom teaching situation as an example, for the turn-based example "According to the Pythagorean theorem we learned in the last class, can you explain why here...?", the text content in the turn "According to the Pythagorean theorem we learned in the last class" represents review and quotation / RB, and the text content "Can you explain why" represents asking a question / Q, that is, the turn corresponds to two discourse function types, and the annotation result of the turn can be expressed as [review and quotation, asking a question] or [RB, Q].

[0099] S103, obtaining a marking result sequence according to the turn order and the marking result in the marked text;

[0100] The annotation result sequence is an abstract representation of the dialogue text, which removes the specific dialogue content in the dialogue text and retains the functional type and order relationship of the turn. It contains the annotation results of each turn in the dialogue text that represents the discourse function type of the turn, and is arranged in the order in which the turns appear in the dialogue text.

[0101] For example, for the example annotated text M2 in the previous text, taking the annotation result in coded form as an example, the annotation result sequence corresponding to the annotated text can be expressed as [ELI, EL, REI, RE].

[0102] S104, determining a sequence recognition result of the annotation result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes a dialogue type to which the annotation result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs;

[0103] The preset dialogue type is a dialogue type set according to different dialogue goals in the dialogue context to which the collected dialogue audio belongs. Different dialogue types are used to indicate that the dialogue in the dialogue context has different orientations and characteristics. For example, taking the dialogue context as classroom teaching, different teaching goals in the classroom are reflected through the interaction between teachers and students and the content of the topic, such as Figure 1C As shown in the figure, classroom dialogues can be divided into four categories according to teaching objectives: critical inquiry, collaborative knowledge construction, teaching and support dialogue, and reflection and metacognitive dialogue. Each type of dialogue corresponds to a different educational objective shown in the figure. Among them, critical inquiry dialogues are used to stimulate students' in-depth thinking and questioning ability about the knowledge they have learned; collaborative knowledge construction dialogues refer to the construction of knowledge through the coordination of multiple views, joint problem solving, and sharing of resources and ideas; teaching and support dialogues are used to indicate that teachers provide knowledge explanations through dialogues with students and provide teaching examples to support their arguments; reflection and metacognitive dialogues are used to guide dialogue participants to reflect on the knowledge they have learned or the methods and strategies adopted in history.

[0104] Based on the different annotation result sequence patterns for different dialogue types, this embodiment sets sequence detection rules for each dialogue type. The sequence detection rules include a set of pre-defined rules, which are set based on the dialogue target corresponding to the dialogue type, and the characteristics, structure and other information of the dialogue type. The rules are set based on the speech function type of the turn in the dialogue context to which the dialogue audio belongs, taking into account factors such as the order of turns in the dialogue, the relationship between the speech function types between turns, the frequency of occurrence or combination of specific turns, etc., and are used to identify the dialogue type to which the annotated text belongs according to the annotation result sequence corresponding to the annotated text.

[0105] For example, taking the dialogue type - critical inquiry in the aforementioned classroom teaching as an example, its sequence detection rule can be expressed as: [the following three conditions are met simultaneously in the same topic: ① at least one of invitation for elaboration ELI, invitation for reasoning REI, and invitation for coordination CI; ② at least one of elaboration EL and explanation of reasons RE; ③ raising questions Q]. Based on this exemplary sequence detection rule, for example, the annotation result sequence pattern that meets the sequence detection rule can include but is not limited to [REI, RE, Q], [Q, RE, REI], [CI, Q, RE], [ELI, Q, RE], etc.

[0106] According to the preset sequence detection rules of different dialogue types, this embodiment matches or compares the annotation result sequence obtained in step S103 with the preset sequence detection rules of each dialogue type, analyzes the matching degree of the functional type and order relationship of the turn in the annotation result sequence with the sequence detection rules, determines the dialogue type to which the annotation result sequence belongs according to whether the annotation result sequence corresponding to the annotated text satisfies any sequence detection rule, and generates a sequence recognition result. The sequence recognition result is used to indicate the matching situation of the annotation result sequence and the sequence detection rule, and provides the overall structure and type information of the dialogue audio. Based on the fact that this embodiment annotates the dialogue text received in real time to obtain the annotation result sequence, the matching situation corresponding to the annotation result sequence is divided into two types, namely, satisfying the sequence detection rules of one preset dialogue type and not satisfying the sequence detection rules of all preset dialogue types. For the matching situation of not satisfying the sequence detection rules of all preset dialogue types, it can be determined that the dialogue type to which the annotation result sequence belongs is an unknown type.

[0107] S105, based on the sequence recognition result, diagnose the annotated text using a second text network to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue status to optimize the dialogue.

[0108] The second text network is used to further analyze, diagnose or evaluate the annotated text. It can be a pre-trained deep learning model such as a convolutional neural network, a recurrent neural network or a variant thereof, or it can be other types of machine learning models that are fine-tuned for text diagnosis tasks. Compared with the first text network, the first text network described in the above steps is mainly used for annotation of dialogue texts, while the second text network focuses on analyzing and diagnosing the annotated text, thereby outputting feedback information for guiding and adjusting the dialogue, so as to achieve the effect of instant feedback on the dialogue situation and flexible adjustment of the dialogue based on the feedback information.

[0109] In this embodiment, the second text network is used to perform a deeper analysis on the annotated text according to the dialogue target corresponding to the dialogue type included in the sequence recognition result, obtain the problems or improvement points in the dialogue process represented by the annotated text, and generate feedback information to optimize the current dialogue process. The diagnosis of the second text network may include but is not limited to data analysis and processing in aspects such as discourse function diagnosis, turn structure analysis, and participation-related factor analysis.

[0110] The feedback information is the result generated by the second text network after diagnosing the annotated text, and is used to provide information or suggestions about the dialogue situation to the dialogue leader, which may include but is not limited to dialogue quality assessment, dialogue improvement suggestions, dialogue precautions, etc., such as the evaluation of the fluency, participation, topic concentration, etc. of the dialogue, whether the progress of the dialogue is in line with expectations, whether there are certain types of discourse function missing or overused, etc., so that the dialogue leader can understand the current state of the dialogue, adjust the dialogue strategy in time, and improve the dialogue effect. Among them, the dialogue leader refers to the speaker who plays a guiding role in the dialogue, who is responsible for raising topics, guiding discussions, or ensuring the smooth progress of topics, such as teachers in classroom teaching scenarios, hosts in hosting scenarios, and psychological counselors in psychological counseling scenarios.

[0111] Regarding diagnosing the annotated text using the second text network based on the sequence recognition result, after obtaining the sequence recognition result, based on the dialogue type to which the annotated result sequence included in the sequence recognition result belongs, a preset prompt template corresponding to the dialogue type is obtained, and the prompt template is filled in according to the information carried in the sequence recognition result to obtain a diagnosis prompt text, and then the diagnosis prompt text and the annotated text are input into the trained second text network together to obtain feedback information output by the second text network based on the diagnosis result.

[0112] For example, in a classroom teaching scenario, the sequence recognition result is obtained through the above steps, and a diagnostic prompt text is constructed based on the sequence recognition result, and the diagnostic prompt text and the annotated text are used as the input of the second text network; the second text network extracts high-level text features from the input based on the parameters learned in the pre-training stage, and generates corresponding feedback information through the output layer of the feature-generated text. For example, the feedback information can include teacher guidance suggestions generated according to the classroom situation, such as "please avoid long explanations", "it is recommended to switch to the question-and-answer mode", "it is recommended to further explain the current topic", to guide the adjustment of teaching strategies.

[0113] In the disclosed embodiment, by quickly converting the conversation audio received in real time into conversation text, and using the first text network to construct annotated text with annotation results, the discourse function type of each turn in the conversation text is accurately identified, so as to facilitate a better and in-depth understanding of the structure and content of the conversation, generate an annotation result sequence based on the order of turns in the annotated text and the annotation results, and use sequence detection rules of multiple conversation types with different settings based on conversation goals in the current conversation context to identify the conversation type to which the annotation result sequence belongs to generate a sequence recognition result, and then combine the sequence recognition result with machine learning to perform in-depth diagnosis of the annotated text, identify problems or deficiencies in the conversation, generate feedback information and convey it to the conversation leader, realize instant conversation analysis and conversation situation feedback during the conversation, enable the conversation leader to understand the conversation situation in real time during the conversation, and flexibly adjust the conversation rhythm and direction according to the feedback, which helps to guide the conversation to a more efficient and in-depth direction and improve the conversation quality and effect.

[0114] Based on the conversation analysis method provided in the above-mentioned embodiment, taking the classroom teaching scenario as an example, during the classroom teaching dialogue, the teacher can obtain feedback information instantly through real-time conversation analysis to understand the students' participation, understanding and interaction quality; for example, if the feedback information shows that some students have low participation in a certain topic, the teacher can immediately adjust the teaching strategy, increase the interactive link, and improve the students' participation and learning effect. Taking the hosting scenario as an example, by real-time collection of the dialogue during the hosting process and using this method for analysis, the host can obtain feedback information instantly during the program to understand the audience's reaction and interaction; for example, if the feedback information shows that the audience is not very interested in a certain topic, the host can immediately adjust the topic, increase the interactive link, and improve the attractiveness of the program and the audience's participation.

[0115] After describing the overall process of the conversation analysis method provided in the present application through the aforementioned embodiments, the implementation details in each process link will be further explained and illustrated.

[0116] In some embodiments, the conversion of the conversation audio received in real time into conversation text as described in the aforementioned step S101 can be achieved through automatic speech recognition technology, namely ASR (Automatic Speech Recognition). ASR technology can convert speech signals into text through steps such as speech preprocessing, feature extraction, acoustic model matching and language model decoding. Alternatively, the model can be trained and optimized through technologies such as machine learning and natural language processing, and the trained model can be used to process the input speech content and output text. For example, the acoustic model can be trained with a large amount of speech data to improve the recognition ability of different pronunciations and accents, and the converted text can be verified and corrected in terms of grammar and semantics through natural language processing technology.

[0117] Taking automatic speech recognition technology as an example, the conversation text can be obtained through the following steps, namely: for the conversation audio, real-time text transcription of the conversation audio is performed through speech recognition technology, and an initial text in the form of a conversation is output; data preprocessing is performed on the initial text to obtain the conversation text; the data preprocessing includes at least one of data cleaning, syntactic analysis, and text normalization.

[0118] That is, when the initial text after the speech conversion text is obtained through speech recognition technology, the converted initial text can be preprocessed to further improve the accuracy and fluency of the dialogue text, and enhance the text quality and readability. Among them, data cleaning is used to remove noise and irrelevant information in the initial text, such as incorrect punctuation, extra spaces, repeated words, etc., to ensure the neatness and consistency of the text; syntactic analysis is used to parse the grammatical structure of the initial text, perform part-of-speech tagging and other text processing that is conducive to understanding the sentence, help identify and correct grammatical errors in the sentence, and improve the logic and readability of the text; text normalization is to convert non-standard terms, abbreviations, slang, etc. in the initial text into standard language forms to ensure that the text conforms to the general language standards and is easy to understand and use. Through this data preprocessing step, the quality of the dialogue text output by the automatic speech recognition technology can be significantly improved, making it more accurate, fluent and easy to understand, providing a more reliable foundation for subsequent dialogue text analysis, information extraction and other tasks.

[0119] In some embodiments, for the aforementioned step S102, each turn in the dialogue text is annotated using the first text network to obtain an annotated text with annotated results. The annotated results added by the annotation can exist in the form of text labels, digital codes or other metadata. The annotated results are used to enrich the content of the dialogue text, so as to fully understand the structure and content of the dialogue text and accurately generate feedback information. In the process of annotating each turn in the dialogue text using the first text network, the first text network can first determine the dialogue context to which the collected dialogue audio belongs, and obtain multiple speech function types that the turn may have that are predefined for the dialogue context; next, for each turn in the dialogue text, the first text network is used to identify whether the text content in the turn conforms to one or more of the predefined multiple speech function types, and the turn is annotated according to the order of the text content in the turn and the speech function type dialogue turn that the text content conforms to, to obtain the annotation result of the turn.

[0120] Regarding determining the discourse function type that a turn corresponds to, it can be achieved by setting the first text network as a large language model and combining it with a prompting project, or by setting the first text network as a pre-trained classification model. Furthermore, when the discourse function type of the turn is determined, the first text network can also perform annotation processing synchronously, that is, input the dialogue text to the first text network, and obtain the annotated text composed of the turn with the annotation results output by the first text network.

[0121] Taking the first text network including a large language model and combined with the prompt project as an example, the step of using the first text network to annotate each turn in the dialogue text can be implemented in the following way: obtaining the classification conditions of each preset annotation label; the preset annotation label corresponds one-to-one to the discourse function type of the turn in the dialogue context; constructing a prompt text suitable for the prompt project according to the classification conditions; using the first text network and the prompt text, identifying the annotation labels belonging to each turn in the dialogue text and annotating them.

[0122] The classification condition is used to define the discourse function type corresponding to the preset annotation label, and can be set from the aspects of text content, tone, vocabulary and phrases in the turn. For example, for the discourse function type "invite to elaborate", its classification condition is used to define or explain what kind of text content meets the category of invitation to elaborate.

[0123] The prompt text suitable for the prompt project constructed according to the classification conditions includes a description of each preset annotation label, which is used to guide the first text network to more accurately identify the discourse function type that the turn conforms to, that is, the annotation label to which the turn belongs. The first text network uses the prompt text as auxiliary information to annotate each turn in the dialogue text, that is, according to the content of the prompt text and the dialogue text, the annotation label to which each turn belongs can be identified and annotated accordingly.

[0124] In this annotation method, the classification conditions provide clear definitions and judgment criteria for preset annotation labels, ensuring the consistency and accuracy of annotation. Prompt text is constructed based on the classification conditions as auxiliary information, which can guide the first text network to use contextual information more efficiently, reduce dependence on additional annotation resources, more accurately identify the discourse function type of the turn, reduce mislabeling and omissions, realize automatic annotation, and improve the accuracy and efficiency of annotation.

[0125] Taking the example that the first text network includes a pre-trained classification model, the step of using the first text network to annotate each turn in the dialogue text can also be achieved in the following way: input the text content in the turn into the first text network to obtain the annotation result output by the first text network; wherein, the output layer of the first text network is provided with multiple classification labels, and the classification labels correspond one-to-one to the discourse function type of the turn in the dialogue context; the annotation result is obtained according to the classification label annotation that is greater than a set threshold in the probability distribution of the multiple classification labels generated by the first text network.

[0126] Wherein, in the case where the first text network includes a classification model, the training samples used in the pre-training process of the first text network include multiple sample turns, and each sample turn has a known annotation representing the discourse function type; in the pre-training process, the sample turn is input into the first text network to obtain a predicted label output by the first text network, the predicted label and the known annotation corresponding to the sample turn are calculated to perform loss calculation and minimize the loss, the first text network is iteratively trained until the training convergence condition is reached, and the trained first text network is obtained, which can use the knowledge learned in the training process to understand and analyze the input turn, and complete the labeling task based on the turn function type.

[0127] In this annotation method, each turn in the dialogue text is automatically annotated by using a pre-trained first text network, which saves labor costs and improves annotation efficiency.

[0128] In some embodiments, based on the present application, the conversation text obtained by converting the conversation audio received in real time is accurately annotated to obtain an annotation result sequence. The conversation text will be continuously updated as the conversation progresses. Therefore, the process of annotating the turn in the conversation text and obtaining the annotation result sequence and then performing recognition according to the sequence detection rule is real-time. In order to avoid the consumption of computing resources and redundant recognition results caused by frequent turn annotation and sequence recognition, the corresponding annotation result sequence can be obtained for the annotated text to be analyzed according to a fixed period, and the sequence recognition result of the annotation result sequence can be obtained according to the sequence detection rule.

[0129] The annotated text to be analyzed can be determined in any of the following ways: the annotated text of the conversation before the current time and within a preset time (such as 5 minutes) can be used as the annotated text to be analyzed, and the current time refers to the time when the annotation result sequence is obtained. This method can ensure that the conversation analysis takes into account the conversation content in the recent period of time, which is convenient for identifying the ongoing conversation type or trend. Alternatively, when the most recent detection in the historical detection exists and the annotation result sequence meets the sequence detection rules of any preset conversation type, all the annotated texts after the annotation result sequence that meets the sequence detection rules can be used as the text to be analyzed. This method can accurately locate the node where the conversation type changes, and use the conversation text after the node as the text range to be analyzed, thereby reducing redundant analysis and improving processing efficiency.

[0130] During the conversation, the conversation text received in real time is in a state of dynamic updating. Since the conversation has not yet ended, the conversation text has the characteristic of incomplete information. Based on this, the annotation result sequence obtained after the turns in the conversation text are annotated may not fully comply with the sequence detection rules of any preset conversation type. That is, before the conversation text reaches a complete state, it is impossible to accurately determine the conversation type to which the annotation result sequence belongs. Therefore, regarding the aforementioned step S104, the sequence recognition result of the annotation result sequence is determined according to the sequence detection rules of the preset conversation type, such as Figure 2 As shown, the sequence recognition result can be determined at least in the following ways:

[0131] S201, comparing the annotation result sequence corresponding to the annotated text with the sequence detection rules of each preset dialogue type respectively;

[0132] That is, for the pre-defined sequence detection rules for each dialogue type, the annotation result sequence is compared with all the conditions included in the sequence detection rule. If the annotation result sequence meets all the conditions, it is determined that the annotation result sequence meets the sequence detection rule; otherwise, if the annotation result sequence does not meet at least one condition in the sequence detection rule, it is determined that the annotation result sequence does not meet the sequence detection rule, and the annotation result sequence will continue to be compared with other sequence detection rules one by one.

[0133] For example, the conversation types include A, B, and C, and the corresponding sequence detection rules are Z1, Z2, and Z3 respectively. Assume that Z1 includes: [the following three conditions are met in the same topic: at least one of invitation for elaboration ELI, invitation for reasoning REI, and invitation for coordination CI; at least one of elaboration EL and explanation of reasons RE; and raising questions Q]. Then, when the annotation result sequence meets these three conditions at the same time, it is determined that the annotation result sequence meets the sequence detection rule Z1 of conversation type A.

[0134] S202, when the annotation result sequence satisfies any of the sequence detection rules, determining a preset dialogue type corresponding to the satisfied sequence detection rule as the dialogue type to which the annotation result sequence belongs, and generating a first recognition result;

[0135] For the sequence detection rules of multiple pre-set dialogue types, when the marked result sequence satisfies any one of the sequence detection rules, the dialogue type to which the sequence detection rule satisfied by the marked result sequence belongs can be used as the dialogue type to which the marked result sequence belongs, and then the first recognition result can be generated according to the dialogue type to which the marked result sequence belongs. In addition, when it is determined that the marked result sequence satisfies the sequence detection rule, the comparison of the marked result sequence with the sequence detection rules of other dialogue types can be stopped to avoid wasting resources.

[0136] The first recognition result may include the conversation type or the identifier of the conversation type to which the marked result sequence belongs, and may also include other additional information, such as timestamp information of the start and end of the conversation corresponding to the marked result sequence, and the conversation target corresponding to the conversation type to which the marked result sequence belongs.

[0137] S203, when the annotation result sequence does not satisfy the sequence detection rules of all preset dialogue types, determine that the dialogue type to which it belongs indicates an unknown dialogue type, and determine the rule information that the annotation result sequence does not satisfy for each sequence detection rule, and generate a second recognition result; wherein the second recognition result may at least include the rule information that the annotation result sequence does not satisfy.

[0138] When the annotation result sequence cannot meet the sequence detection rules of any pre-set dialogue type, it can be determined that the dialogue type to which the annotation result sequence belongs is an unknown dialogue type, which is used to identify the dialogue type that cannot be classified, indicating that the current annotation result sequence has characteristics or patterns that are different from any preset dialogue type.

[0139] At the same time, in order to deeply analyze why the annotation result sequence cannot match any preset dialogue type, for each preset dialogue type sequence detection rule, the rule information that the annotation result sequence does not meet can also be determined, that is, the specific reason why the annotation result sequence does not meet the sequence detection rule is clearly defined. For example, for the sequence detection rule Z1 corresponding to the aforementioned dialogue type A, the annotation result sequence does not meet the condition "including raising a question Q" in Z1, then the rule information that the annotation result sequence does not meet is "Dialogue type A: does not meet the condition including raising a question Q", which is used to clearly identify the specific conditions that the annotation result sequence fails to meet under the sequence detection rule Z1 of dialogue type A, and provide data support for the subsequent diagnosis and generation of feedback information of the second text network.

[0140] Based on the determined unknown dialogue type and the rule information not satisfied by the annotation result sequence under each dialogue type, a second recognition result is constructed. The second recognition result cooperates with the first recognition result to fully cover and process the annotation result sequence that cannot be classified into the preset dialogue type and the annotation result sequence classified into the known dialogue type, and the subsequent diagnosis of the annotation text according to the sequence recognition result to generate feedback information increases the comprehensiveness and pertinence of the feedback information, and realizes a comprehensive understanding of the dialogue content and optimization suggestions.

[0141] In the disclosed embodiment, the conversation type to which the annotation result sequence belongs is accurately determined by using a sequence detection rule of a preset conversation type. When the annotation result sequence satisfies a certain sequence detection rule, the preset conversation type to which it belongs is determined and a first recognition result is generated. For the annotation result sequence that does not satisfy all preset conversation type sequence detection rules, it is determined that it belongs to an unknown conversation type, and the rule information that is not satisfied is determined for each sequence detection rule to generate a second recognition result. The sequence recognition result generated in the above manner provides a detailed diagnostic basis for the second text network to diagnose the annotated text, which facilitates the second text network to deeply analyze the characteristics and patterns of the annotation result sequence, thereby improving the pertinence and accuracy of the generated feedback information.

[0142] Based on the above embodiment, the sequence recognition result includes a first recognition result and a second recognition result. In some embodiments, when the sequence recognition result is the first recognition result, for the aforementioned step S105, according to the sequence recognition result, the second text network is used to diagnose the annotated text, such as Figure 3The second text network diagnosis step flow chart shown can implement the diagnosis of the labeling result sequence belonging to the preset dialogue type through the following steps:

[0143] S301, when the sequence recognition result is the first recognition result, obtaining a dialogue target of a preset dialogue type corresponding to the satisfied sequence detection rule;

[0144] That is, based on the conversation type to which the annotation result sequence belongs, which is included in the generated first recognition result, it is known that the conversation type refers to the conversation type to which the sequence detection rule satisfied by the annotation result sequence belongs, that is, it is classified as a preset conversation type. Therefore, the conversation target of the conversation type to which the annotation result belongs can be determined based on the correspondence between the preset conversation type and the conversation target.

[0145] S302: Diagnose the annotated text using the second text network according to the dialogue goal.

[0146] The dialogue goal can be used as auxiliary information and introduced into the diagnosis process of the second text network to guide the second text network to diagnose the annotated text, so that the second text network can evaluate whether the content, structure, grammar and other aspects of the annotated text meet expectations based on the dialogue goal, and deeply analyze possible problems in the annotated text, thereby generating suggestions on adjusting the dialogue strategy to better meet the dialogue goal, or other information as feedback information generated by the second text network after diagnosis.

[0147] For example, taking the dialogue type - critical inquiry in a classroom teaching scenario as an example, when the annotation result sequence belongs to the critical inquiry type dialogue, according to its dialogue goal - cultivating students' critical thinking, the dialogue goal can be introduced into the second text network in the form of prompt text or other forms. The diagnosis of the annotation text by the second text network can include but is not limited to checking whether the key elements of critical inquiry are fully reflected in the dialogue, whether the dialogue has sufficient expression of discourse functions such as questioning and reasoning, etc.

[0148] In an embodiment of the present disclosure, when the annotation result sequence is classified into a preset dialogue type, the dialogue target of the dialogue type to which the annotation result sequence belongs is introduced as auxiliary information into the diagnosis process of the second text network, so that the diagnosis is more targeted, thereby generating dialogue adjustment suggestions or information that are more in line with the dialogue target, so that the dialogue is more in line with the user's expectations and needs.

[0149] In the case where the sequence recognition result is the second recognition result, this embodiment provides a diagnosis method for indicating an unknown dialogue type for the dialogue type to which the marked result sequence belongs, that is, according to the sequence recognition result described in the aforementioned step S105, the second text network is used to diagnose the marked text, such as Figure 4 The second text network diagnosis step flow chart shown in the figure can be implemented in the following manner when diagnosing the annotated text based on the second recognition result:

[0150] S401, when the sequence recognition result is the second recognition result, obtaining the rule information included in the second recognition result;

[0151] S402: According to the rule information and the unknown dialog type indicated by the dialog type, the annotated text is diagnosed using the second text network.

[0152] That is, for the case where the annotation result sequence does not belong to the preset dialogue type, the rule information that is not satisfied by the annotation result sequence is determined for each sequence detection rule when generating the second recognition result. Based on this, in the process of diagnosing the annotated text according to the second recognition result, the rule information that is not satisfied by the annotation result sequence can be introduced into the second text network as prompt information, so that the second text network can give targeted dialogue adjustment suggestions as feedback information based on the prompt information.

[0153] In the process of diagnosing the annotated text using the second text network according to the unknown dialogue type indicated by the rule information and the dialogue type to which it belongs, it is also possible to first determine the degree of match between the annotated text and each dialogue target in the dialogue context to which the dialogue audio belongs through a large language network by performing content analysis based on the text content of the annotated text and the annotation result sequence; next, the dialogue target that meets the highest degree of match, or meets the highest degree of match and is greater than a set threshold, is taken as the designated target, and based on the rule information that is not met by the annotation result sequence under the preset dialogue type corresponding to the designated target, a prompt text is generated as auxiliary information to guide the second text network to perform diagnosis, and the feedback information finally generated can specifically prompt the dialogue leader to adjust in the direction of the preset dialogue type corresponding to the designated target.

[0154] For example, taking the preset dialogue types - critical inquiry, collaborative knowledge construction, teaching and guiding dialogue as an example, each preset dialogue type corresponds to a dialogue goal. When the annotation result sequence corresponding to the annotated text does not meet the sequence detection rules of the three preset dialogue types, a second recognition result is generated. When the annotated text is diagnosed using the second text network, the text content of the annotated text and the annotation result sequence can be used to make an in-depth text analysis to predict the dialogue goal that the annotated text best meets. For example, if the annotated text best matches the dialogue goal of cultivating critical thinking, the rule information that is not met by the annotation result sequence under the sequence detection rules of the critical inquiry dialogue can be used to generate prompt information based on the rule information and combine it with the annotated text as the input of the second text network to facilitate the generation of diagnosis and feedback information of the second text network.

[0155] In the embodiment of the present disclosure, rule information that is not satisfied by the annotation result sequence determined for each sequence detection rule is introduced as auxiliary information into the second text network, and the second text network is used to perform diagnosis in combination with the rule information, thereby providing a more fine-grained diagnosis method, so that the diagnosis results are more in line with user expectations and the generated feedback information is more accurate.

[0156] In some embodiments, the aforementioned step S105 of diagnosing the annotated text using a second text network to generate feedback information based on the sequence recognition result can also be achieved in the following manner: determining a target diagnosis method that matches the sequence recognition result based on a pre-constructed mapping relationship; the mapping relationship includes different recognition results and their corresponding diagnosis methods; the different recognition results include a result indicating that the annotated result sequence belongs to one of the preset dialogue types, and a result indicating that the annotated result sequence does not belong to any preset dialogue type; through the machine learning, the annotated text is sequence diagnosed according to the target diagnosis method.

[0157] In the disclosed embodiment, by pre-building a mapping relationship, different recognition results are associated with their corresponding diagnostic methods, so that flexible configuration of the diagnostic method is achieved. When a new dialogue type needs to be added or the diagnostic logic of an existing dialogue type needs to be modified, only the mapping relationship needs to be updated, thereby improving scalability. At the same time, by selecting the corresponding target diagnostic method according to the recognition result, the diagnostic process can be more targeted, unnecessary calculations and resource consumption can be reduced, and the accuracy of the generation of diagnosis and feedback information can be improved.

[0158] In some embodiments, in order to further improve the meticulousness and practicality of the diagnosis, this embodiment introduces an innovative diagnostic dimension, namely, the level of engagement of the conversational person, to enrich the diagnostic dimension and method of the second text network for the annotated text. Based on this diagnostic dimension, the aforementioned conversation analysis method may also include the following steps of determining detection parameters, namely, obtaining video data collected simultaneously with the conversation audio; aligning the video data with the annotated text according to timestamps; determining the level of engagement of the conversational person based on the video data and the annotated text; determining the detection parameters that match the level of engagement of the conversational person based on a preset correspondence; the correspondence includes a preset level of engagement, and the detection parameters corresponding to each level of engagement.

[0159] That is, while processing the conversation audio, video data synchronized with the audio can be collected through video acquisition devices such as cameras, and the consistency and accuracy of the analysis can be ensured by accurately aligning the video data with the annotated text according to the timestamp information. Based on this, video analysis technology and natural language processing technology can be used to comprehensively evaluate the behavior, expression and speech content of the interlocutors, and evaluate the involvement of the interlocutors in the conversation context to which the conversation audio belongs, so as to determine the involvement level of the interlocutors. Next, based on the pre-set corresponding relationship, the detection parameters corresponding to each involvement level are clarified. The detection parameters are used to capture key information related to the involvement of the interlocutors, and are used to analyze the reasons for the involvement level of the interlocutors. Different detection parameters are set for different involvement levels of the interlocutors, indicating different annotated text diagnostic focuses.

[0160] For example, taking the classroom teaching scenario as an example, the participation level of classroom dialogue participants can be divided into three levels: low participation, medium participation and high participation. Taking the low participation level as an example, the set detection parameters may include but are not limited to whether the teacher explains for a long time, the students' silence mode and silence duration, whether the teacher asks questions too frequently, etc.

[0161] Based on the detection parameters matched with the engagement level of the conversation personnel determined in the above manner, for the aforementioned step S105 of diagnosing the annotated text using the second text network according to the sequence recognition result, prompt information can be generated based on the sequence recognition result and the detection parameters, and when the annotated text is diagnosed using the second text network, the second text network is guided by the prompt information.

[0162] In the disclosed embodiment, by combining video data and annotated text, the information in the conversation process is comprehensively captured, the participation level of the conversation parties is evaluated, and based on the participation level, detection parameters matching the participation level are introduced as the diagnostic dimension for the second text network to diagnose the annotated text, thereby improving the accuracy of the diagnosis results and generating feedback information that better meets user needs.

[0163] In some embodiments, after the feedback information is generated in the aforementioned step S105, the method may further include the following display step, through which the generated feedback information may be converted into a visual page for the user to view, so that the user can quickly grasp the key information of the conversation and make corresponding adjustments, thereby improving the user experience and interaction efficiency. The display step may be implemented in the following manner: outputting the feedback information in a structured mode; the feedback information may include at least one of the following: conversation participation status, interaction prompts, and conversation direction suggestions; according to a preset page layout, the feedback information is converted into a virtual page and projected and displayed in the user's field of vision.

[0164] The structured mode output is used to represent the feedback information organized in a clear and orderly manner, and can be represented by a table, list or other forms of structured data. In this embodiment, the feedback information is used to guide the suggestions and information for the progress or improvement of the dialogue, the dialogue participation situation can at least be used to describe the activity level and contribution of each participant in the dialogue, and the interactive prompts can include guiding prompts or suggestions for enhancing the interactivity of the dialogue. The dialogue direction suggestion is used to indicate the dialogue direction prompt for the continuation of the dialogue, such as asking more questions and interacting with Topic A.

[0165] The preset page layout refers to the design layout of the page or interface used to display information in a virtual environment, and the page layout can be customized according to user needs and system characteristics. Virtual page projection display refers to the technology of projecting information in the form of a page into the user's field of vision using virtual reality, augmented reality (AR) or other display technologies. After obtaining the structured feedback data, the feedback data can be filled into the corresponding position of the virtual page to generate a complete virtual page, which is projected and displayed in the user's field of vision.

[0166] When displaying feedback information, the feedback information can be presented in simple icons and texts to enhance the readability of the information. Different colors or icons can also be used to indicate different feedback status, participation levels, and interaction modes, so as to provide conversation feedback without interfering with the conversation rhythm of the conversation leader.

[0167] In the disclosed embodiments, in order to improve user experience and interaction efficiency, after feedback information is generated based on the real-time received conversation audio analysis, the feedback information is converted into a virtual page according to a preset page layout and instantly projected and displayed within the user's field of vision, providing an intuitive and easy-to-read information display platform, reducing the difficulty for users to understand information, and enabling users to receive feedback information in real time while the conversation is ongoing and quickly adjust conversation strategies based on the feedback information to improve conversation quality.

[0168] Based on the conversation analysis method provided in the aforementioned embodiment, the present application also provides a pair of glasses that can display feedback information in real time, can collect conversation audio in real time, and generate feedback information for the conversation audio through the aforementioned conversation analysis method. The glasses are used to display the feedback information in the user's field of vision, so that the user can quickly grasp the conversation situation and make adjustments to the conversation.

[0169] like Figure 5 The schematic diagram of the glasses structure is shown as an example, and the glasses may at least include an audio acquisition module 501, a power management module 502, a data processing module 503, a wireless transmission module 504 and a display module 505. It should be understood that the module positions and module settings in the figure are only exemplary displays, and this application does not limit this. For the glasses structure in the figure, the audio acquisition module is used to collect conversation audio in real time; the power management module is used to provide the power required for the operation of the glasses; the wireless transmission module is used to transmit the conversation audio collected by the audio acquisition module to the data processing module in real time; the data processing module is used to generate feedback information for the conversation audio received in real time using the conversation analysis method provided in the above embodiment, and send the feedback information to the display module through the wireless transmission module; the display module is used to project the feedback information within the user's field of vision based on augmented reality technology, and the display template can be embedded in the glasses lens.

[0170] The audio acquisition module may include at least one of a built-in camera and a microphone, wherein the camera is used to capture the conversation environment in real time, and the microphone is used to collect the conversation sound in real time. The audio acquisition module may also include a preprocessing module for performing preliminary processing on the collected audio and video data.

[0171] The display template may include a head-up display, which is respectively embedded in two lenses of the glasses, and the head-up display is used to display the feedback information within the user's field of vision through transparent projection technology.

[0172] Regarding the data processing module using the conversation analysis method provided in the aforementioned embodiment to generate feedback information, when the computing resources of the data processing module itself support running the conversation analysis method, the data processing module runs the conversation analysis method to process the conversation audio received in real time, thereby generating feedback information; when the conversation audio received in real time is processed by a server located at the remote end or other equipment or devices that support running the conversation analysis method, a communication connection between the data processing module and the server located at the remote end or other equipment or devices that support running the conversation analysis method can be established in advance, and the data processing module receives the returned feedback information and transmits it to the display module.

[0173] In the disclosed embodiment, by integrating audio acquisition, power management, data processing, wireless transmission and augmented reality display technology into a pair of glasses, a highly integrated, real-time interactive conversation feedback system and augmented reality glasses are provided to users. The built-in audio acquisition module of the glasses can capture the conversation audio in real time and quickly transmit it to the data processing module for analysis through the wireless transmission module. The real-time processing capability ensures that the user can obtain the feedback information of the conversation immediately without waiting or relying on external devices; the display module embedded in the glasses lens realizes the instant display of feedback information, so that the user can directly obtain key information within the field of vision without being distracted to check other devices, which greatly improves the convenience of use. In addition, using augmented reality technology, the display module can project the feedback information in the form of a virtual screen within the user's field of vision, reducing the risk of sensitive information leakage. The display of feedback information can be seamlessly integrated with the user's real environment. The immersive visual experience not only improves the intuitiveness of the information, but also enhances the user's sense of participation and interactivity. Since all functions are integrated in the glasses, users can obtain the required feedback information without holding a handheld device or performing complex operations, which brings convenience to user operation.

[0174] Corresponding to the embodiment of the above-mentioned conversation analysis method, see Figure 6 As shown, the present application also provides an embodiment of a conversation analysis device, the device comprising:

[0175] An audio conversion module 601 is used to convert the conversation audio received in real time into a conversation text;

[0176] A tagging module 602 is used to tag each turn in the dialogue text using a first text network to obtain a tagged text with a tagging result; the tagging result is used to indicate the discourse function type of the turn;

[0177] A sequence acquisition module 603, configured to obtain a sequence of annotation results according to the turn order and the annotation results in the annotated text;

[0178] A sequence recognition module 604 is used to determine a sequence recognition result of the annotation result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes the dialogue type to which the annotation result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs;

[0179] The feedback information generating module 605 is used to diagnose the annotated text using the second text network according to the sequence recognition result to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue status to optimize the dialogue.

[0180] In some embodiments, the annotation module is specifically used to:

[0181] Obtain classification conditions for each preset annotation label; the preset annotation label corresponds one-to-one to the discourse function type of the turn in the dialogue context; construct a prompt text suitable for the prompt project according to the classification conditions; use the first text network and the prompt text to identify the annotation label belonging to each turn in the dialogue text and annotate it.

[0182] In some embodiments, the annotation module is specifically used to:

[0183] The text content in the turn is input into the first text network to obtain the annotation result output by the first text network; wherein, the output layer of the first text network is provided with a plurality of classification labels, and the classification labels correspond one-to-one to the discourse function types of the turn in the dialogue context; the annotation result is obtained according to the classification labels greater than a set threshold in the probability distribution of the plurality of classification labels generated by the first text network.

[0184] In some embodiments, the sequence recognition module is specifically used to:

[0185] In the case where the marked result sequence satisfies any of the sequence detection rules, determining a preset dialogue type corresponding to the satisfied sequence detection rule as the dialogue type to which the marked result sequence belongs, and generating a first recognition result;

[0186] When the marked result sequence does not satisfy the sequence detection rules of all preset dialogue types, it is determined that the dialogue type indicates an unknown dialogue type, and the rule information that the marked result sequence does not satisfy is determined for each sequence detection rule to generate a second recognition result.

[0187] In some embodiments, the feedback information generating module is specifically used to:

[0188] When the sequence recognition result is the first recognition result, a dialogue target of a preset dialogue type corresponding to the satisfied sequence detection rule is obtained; and according to the dialogue target, the annotated text is diagnosed using the second text network.

[0189] In some embodiments, the feedback information generating module is specifically used to:

[0190] In the case where the sequence recognition result is the second recognition result, the rule information included in the second recognition result is obtained; and according to the rule information and the unknown dialogue type indicating the corresponding dialogue type, the annotated text is diagnosed using the second text network.

[0191] In some embodiments, the apparatus further comprises:

[0192] Acquire video data collected simultaneously with the conversation audio; align the video data with the annotated text according to timestamps; determine the level of participation of the conversation person according to the video data and the annotated text; determine detection parameters matching the level of participation of the conversation person according to a preset corresponding relationship; the corresponding relationship includes a preset level of participation and detection parameters corresponding to each level of participation;

[0193] The feedback information generation module is specifically used to: diagnose the annotated text using the second text network according to the sequence recognition result and the detection parameter.

[0194] In some embodiments, the apparatus further comprises:

[0195] The feedback information is output in a structured mode; the feedback information may include at least one of dialogue participation status, interaction prompts, and dialogue direction suggestions; according to a preset page layout, the feedback information is converted into a virtual page and projected and displayed in the user's field of vision.

[0196] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0197] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work.

[0198] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 7 As shown, the electronic device 700 includes at least one processor 701, a memory 702 and a bus 703, and the at least one processor 701 is electrically connected to the memory 702; the memory 702 is configured to store at least one computer-executable instruction, and the processor 701 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any conversation analysis method provided in any embodiment or any optional implementation manner in the present application.

[0199] Furthermore, the processor 701 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0200] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any conversation analysis method provided in any embodiment or any optional implementation manner in the present application when executed by a processor.

[0201] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0202] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0203] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A conversation analysis method, characterized in that: The method comprises: Converting the conversation audio received in real time into conversation text; Using a first text network to annotate each turn in the dialogue text, obtaining an annotated text with annotated results; the annotated results are used to indicate the discourse function type of the turn; Obtaining a sequence of annotation results according to the turn order and the annotation results in the annotated text; Determine a sequence recognition result of the labeled result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes a dialogue type to which the labeled result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs; According to the sequence recognition result, the annotated text is diagnosed using a second text network to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue situation to optimize the dialogue.

2. The method according to claim 1, characterized in that The step of using the first text network to mark each turn in the dialogue text includes: Acquire classification conditions of each preset annotation label; the preset annotation label corresponds to the speech function type of the turn in the dialogue context one by one; Constructing a prompt text suitable for the prompt project according to the classification conditions; The first text network and the prompt text are used to identify and annotate the annotation tags of each turn in the dialogue text.

3. The method according to claim 1, characterized in that The step of using the first text network to mark each turn in the dialogue text includes: Inputting the text content in the turn into the first text network to obtain the annotation result output by the first text network; Among them, the output layer of the first text network is set with multiple classification labels, and the classification labels correspond one-to-one to the discourse function types of the turns in the dialogue context; the labeling results are obtained according to the classification labels greater than the set threshold in the probability distribution of the multiple classification labels generated by the first text network.

4. The method according to claim 1, characterized in that The determining the sequence recognition result of the labeled result sequence according to the sequence detection rule of the preset dialogue type includes: In the case where the marked result sequence satisfies any of the sequence detection rules, determining a preset dialogue type corresponding to the satisfied sequence detection rule as the dialogue type to which the marked result sequence belongs, and generating a first recognition result; When the marked result sequence does not satisfy the sequence detection rules of all preset dialogue types, it is determined that the dialogue type indicates an unknown dialogue type, and the rule information that the marked result sequence does not satisfy is determined for each sequence detection rule to generate a second recognition result.

5. The method according to claim 4, characterized in that The step of diagnosing the annotated text using a second text network according to the sequence recognition result includes: When the sequence recognition result is the first recognition result, obtaining a dialogue target of a preset dialogue type corresponding to the satisfied sequence detection rule; According to the dialogue goal, the annotated text is diagnosed using the second text network.

6. The method according to claim 4, characterized in that The step of diagnosing the annotated text using a second text network according to the sequence recognition result includes: When the sequence recognition result is the second recognition result, acquiring the rule information included in the second recognition result; The second text network is used to diagnose the annotated text according to the rule information and the unknown dialog type indicated by the dialog type.

7. The method according to claim 1, characterized in that The method further comprises: Acquire video data collected simultaneously with the conversation audio; Aligning the video data with the annotated text according to timestamps; Determining a level of participation of a conversational person based on the video data and the annotated text; Determine the detection parameters that match the engagement level of the dialogue personnel according to the preset corresponding relationship; the corresponding relationship includes the preset engagement level and the detection parameters corresponding to each engagement level; The step of diagnosing the annotated text using a second text network according to the sequence recognition result includes: The annotated text is diagnosed using the second text network according to the sequence recognition result and the detection parameter.

8. The method according to claim 1, characterized in that The method further comprises: Outputting the feedback information in a structured mode; the feedback information may include at least one of dialogue participation status, interaction prompts, and dialogue direction suggestions; According to a preset page layout, the feedback information is converted into a virtual page and projected and displayed in the user's field of vision.

9. A pair of glasses, characterized in that: The glasses at least include an audio acquisition module, a power management module, a data processing module, a wireless transmission module and a display module; The audio collection module is used to collect conversation audio in real time; The power management module is used to provide the power required for the glasses to operate; The wireless transmission module is used to transmit the conversation audio collected by the audio collection module to the data processing module in real time; The data processing module is used to generate feedback information for the conversation audio received in real time by using the conversation analysis method according to any one of claims 1 to 8, and send the feedback information to the display module through the wireless transmission module; The display module is used to project the feedback information within the user's field of vision based on augmented reality technology.

10. A conversation analysis device, characterized in that: The device comprises: An audio conversion module, used for converting the conversation audio received in real time into conversation text; a marking module, configured to mark each turn in the dialogue text using a first text network to obtain a marked text with marking results; the marking results are used to indicate the discourse function type of the turn; A sequence acquisition module, used for obtaining a sequence of annotation results according to the turn order and the annotation results in the annotated text; A sequence recognition module, used to determine a sequence recognition result of the labeled result sequence according to a sequence detection rule of a preset dialogue type; the sequence recognition result includes the dialogue type to which the labeled result sequence belongs; the preset dialogue type is determined according to different dialogue goals in the dialogue context to which the dialogue audio belongs; A feedback information generating module is used to diagnose the annotated text using a second text network according to the sequence recognition result to generate feedback information; the feedback information is at least used to prompt the dialogue leader in real time about the dialogue situation to optimize the dialogue.

11. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-8.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images