Personality prediction method and device based on voice conversation, electronic equipment and medium

Abstract

The invention provides a personality prediction method and device based on voice conversation, electronic equipment and a medium. The method comprises the following steps: acquiring conversation voice information between a to-be-predicted user and an intelligent agent; extracting a target dialogue feature and a dialogue log of the to-be-predicted user from the dialogue voice information; according to the dialogue log and the target dialogue feature, updating the dialogue log and dialogue summary information of the to-be-predicted user; based on the fused target dialogue features, the updated dialogue log and the dialogue summary information, obtaining at least one prediction label of the to-be-predicted user; and inputting the user standardized text description information obtained based on the at least one prediction label fusion into a pre-trained large language model, so that the large language model outputs a personality prediction result of the user to be predicted. Therefore, the accuracy of personality prediction of the to-be-predicted user can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to personality prediction methods, devices, electronic devices, and media based on voice dialogue. Background Technology

[0002] An intelligent agent is a proxy capable of perceiving its environment and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. Specifically, intelligent agents can be applied in fields such as automated systems, robots, virtual assistants, and game characters. Users can obtain corresponding dialogue information through interaction with the intelligent agent. During dialogue interaction between the user and the intelligent agent, the user's personality can be predicted, thereby better assisting the intelligent agent in subsequent dialogue and communication.

[0003] In related technologies, when predicting the personality of a user based on dialogue voice information, the analysis is performed by converting the user's dialogue voice information into text feature information. However, the information contained in the text feature information corresponding to the dialogue voice information is insufficient to cover all the dynamic information of the user, resulting in low accuracy in predicting the user's personality. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a personality prediction method, device, electronic device and medium based on voice dialogue. After obtaining the voice dialogue information between the user to be predicted and the intelligent agent, the target dialogue features of the user to be predicted can be extracted based on the voice dialogue information and historical dialogue extraction information. The target dialogue features include not only text dialogue features, but also acoustic dialogue features and behavioral features. In this way, the prediction label of the user to be predicted can be determined from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps to improve the accuracy of personality prediction for the user to be predicted.

[0005] In a first aspect, embodiments of this application provide a personality prediction method based on voice dialogue, the personality prediction method comprising: Acquire the voice dialogue information between the user to be predicted and the intelligent agent; Extracting target dialogue features of the user to be predicted from the dialogue voice information, and extracting dialogue logs from the dialogue voice information; wherein, the target dialogue features include text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained. The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

[0006] Secondly, embodiments of this application also provide a personality prediction device based on voice dialogue, the personality prediction device comprising: The dialogue voice information acquisition module is used to acquire the dialogue voice information between the user to be predicted and the intelligent agent; The dialogue feature extraction module is used to extract the target dialogue features of the user to be predicted from the dialogue voice information, and to extract the dialogue log from the dialogue voice information; wherein, the target dialogue features include the text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; The information update module is used to update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features. The prediction label determination module is used to obtain at least one prediction label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information. The personality prediction module is used to input standardized text description information of users obtained by fusing the at least one prediction label into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted.

[0007] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the personality prediction method based on voice dialogue as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the personality prediction method based on voice dialogue as described in the first aspect.

[0009] The personality prediction method, apparatus, electronic device, and medium based on voice dialogue provided in this application acquire voice dialogue information between a user to be predicted and an intelligent agent; extract target dialogue features of the user to be predicted from the voice dialogue information; and extract dialogue logs from the voice dialogue information. The target dialogue features include textual dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted. The user's dialogue logs and dialogue summary information are updated based on the dialogue logs and the target dialogue features. At least one predicted label for the user to be predicted is obtained based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information. The user's standardized text description information obtained based on the fusion of at least one predicted label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result for the user to be predicted. Thus, after acquiring the voice dialogue information between the user to be predicted and the intelligent agent, the target dialogue features of the user to be predicted can be extracted based on the voice dialogue information and historical dialogue extraction information. These target dialogue features include not only textual dialogue features but also acoustic dialogue features and behavioral features. This allows for the determination of the predicted label for the user to be predicted from multiple dimensions, thereby determining the personality prediction result and improving the accuracy of personality prediction for the user to be predicted.

[0010] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart illustrating a personality prediction method based on voice dialogue provided in an embodiment of this application; Figure 2 A flowchart illustrating the extraction of target dialogue features provided in an embodiment of this application; Figure 3 This is a flowchart of user personality prediction provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a personality prediction device based on voice dialogue provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0014] First, the technical background of this disclosure will be introduced: An intelligent agent is a proxy capable of perceiving its environment and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. Specifically, intelligent agents can be applied in fields such as automated systems, robots, virtual assistants, and game characters. Users can obtain corresponding dialogue information through interaction with the intelligent agent. During dialogue interaction between the user and the intelligent agent, the user's personality can be predicted, thereby better assisting the intelligent agent in subsequent dialogue and communication.

[0015] In related technologies, when predicting the personality of a user based on dialogue voice information, the analysis is performed by converting the user's dialogue voice information into text feature information. However, the information contained in the text feature information corresponding to the dialogue voice information is insufficient to cover all the dynamic information of the user, resulting in low accuracy in predicting the user's personality.

[0016] Based on this, embodiments of this application provide a personality prediction method based on voice dialogue to improve the accuracy of personality prediction for users to be predicted.

[0017] Please see Figure 1 , Figure 1 A flowchart illustrating a personality prediction method based on voice dialogue provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the personality prediction method based on voice dialogue includes: S101. Obtain the voice dialogue information between the user to be predicted and the intelligent agent.

[0018] S102. Extract the target dialogue features of the user to be predicted from the dialogue voice information, and extract the dialogue log from the dialogue voice information; wherein, the target dialogue features include the text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted.

[0019] S103. Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features.

[0020] S104. Based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained.

[0021] S105. Input the user-standardized text description information obtained by fusing the at least one prediction label into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

[0022] The personality prediction method based on voice dialogue provided in this disclosure, after obtaining the voice dialogue information between the user to be predicted and the intelligent agent, can extract the target dialogue features of the user to be predicted based on the voice dialogue information and historical dialogue extraction information. The target dialogue features include not only text dialogue features, but also acoustic dialogue features and behavioral features. In this way, the prediction label of the user to be predicted can be determined from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps to improve the accuracy of personality prediction for the user to be predicted.

[0023] The exemplary steps of the embodiments of this disclosure are described below: S101. Obtain the voice dialogue information between the user to be predicted and the intelligent agent.

[0024] In this application embodiment, an intelligent agent refers to a proxy that can perceive the environment and take actions to achieve a specific goal. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. Specifically, intelligent agents can be applied in fields such as automated systems, robots, virtual assistants, and game characters. Users can obtain corresponding dialogue information through interaction with the intelligent agent.

[0025] In one possible implementation, the user can interact with the intelligent agent through text and voice. In this embodiment, when predicting the personality of the user to be predicted, the voice information of the dialogue between the user to be detected and the intelligent agent is collected.

[0026] Here, personality prediction for the user to be predicted can be done using the Big Five personality model. The Big Five personality model is a psychological framework for identifying an individual's thought, feeling, and behavior patterns, which includes five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. When performing personality identification on the user to be predicted, the output prediction results are the user's rating information on each personality dimension and whether the evaluation is consistent.

[0027] In related technologies, when predicting the personality of a user based on dialogue voice information, the analysis is performed by converting the user's dialogue voice information into text feature information. However, the information contained in the text feature information corresponding to the dialogue voice information is insufficient to cover all the dynamic information of the user, resulting in low accuracy in predicting the user's personality.

[0028] In this embodiment, after obtaining the dialogue voice information between the user to be predicted and the intelligent agent, the target dialogue features of the user to be predicted can be extracted based on the dialogue voice information and historical dialogue extraction information. The target dialogue features include not only text dialogue features, but also acoustic dialogue features and behavioral features. In this way, the prediction label of the user to be predicted can be determined from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps to improve the accuracy of personality prediction for the user to be predicted.

[0029] S102, extract the target dialogue features of the user to be predicted from the dialogue voice information, and extract the dialogue log from the dialogue voice information.

[0030] In this embodiment of the application, after obtaining the dialogue voice information between the user to be predicted and the intelligent agent, the target dialogue features and dialogue logs of the user to be predicted can be extracted from the dialogue voice information, so as to predict the personality of the user to be predicted in the future by using the obtained target dialogue features and dialogue logs.

[0031] In one possible implementation, in order to extract the target dialogue features of the user to be predicted more accurately, the historical dialogue extraction information of the user to be predicted in the historical dialogue analysis process can be referenced when extracting the target dialogue features of the user to be predicted, so as to better analyze the target dialogue features of the current user to be predicted.

[0032] Specifically, the step "extracting the target dialogue features of the user to be predicted from the dialogue voice information" includes: a1: Based on the dialogue voice information and the historical dialogue extraction information of the user to be predicted, extract the target dialogue features of the user to be predicted from the dialogue voice information.

[0033] For example, historical dialogue extraction information may include historical dialogue logs, historical dialogue features, and historical dialogue summaries. When extracting acoustic dialogue features from the target dialogue features of the user to be predicted, based on the current dialogue voice information between the user to be predicted and the intelligent agent, it is determined that the user to be predicted is laughing. However, according to the historical dialogue extraction information, it can be known that in the previous dialogue, the user to be predicted and the intelligent agent were discussing objects that the user to be predicted disliked. Therefore, the laughter in the acoustic dialogue features of the user to be predicted at this time cannot determine that the user to be predicted is happy, but should be sarcastic. This can improve the accuracy of subsequent predictions.

[0034] In this embodiment of the application, after obtaining the dialogue voice information between the user to be predicted and the intelligent agent, the target dialogue features, dialogue logs and dialogue summary information of the user to be predicted can be extracted from the dialogue voice information. The extraction process of different information will be described below.

[0035] Specifically, the step "extracting the dialogue log from the dialogue voice information" includes: b1: Input the dialogue voice information into the automatic speech recognition system, convert the dialogue voice information into text dialogue information, and determine each word contained in the text dialogue information and the word association information corresponding to each word.

[0036] b2: Based on the text dialogue information and the timestamp information corresponding to each word, determine at least one response unit in the dialogue voice information.

[0037] b3: Based on the at least one response unit, each word, and the word association information corresponding to each word, the dialogue log is obtained.

[0038] In one possible implementation, the voice dialogue information between the user and the agent to be predicted can be the voice dialogue information between the user and the agent within a preset dialogue time period.

[0039] Here, the preset dialogue time period can be set according to the dialogue analysis needs. For example, the preset dialogue time period can be set to 12 minutes.

[0040] In this embodiment of the application, after obtaining the dialogue voice information, the dialogue voice information can be input into the voice recognition system for processing.

[0041] Here, Automatic Speech Recognition (ASR) aims to convert the lexical content of human speech into computer-readable input, thereby recognizing at least one word contained in conversational speech information.

[0042] Specifically, the system can input two-channel dialogue voice information between the user to be predicted and the agent into the ASR system, convert the dialogue voice information into dialogue text information, and determine at least one word and word association information for each word from the dialogue text information.

[0043] Specifically, the word association information for each word may include at least one of the following: the start timestamp of the word, the end timestamp of the word, the word content, and the speaker ID corresponding to each word.

[0044] In one possible implementation, in order to ensure the validity of the acquired information, after determining multiple timestamps, it is necessary to detect the silent time interval in each timestamp and remove the silent time contained in each timestamp to ensure that the dialogue voice information analyzed subsequently contains valid dialogue information. This can reduce the amount of data processing and improve the efficiency and accuracy of dialogue voice information processing.

[0045] Furthermore, after obtaining each word and its corresponding word association information, it is necessary to store each word and its corresponding word association information for use in subsequent dialogue logs.

[0046] In one possible implementation, after determining the timestamp information of each word, it is necessary to determine at least one response unit in the dialogue speech information based on the timestamp information of each word. When assigning corresponding labels to different dialogue features in the future, the processing is carried out on a unit-by-unit basis.

[0047] In one possible implementation, at least two consecutive words can be determined based on the timestamp information corresponding to each word in the text dialogue information, and the semantic continuity between at least two consecutive words can be determined based on the obtained dialogue text information, thereby determining the response boundary in the text dialogue information, and then determining at least one response unit in the dialogue speech information based on the response boundary.

[0048] Specifically, the step "determining at least one response unit in the dialogue voice information based on the text dialogue information and the timestamp information corresponding to each word" includes: c1: Based on the text dialogue information and the timestamp information corresponding to each word, determine the time interval and semantic continuity between at least two consecutive words.

[0049] c2: Based on the time interval between at least two consecutive words and semantic continuity, merge at least two consecutive words to obtain the response unit.

[0050] Specifically, the response boundary refers to the node where a pause occurs during the dialogue between the user to be predicted and the agent. This node is determined after identifying at least two consecutive words based on the timestamps corresponding to each word in the text dialogue information. At this point, the agent needs to respond to the user to be predicted at this node, which can be the response boundary in the identified text dialogue information, identifying at least one response unit contained in the text dialogue information.

[0051] Here, a response unit refers to a different unit that the agent needs to reply to the user to be predicted. After the user to be predicted outputs a response unit, the agent begins to reply according to the response unit output by the user to be predicted. After the user to be predicted outputs a response unit, it may need to pause for a period of time to wait for the agent's reply. Therefore, at least one response unit in the dialogue speech information can be determined based on the time interval between different words.

[0052] Specifically, the step "merging at least two consecutive words based on the time interval between at least two consecutive words and semantic continuity to obtain the response unit" includes: d1: If the time interval between at least two consecutive words is greater than the first preset time interval, merge at least two consecutive words to obtain the first merged word group.

[0053] d2: Detect whether there is semantic continuity between at least two words in the first merged word group. If there is semantic continuity between at least two words, determine the first merged word group as a response unit.

[0054] d3: If there is no semantic continuity between at least two words, when the time interval between at least two consecutive words is greater than the second preset time interval, merge at least two consecutive words to obtain a second merged word group, and determine the second merged word group as a response unit.

[0055] The first preset time interval and the second time interval can be determined based on the pause response time of different users during historical dialogues.

[0056] For example, the first preset time interval can be set to 300 milliseconds, and the second preset time interval can be set to 800 milliseconds.

[0057] In one possible implementation, if the time interval between at least two consecutive words is greater than a first preset time interval, it is determined that the user being predicted is pausing during the dialogue. At least two consecutive words are then directly merged to obtain a first merged word group. Semantic analysis can be performed on the first merged word group to determine whether there is continuity between the words within it.

[0058] Here, the first merged word group can be input into the semantic understanding model to determine whether there is continuity between the words in the first merged word group.

[0059] Furthermore, if at least two words in the first merged word group have semantic continuity, the first merged word group is determined as a response unit.

[0060] In another possible implementation, if at least two words in the first merged word group do not have semantic continuity, the time intervals corresponding to the consecutive words are further judged. If the time intervals corresponding to at least two consecutive words are greater than the second preset time interval, the at least two consecutive words are merged to obtain the second merged word group, and the second merged word group is determined as a response unit.

[0061] In one possible implementation, textual dialogue features can be extracted based on dialogue logs extracted from the speech information. For example, textual dialogue features can be obtained by performing sentiment detection, emotion detection, and feedback signal detection on the dialogue logs using a Large Language Model (LLM).

[0062] Specifically, the text dialogue features are extracted from the dialogue voice information through the following steps: e1: Input the dialogue log into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label and emotion intensity information corresponding to each response unit in the dialogue voice information.

[0063] e2: Input the dialogue log into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label corresponding to each response unit in the dialogue voice information.

[0064] e3: Input the dialogue log into a pre-trained feedback classifier so that the feedback classifier outputs the feedback label corresponding to each response unit in the dialogue voice information.

[0065] In one possible implementation, the dialogue log is input into a pre-trained emotion classifier to detect the emotion information corresponding to each response unit in the dialogue voice information and output the emotion label corresponding to each response unit.

[0066] Here, the emotion label can include at least one of anger, disgust, fear, joy, neutrality, sadness, and surprise.

[0067] Furthermore, after outputting the sentiment label corresponding to each response unit, the corresponding sentiment intensity can also be output. Here, the sentiment intensity can be different percentage values ​​output by the sentiment classifier, or the sentiment intensity can be divided into different intensity levels such as extreme, mild, and normal. At the same time, different intensity levels correspond to different percentage ranges. Based on the percentage range to which the percentage value output by the sentiment classifier belongs, the corresponding sentiment intensity is determined.

[0068] For example, after the emotion classifier determines that the emotion label corresponding to the first response unit in the current dialogue voice information is sadness, and the output percentage value is 40%, the percentage range corresponding to the normal intensity level is (30, 50), so the emotional intensity of sadness is normal at this time.

[0069] In one possible implementation, the dialogue log is input into a pre-trained emotion classifier, which detects the emotion information corresponding to each response unit in the dialogue voice information and outputs the emotion label corresponding to each response unit.

[0070] Here, the emotion label can include at least one of the following: anger, disgust, fear, joy, neutral, sadness, surprise, positive, neutral, and negative.

[0071] In one possible implementation, the dialogue log is input into a pre-trained emotion classifier, which detects the feedback information corresponding to each response unit in the dialogue voice information and outputs the feedback label corresponding to each response unit.

[0072] Here, the feedback tag can include at least one of the following: agree, feedback, no feedback.

[0073] In one possible implementation, the extraction of acoustic dialogue features from dialogue speech information can be achieved by directly extracting the dialogue speech information, detecting speech events, speech emotions, speaking style, speaker age / gender, multiple speakers, acoustic events, etc., and outputting the corresponding acoustic dialogue features.

[0074] Specifically, the acoustic dialogue features are extracted from the dialogue speech information through the following steps: f1: Input the dialogue speech information into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event.

[0075] f2: Input the dialogue speech information into a pre-trained speech emotion detection model so that the speech emotion detection model outputs the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information.

[0076] f3: Input the dialogue speech information into the pre-trained speech style detection model so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted.

[0077] f4: Input the dialogue voice information into a pre-trained speaker attribute detection model so that the speaker attribute detection model outputs the speaker attribute information corresponding to the user to be predicted.

[0078] f5: Input the dialogue voice information into the pre-trained speaker detection model so that the speaker detection model outputs a speaker log.

[0079] f6: Input the dialogue speech information into a pre-trained acoustic event detection model so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

[0080] In one possible implementation, voice event detection in dialogue voice information is the detection of specific voice events contained in the dialogue voice. Specifically, voice events may include laughter, crying, sighing, etc.

[0081] Specifically, the dialogue speech information can be input into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event.

[0082] Specifically, the speech event detection model outputs at least one speech event contained in the dialogue speech information and the timestamp of each speech event.

[0083] Furthermore, to improve the accuracy of subsequent voice event analysis, each output voice event can be finely annotated. For example, the same laughter voice event can be further subdivided into normal laughter, loud laughter, snickering laughter, awkward laughter, disdainful laughter, etc.

[0084] In one possible implementation, voice emotion detection for dialogue voice information involves detecting emotions such as happiness, sadness, and anger included in the dialogue voice information.

[0085] Specifically, the dialogue speech information is input into a pre-trained speech emotion detection model, and the output is the emotion classification label and emotion intensity information corresponding to each response unit contained in the dialogue speech information. The specific method for determining the emotion classification label and emotion intensity information of each response unit is the same as the method for determining the emotion classification label and emotion intensity information in the text acoustic features, and will not be repeated here.

[0086] In one possible implementation, the speaking style classification of the user to be predicted can be achieved by detecting the volume, speed, and tone of the user's speech, and then detecting the user's storytelling style by detecting the volume, speed, and tone of the user's speech.

[0087] Specifically, the dialogue speech information is input into a pre-trained speech style detection model and you want to give the speech style detection, so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted.

[0088] For example, the speaking style of the user to be predicted may include at least one of the following: emphasis, calm, rapid, and whisper.

[0089] In one possible implementation, the speaker attribute information of the user to be predicted may include the user's age and gender attributes. The dialogue speech information is input into a pre-trained speaker attribute detection model to detect the user's age and gender, and the model outputs the user's age and / or gender information.

[0090] In one possible implementation, the speaker log records the number of speakers currently participating in the communication and their IDs.

[0091] Specifically, the dialogue speech information is input into a pre-trained speaker detection model to detect speaker information, so that the speaker detection model outputs a speaker log.

[0092] In one possible implementation, acoustic events include sounds in the surrounding environment during the conversation between the user to be predicted and the agent.

[0093] For example, acoustic events may include at least one of the following: animal calls, noise, etc.

[0094] In one possible implementation, the dialogue speech information is input into a pre-trained acoustic event detection model, so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

[0095] Furthermore, based on at least one acoustic event output by the acoustic event detection model, an acoustic event log of the dialogue between the user to be predicted and the agent can be output, thus enabling the analysis of the environmental information of the user to be predicted.

[0096] In another possible implementation, the dialogue speech information can be directly input into a pre-trained audio acoustic feature detection model, so that the audio acoustic feature detection model can directly output at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event, the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information, the speaking style classification information corresponding to the user to be predicted, the speaker attribute information corresponding to the user to be predicted, the speaker log, and at least one acoustic event contained in the dialogue speech information.

[0097] Regarding the two methods mentioned above, the first method can extract the acoustic dialogue features of the dialogue speech information in a more targeted manner, while the second method can extract the acoustic dialogue features of the dialogue speech information more quickly. Different extraction methods can be selected according to the extraction requirements to extract the acoustic dialogue features of the dialogue speech information.

[0098] Furthermore, after extracting dialogue logs, text dialogue features, and acoustic text features from the dialogue voice information, the behavioral characteristics of the user to be predicted can be determined by combining the dialogue logs, text dialogue features, and acoustic text features.

[0099] Specifically, the behavioral features are extracted from the dialogue voice information through the following steps: g1: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, perform target behavior detection on the user to be predicted.

[0100] g2: Determine at least one target behavioral feature of the user to be predicted during the dialogue process.

[0101] The target behavior includes at least one of the following: Interrupting conversations, disrupting dialogues, and engaging in conversations with other than the stated agent.

[0102] Here, the behavioral characteristics of engaging in dialogue with other people besides the intelligent agent can be called side behavioral characteristics. During the dialogue between the user to be predicted and the intelligent agent, the user to be predicted may also communicate with other users. At this time, other dialogue voice information between the user to be predicted and other users is collected and analyzed to determine whether the current dialogue between the user to be predicted and other users is related to the dialogue between the user to be predicted and the intelligent agent, so as to determine whether to control the intelligent agent to participate in the dialogue between the user to be predicted and other users.

[0103] In one possible implementation, the target behavior of interrupting and disrupting dialogue is determined by identifying the degree of overlap in the speech between different speakers.

[0104] Specifically, it is necessary to identify two types of overlap in the speech of different speakers, which can include partial overlap and complete overlap.

[0105] Specifically, the target behavioral feature of the user to be predicted during the dialogue is determined as the interruption behavior through the following steps: h1: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, determine the overlap type of the dialogue speech between the user to be predicted and other speakers during the dialogue process.

[0106] h2: If the overlap type of the dialogue between the user to be predicted and other speakers is determined to be a partial overlap type, the target behavioral feature of the user to be predicted during the dialogue is determined to be the interruption behavior.

[0107] Here, partial overlap refers to the overlapping person stopping speaking before the overlapping person finishes speaking; complete overlap refers to the overlapping person continuing to speak after the overlapping person has stopped speaking.

[0108] In one possible implementation, it is first necessary to determine the overlap time of the dialogue between different speakers. If it is determined that the overlap time of the dialogue between different speakers is less than a third preset time interval, then it is determined that the dialogue between different speakers does not overlap.

[0109] Specifically, the third preset time interval can be set to 700 milliseconds.

[0110] Furthermore, if the overlap type of the dialogue speech between different speakers is determined to be a complete overlap type, then the response of the user to be predicted is classified as a potential feedback signal; if the overlap type of the dialogue speech between different speakers is determined to be a partial overlap type, then the target behavioral feature of the user to be predicted is determined to be an interruption behavior.

[0111] In one possible implementation, interrupting speech behavior can also be determined by identifying the degree of overlap between the speech of different speakers. Specifically, if it is determined that the overlap type of the speech between the user to be predicted and other speakers during the conversation is partial overlap, and the speech information of the user to be predicted is received after the other speakers have spoken, and no further speech information of other speakers is received after the user to be predicted has spoken, then it is determined that the user to be predicted has interrupted the conversation of other speakers, and the target behavioral feature of the user to be predicted is determined to be interrupting speech behavior.

[0112] In one possible implementation, after extracting textual dialogue features, acoustic dialogue features, and behavioral features from the dialogue speech information, the textual dialogue features, acoustic dialogue features, and behavioral features can be fused.

[0113] Specifically, the personality prediction method further includes: i1: The text dialogue features, the acoustic dialogue features, and the behavioral features are fused to obtain the fused target dialogue features.

[0114] In one possible implementation, after extracting text dialogue features, acoustic dialogue features, and behavioral features, there may be duplicate labels for the text dialogue features, acoustic dialogue features, and behavioral features. Therefore, each response unit in the dialogue speech information can be used as the judgment unit to determine whether the labels of the text dialogue features, acoustic dialogue features, and behavioral features corresponding to each response unit are duplicated, and the speech feature label corresponding to the final response unit is determined according to the confidence level of different labels.

[0115] Specifically, the step "fusing the textual dialogue features, acoustic dialogue features, and behavioral features to obtain the fused target dialogue features" includes: j1: For each response unit in the dialogue speech information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are used as the speech feature labels corresponding to the response unit.

[0116] j2: For each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the speech feature label corresponding to the response unit is determined by fusing the duplicate labels.

[0117] j3: Collect the speech feature labels corresponding to each response unit to obtain the fused target dialogue features.

[0118] In one possible implementation, for each response unit in the dialogue voice information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are directly put into the target dialogue features without any processing.

[0119] In another possible implementation, for each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the duplicate labels are fused together to determine the speech feature label corresponding to the response unit, and then added to the target dialogue features.

[0120] Specifically, when fusing the repeated labels of the response unit, it is necessary to process them according to the prediction results and prediction confidence of the repeated labels in different features.

[0121] Specifically, the step "For each response unit in the dialogue speech information, if the text dialogue features, acoustic dialogue features, and behavioral features of the response unit have duplicate labels, after fusing the duplicate labels, the speech feature label corresponding to the response unit" includes: k1: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are consistent, the corresponding prediction results shall be used as the speech feature labels of the response unit.

[0122] k2: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is greater than the preset confidence threshold, the prediction result with the highest confidence level among the repeated labels shall be used as the speech feature label corresponding to the response unit.

[0123] k3: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is not greater than the preset confidence threshold, the speech feature label corresponding to the response unit is determined based on the historical speech feature labels extracted from the historical dialogue information.

[0124] In one possible implementation, for each response unit, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are consistent, then the same prediction result is directly used as the speech feature label of the response unit.

[0125] For example, both text dialogue features and acoustic dialogue features have sentiment predictions. If the same response unit is targeted, and both the text dialogue features and acoustic dialogue features predict that the sentiment is sad, then the prediction results of the text dialogue features and acoustic dialogue features are consistent. Therefore, sadness can be directly used as the speech feature label of the response unit.

[0126] In another possible implementation, for each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, it is necessary to refer to the confidence level of each repeated label. If it is determined that the difference in confidence level between repeated labels is greater than a preset confidence threshold, then the prediction result with the highest confidence level in the repeated label pool is taken as the speech feature label corresponding to the response unit.

[0127] For example, both text dialogue features and acoustic dialogue features have sentiment predictions. If the same response unit is targeted, the text dialogue features predict the sentiment as sadness with a prediction confidence of 80%, while the acoustic dialogue features predict the sentiment as anger with a prediction confidence of 30%. In this case, the prediction results of the text dialogue features and the acoustic dialogue features are inconsistent, and the difference between the prediction confidences is large. The prediction result with the higher confidence is taken as the speech label feature of the response unit, that is, the speech feature label of the response unit is sadness.

[0128] In another possible implementation, for each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are inconsistent, and the difference between the confidence levels of each repeated label is not greater than a preset confidence threshold, then the speech feature labels corresponding to the response unit can be determined by referring to the historical speech feature labels extracted from the historical dialogue information.

[0129] For example, both text dialogue features and acoustic dialogue features have sentiment predictions. If, for the same response unit, the text dialogue features predict sadness with a prediction confidence of 70%, while the acoustic dialogue features predict anger with a prediction confidence of 65%, then the prediction results of the text dialogue features and acoustic dialogue features are inconsistent, and the difference between the prediction confidences is small. It is necessary to refer to the historical speech feature labels of the user to be predicted in the historical speech feature labels extracted from the historical dialogue information to make a judgment. According to the historical speech feature labels, the user to be predicted is in a sad mood, so it can be determined that the speech feature label of this response unit is sadness.

[0130] Furthermore, during the dialogue between the agent and the user to be predicted, the dialogue log and dialogue summary information of the two are stored in real time. After extracting the dialogue log and dialogue summary information from the voice information of the dialogue between the user to be predicted and the agent, the historical dialogue log corresponding to the user to be predicted can be updated and supplemented based on the target dialogue features to obtain the updated dialogue log.

[0131] S103. Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log extracted from the dialogue voice information and the target dialogue features.

[0132] In one possible implementation, the dialogue log records the dialogue information between the user to be predicted and the agent at different times. The dialogue information includes the target dialogue features of the user to be predicted (the user's textual dialogue features, acoustic dialogue features, and behavioral features) and the timestamps corresponding to the generated dialogue messages. At the same time, the dialogue log also includes the user's state information when the user is having a dialogue with the agent. For example, whether the user is angry or happy, or whether the user is sick during the dialogue.

[0133] Specifically, the dialogue log is updated through the following steps: l1: Based on the target dialogue features, update and supplement the historical dialogue logs corresponding to the user to be predicted to obtain the updated dialogue logs.

[0134] In this embodiment of the application, the historical dialogue log can be enriched based on the target dialogue features from the dialogue voice information, thereby obtaining a more complete and accurate updated dialogue log. Then, the user personality prediction can be performed based on the updated dialogue log, which can improve the accuracy of the prediction.

[0135] Specifically, the historical dialogue logs of the user to be predicted are updated based on the characteristics of the target dialogue. This can include incorporating corresponding voice events or adding prompts.

[0136] For example, after identifying laughter events from acoustic dialogue features, laughter tags can be added based on the relationship between laughter and speech: a. [Laughter]: A separate burst of laughter; b. [StartLaugh] and [EndLaugh]: Laughter that accompanies speech.

[0137] Furthermore, laughter tags are integrated into the transcribed text: a. If the laughter contains the words "Come on!", then mark it as "[StartLaugh] Come on [EndLaugh]"; b. If the laughter does not contain text, it is marked as "[Laughter]".

[0138] For example, after determining the emotion or style of the user to be predicted from acoustic dialogue features or text dialogue features, corresponding prompt words can be added to the dialogue log.

[0139] For example, the historical dialogue log records speaker A: "The weather is so nice today!"; and the analysis determines that speaker A's emotion is happiness and speaking style is loud. Then, the updated dialogue log could be speaker A (happily saying loudly): "The weather is so nice today!".

[0140] In another possible implementation, the dialogue summary can also be updated based on the target dialogue features extracted from the dialogue voice information.

[0141] Specifically, based on historical dialogue summaries, combined with the currently extracted target dialogue features and dialogue logs, the dialogue summary is updated, including background summaries, dialogue content summaries, event summaries, and memory summaries. By combining long-term dialogue history with the current dialogue content and features, the summary and update are carried out. It is necessary to emphasize the impact of current special events, such as the user state to be predicted extracted from dialogue features (such as illness, anger, etc.), in order to better analyze the current user's usual personality and personality fluctuations in the current state, thereby improving the accuracy of personality prediction for the user to be predicted.

[0142] Among them, the background summary information refers to the background information of the user being predicted in the current dialogue with the agent. For example, the user being predicted is talking to the agent after receiving good news at noon on a certain day. The dialogue content summary refers to the content of the dialogue between the user being predicted and the agent. For example, the user being predicted and the agent discussed the local weather conditions. The event summary refers to the specific events mentioned by the user being predicted in the current dialogue with the agent and the impact of the specific events on the user being predicted. For example, the user being predicted received good news, which made the user being predicted in the current dialogue a happy state. The memory summary refers to the summary information of the user being predicted's personality or state based on the user's historical dialogue information with the agent. For example, the user being predicted is in an anxious state in the historical dialogue information with the agent, and is worried that some things will not meet expectations. Furthermore, the summary information of the user being predicted in the historical dialogue information with the agent indicates that the user has a relatively conservative personality.

[0143] Furthermore, after extracting the target dialogue features, the updated dialogue log, and the dialogue summary information from the dialogue voice information, at least one predicted label can be determined based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information.

[0144] S104. Based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained.

[0145] In the embodiments of this application, the prediction labels include numerical prediction labels and descriptive summary prediction labels. The determination of different types of labels is not the same. The determination process of different types of labels will be described below.

[0146] Specifically, when the predicted label includes a numerical predicted label, the step "obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information" includes: m1: Based on the fused target dialogue features, determine the label categories and percentage values ​​of each label category in the target dialogue features, and determine the first numerical category predicted label.

[0147] m2: Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, determine at least one statistical data information corresponding to the user to be predicted and the grouping label corresponding to each statistical data information, and determine the second numerical class prediction label.

[0148] In one possible implementation, the percentage value of each label category can be determined based on multiple label types in the target dialogue features, thereby determining the first numerical category prediction label.

[0149] For example, for emotion tags, an anger category tag can be identified with an anger category tag percentage of 60%, and for emotion tags, a positive category tag can be identified with a positive category tag percentage of 10%, etc.

[0150] In another possible implementation, at least one statistical data can be determined based on the target dialogue features, the updated dialogue log, and the dialogue summary information, and the grouping label corresponding to each statistical data can be determined, thereby determining the first numerical class prediction label.

[0151] Specifically, statistical data may include at least one of the following: number of rounds, average speaking time, average events, average emotional / cognitive feedback, number of interruptions, and number of side conversations (the user to be predicted to have conversations with other objects besides the agent).

[0152] Specifically, the number of rounds is defined as the number of responses at 700 millisecond intervals, excluding feedback signals and unsuccessful interruptions; the average speaking duration is the average duration of speech in each round; the average events are the number of speech events or acoustic events occurring per minute in the speech; the average emotional / cognitive feedback is the number of emotional or cognitive feedback events per minute during another speaker's speech; the number of interruptions is the total number of successful and unsuccessful interruptions within a preset dialogue time period; and the number of side conversations is the total number of side conversations within a preset dialogue time period.

[0153] In one possible implementation, after determining the corresponding statistical data, it is also necessary to determine a corresponding group label for each statistical data. The group label is determined based on the mean and four positional distance values ​​of all speakers in the corresponding statistical data.

[0154] Specifically, the grouping label corresponding to each statistical data point is determined through the following steps: n1: For each statistical data point, the mean of all speakers corresponding to that statistical data point and the four positional distances are calculated.

[0155] n2: For each statistical data information, based on the relationship between the calculated group value corresponding to the statistical data information and the mean and the four positional intervals, determine the group label corresponding to the statistical data information.

[0156] In one possible implementation, for each statistical data information, the mean of all speakers corresponding to the statistical data information and the calculated four positional distances are calculated. Then, based on the relationship between the calculated group value corresponding to the statistical data information, the mean, and the four positional distances, the group label corresponding to the statistical data information is determined.

[0157] Here, the interquartile range (IQR), also known as the quartile difference, is a method in descriptive statistics used to determine the difference between the third quartile and the first quartile.

[0158] In one possible implementation, the group labels are categorized as normal, more, less, very much, and very little.

[0159] Specifically, if the absolute value of the calculated group value corresponding to the statistical data is less than the mean plus 0.8 × the four-part interval, the corresponding group label is normal; if the absolute value of the calculated group value corresponding to the statistical data is greater than the mean plus 0.8 × the four-part interval, the corresponding group label is more or less; if the absolute value of the calculated group value corresponding to the statistical data is greater than the mean plus 1.2 × the four-part interval, the corresponding group label is very more or very less.

[0160] In another possible implementation, when describing summary prediction labels, it is necessary to extract at least one dialogue analysis sample from the dialogue log, and then extract the corresponding non-data type labels and important node labels of the summary.

[0161] Specifically, when the predicted label includes a descriptive summary predicted label, the step "obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information" includes: o1: Extract at least one dialogue analysis sample from the dialogue log.

[0162] o2: For each dialogue analysis sample, extract at least one non-data class label based on the target dialogue features corresponding to the dialogue analysis sample.

[0163] o3: For each dialogue analysis sample, based on the dialogue summary information of that dialogue analysis sample, determine at least one important node label.

[0164] In this embodiment of the application, 20 rounds lasting more than 2 seconds can be randomly selected from the dialogue log as dialogue analysis samples.

[0165] In one possible implementation, non-data tags may include information such as speaking style and speaker information (age / gender), and the corresponding non-data tags are determined based on the above information.

[0166] In one possible implementation, historical general information and current important information are extracted from the dialogue summary to identify at least one important node label.

[0167] For example, the conversation concludes that certain events had a significant impact on the user's prediction, such as falling ill.

[0168] Furthermore, after determining at least one predictive label for the user to be predicted, the at least one predictive label is fused to obtain standardized textual description information of the user, and then the standardized textual description information of the user is input into the large language model to obtain the personality prediction result for the user to be predicted.

[0169] S105. Input the user-standardized text description information obtained by fusing the at least one prediction label into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

[0170] In one possible implementation, a text description category can be preset, and then, by combining the target dialogue features, the updated dialogue log, and the dialogue summary information, the dialogue description information corresponding to different preset text description categories can be determined, thereby determining the user's standardized text description information.

[0171] Specifically, the user-standardized text description information is determined through the following steps: p1: Based on the target dialogue features, the updated dialogue log, the dialogue summary information, and at least one predicted label of the user to be predicted, determine the dialogue description information corresponding to each preset text description category.

[0172] p2: Collect the dialogue description information corresponding to each preset text description category to obtain the user's standardized text description information.

[0173] In one possible implementation, the preset text description classification includes at least one of the following: yes feature classification, important information classification, dialogue log classification, dialogue summary classification, etc.; then, by combining the target dialogue features, the updated dialogue log and dialogue summary information, and determining the dialogue description information corresponding to different preset text description classifications, standardized text description information for users is obtained.

[0174] Furthermore, after determining the user's standardized text targeting information, when predicting the personality of the user to be predicted using the large language model, the input to the large language model is comprehensive prompt information to assist the large language model in predicting the personality information of the user to be predicted.

[0175] Specifically, the comprehensive prompts may include prediction task definition information, personality dimension explanation information, and user standardized text targeting information.

[0176] Here, the personality prediction results for the user to be predicted include the alignment results of the user to be predicted on each personality dimension.

[0177] Specifically, by inputting comprehensive prompts into a pre-trained large language model, the alignment labels of the user to be predicted on five personality dimensions can be predicted. Then, by averaging the five personality dimensions, a reliable alignment score can be obtained, which can output the user's general personality traits and personality characteristics in the current dialogue scenario.

[0178] In one possible implementation, in order to ensure that the agent can communicate more accurately with the user to be predicted, the large language model can output dialogue assistance information while outputting personality prediction results, thereby assisting the agent to demonstrate reasonable and accurate interaction with the user to be predicted.

[0179] Specifically, the personality prediction method further includes: q1: Input the user-standardized text description information obtained by fusing the at least one prediction label into the pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted, and outputs dialogue assistance information for the user to be predicted, so that the agent can carry out interaction with the user to be predicted based on the dialogue assistance information.

[0180] The dialogue assistance information includes at least one of the following: The historical personality information of the user to be predicted, the current emotional information of the user to be predicted, and the dialogue direction indication information.

[0181] In one possible implementation, dialogue assistance information can more accurately characterize the current dialogue state of the user to be predicted, enabling the agent to interact with the user more accurately.

[0182] For example, if the historical dialogue logs indicate that the user is a lively person, a more relaxed tone can be used in the conversation. However, if the current dialogue voice information indicates that the user is in a low mood recently, it is no longer appropriate to communicate in a relaxed tone. This provides additional input to the agent and improves the agent's interactive experience.

[0183] The following specific examples will illustrate the personality prediction process based on voice dialogue in the embodiments of this disclosure.

[0184] Specifically, please refer to Figure 2 , Figure 2 The flowchart for extracting target dialogue features provided in the embodiments of this application is as follows: Figure 2 The process involves receiving dialogue voice information, inputting it into an automatic speech recognition system, and simultaneously extracting dialogue logs by combining historical dialogue logs, historical dialogue features, and historical dialogue summaries. Based on these logs, text feature detection is performed to extract text dialogue features. Furthermore, by directly combining historical dialogue logs, historical dialogue features, and historical dialogue summaries, acoustic feature detection is performed on the dialogue voice information to extract acoustic dialogue features. Finally, the acoustic dialogue features and text dialogue features are fused to obtain the target dialogue features.

[0185] For further details, please refer to Figure 3 , Figure 3 The user personality prediction flowchart provided in the embodiments of this application is as follows: Figure 3 As shown, the dialogue log is updated based on the extracted target dialogue features and the dialogue log; at the same time, the dialogue summary is updated based on the extracted dialogue summary and the updated dialogue log, thereby generating comprehensive prompt information. The comprehensive prompt information is input into the large language model for prediction, and the personality prediction result is output.

[0186] For example, the specific personality prediction process is as follows: Step 1: Collect all statements (dialogue voice information) of the user to be predicted in a 12-minute multi-person conversation. Step 2: Feature Integration i. Extract and calculate numerical label data: a. Emotional distribution: Anger (5%), Happiness (60%), Sadness (10%) b. Emotional distribution: Positive (65%), Negative (15%), Neutral (20%) c. Basic attributes: 1. Number of posts: 45 (categorized as "Many") 2. Average speaking time: 18 seconds (categorized as "Normal") 3. Laughter frequency: 4 times / minute (categorized as "Many") 4. Frequency of emotional agreement: 3 times / minute (categorized as "Many") 5. Number of interruptions: 8 (categorized as "Normal") ii. Extract summary and descriptive data: a. Extract dialogue logs from nearly 20 rounds of conversations; b. Select typical non-numerical features from the dialogue characteristics, such as "light and bright speech, young male, in a multi-person open scene, and lots of children's laughter"; c. Extract information from the dialogue summary: "The user is outgoing and lively, enjoys sharing, and is currently having fun at the amusement park in a great mood." Step 3: Integrate the above attributes into a standardized text description.

[0187] Step 4: A large-scale language model analyzes these descriptions and outputs personality assessment results: i. Openness: aligned ii. Due diligence: Neutral iii. Extraversion: Highly aligned iv. Agreeableness: Aligned v. Unstable: Opposed Step 5: Output auxiliary information: "The user has an extroverted personality and is currently in a relaxed play environment, which further amplifies their personality traits."

[0188] The personality prediction method based on voice dialogue provided in this application involves: acquiring voice dialogue information between the user to be predicted and an agent; extracting target dialogue features of the user to be predicted from the voice dialogue information; and extracting dialogue logs from the voice dialogue information. The target dialogue features include textual dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted. The user's dialogue logs and dialogue summary information are updated based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information. At least one predicted label for the user to be predicted is obtained based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information. The standardized text description information of the user obtained based on the fused at least one predicted label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result for the user to be predicted. Thus, after acquiring the voice dialogue information between the user to be predicted and the agent, the target dialogue features of the user to be predicted can be extracted based on the voice dialogue information and historical dialogue extraction information. These target dialogue features include not only textual dialogue features but also acoustic dialogue features and behavioral features. This allows for the determination of the predicted label of the user to be predicted from multiple dimensions, thereby determining the personality prediction result and improving the accuracy of personality prediction for the user to be predicted.

[0189] Based on the same inventive concept, this disclosure also provides a voice-based personality prediction device corresponding to the voice-based personality prediction method. Since the principle of the device in this disclosure is similar to the above-mentioned voice-based personality prediction method in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0190] Please see Figure 4 , Figure 4 This is a schematic diagram of a personality prediction device based on voice dialogue, provided as an embodiment of this application. Figure 4 As shown, the personality prediction device 400 includes: The dialogue voice information acquisition module 410 is used to acquire the dialogue voice information between the user to be predicted and the intelligent agent. The dialogue feature extraction module 420 is used to extract the target dialogue features of the user to be predicted from the dialogue voice information, and to extract the dialogue log from the dialogue voice information; wherein, the target dialogue features include the text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted. The information update module 430 is used to update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features. The prediction label determination module 440 is used to obtain at least one prediction label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information. The personality prediction module 450 is used to input the user's standardized text description information obtained by fusing the at least one prediction label into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted.

[0191] In an optional implementation, when the dialogue feature extraction module 420 extracts the target dialogue features of the user to be predicted from the dialogue speech information, the dialogue feature extraction module 420 is used to: Based on the dialogue voice information and the historical dialogue extraction information of the user to be predicted, the target dialogue features of the user to be predicted are extracted from the dialogue voice information.

[0192] In an optional implementation, when the dialogue feature extraction module 420 is used to extract the dialogue log from the dialogue voice information, the dialogue feature extraction module 420 is used to: The dialogue voice information is input into the automatic speech recognition system, the dialogue voice information is converted into text dialogue information, and each word contained in the text dialogue information and the word association information corresponding to each word are determined. Based on the text dialogue information and the timestamp information corresponding to each word, at least one response unit in the dialogue voice information is determined. The dialogue log is obtained based on the at least one response unit, each word, and the word association information corresponding to each word.

[0193] In an optional implementation, when the dialogue feature extraction module 420 determines at least one response unit in the dialogue speech information based on the text dialogue information and the timestamp information corresponding to each word, the dialogue feature extraction module 420 is used to: Based on the text dialogue information and the timestamp information corresponding to each word, determine the time interval and semantic continuity between at least two consecutive words; Based on the time interval between at least two consecutive words and semantic continuity, at least two consecutive words are merged to obtain the response unit.

[0194] In an optional implementation, when the dialogue feature extraction module 420 merges at least two consecutive words based on the time interval between at least two consecutive words and semantic continuity to obtain the response unit, the dialogue feature extraction module 420 is used to: If the time interval between at least two consecutive words is greater than the first preset time interval, merge at least two consecutive words to obtain the first merged word group; Detect whether there is semantic continuity between at least two words in the first merged word group. If there is semantic continuity between at least two words, determine the first merged word group as a response unit. If there is no semantic continuity between at least two words, and the time interval between at least two consecutive words is greater than a second preset time interval, at least two consecutive words are merged to obtain a second merged word group, and the second merged word group is determined as a response unit.

[0195] In an optional implementation, the dialogue feature extraction module 420 is used to extract the text dialogue features from the dialogue voice information through the following steps: The dialogue log is input into a pre-trained sentiment classifier so that the sentiment classifier outputs the sentiment label and sentiment intensity information corresponding to each response unit in the dialogue speech information; The dialogue log is input into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label corresponding to each response unit in the dialogue voice information. The dialogue log is input into a pre-trained feedback classifier so that the feedback classifier outputs a feedback label corresponding to each response unit in the dialogue voice information.

[0196] In an optional implementation, the dialogue feature extraction module 420 is used to extract the acoustic dialogue features from the dialogue speech information through the following steps: The dialogue speech information is input into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event. The dialogue speech information is input into a pre-trained speech emotion detection model so that the speech emotion detection model outputs the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information; The dialogue speech information is input into a pre-trained speech style detection model so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker attribute detection model so that the speaker attribute detection model outputs the speaker attribute information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker detection model so that the speaker detection model outputs a speaker log. The dialogue speech information is input into a pre-trained acoustic event detection model so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

[0197] In an optional implementation, the dialogue feature extraction module 420 is used to extract the behavioral features from the dialogue voice information through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, target behavior detection is performed on the user to be predicted. Determine at least one target behavioral feature of the user to be predicted during the dialogue process; The target behavior includes at least one of the following: Interrupting conversations, disrupting dialogues, and engaging in conversations with other than the stated agent.

[0198] In an optional implementation, the dialogue feature extraction module 420 is used to determine the target behavioral feature of the user to be predicted during the dialogue as the interruption behavior through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, determine the overlap type of the dialogue speech between the user to be predicted and other speakers during the dialogue process; If the overlap type of the dialogue between the user to be predicted and other speakers is determined to be a partial overlap type, then the target behavioral feature of the user to be predicted during the dialogue is determined to be the interruption behavior.

[0199] In an optional embodiment, the personality prediction device 400 further includes a feature fusion module (not shown in the figure), the feature fusion module being used for: The textual dialogue features, the acoustic dialogue features, and the behavioral features are fused to obtain the fused target dialogue features.

[0200] In an optional implementation, when the feature fusion module fuses the text dialogue features, acoustic dialogue features, and behavioral features to obtain the fused target dialogue features, the feature fusion module is used to: For each response unit in the dialogue speech information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are used as the speech feature labels corresponding to the response unit. For each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the duplicate labels are fused together to determine the speech feature label corresponding to the response unit. By combining the speech feature labels corresponding to each response unit, the fused target dialogue features are obtained.

[0201] In an optional implementation, when the feature fusion module determines the speech feature label corresponding to each response unit in the dialogue speech information by fusing the repeated labels after fusing the repeated labels, the feature fusion module is used to: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are consistent, the corresponding prediction results are used as the speech feature labels of the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is greater than the preset confidence threshold, the prediction result with the highest confidence level among each repeated label shall be used as the speech feature label corresponding to the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is not greater than a preset confidence threshold, the speech feature label corresponding to the response unit is determined based on the historical speech feature labels extracted from the historical dialogue information.

[0202] In an optional implementation, the dialogue feature extraction module 420 is used to update the dialogue log through the following steps: Based on the target dialogue features, the historical dialogue logs corresponding to the user to be predicted are updated and supplemented to obtain the updated dialogue logs.

[0203] In an optional implementation, the predicted labels include numerical predicted labels and descriptive summary type predicted labels; when the predicted labels include numerical predicted labels, the predicted label determining module 440, when used to obtain at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information, is configured to: Based on the fused target dialogue features, determine the label categories and percentage values ​​of each label category in the target dialogue features, and determine the first numerical category predicted label; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one statistical data item corresponding to the user to be predicted and a grouping label corresponding to each statistical data item are determined, and a second numerical class prediction label is determined.

[0204] In an optional implementation, when the predicted label includes a descriptive summary predicted label, the predicted label determining module 440, when used to obtain at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information, is configured to: Extract at least one dialogue analysis sample from the dialogue log; For each dialogue analysis sample, at least one non-data class label is extracted based on the target dialogue features corresponding to the dialogue analysis sample. For each dialogue analysis sample, at least one important node label is determined based on the dialogue summary information of that dialogue analysis sample.

[0205] In an optional implementation, the prediction label determination module 440 is used to determine the group label corresponding to each statistical data item through the following steps: For each statistical data point, the mean of all speakers corresponding to that statistical data point and the four positional distances are calculated. For each statistical data point, the group label corresponding to that statistical data point is determined based on the relationship between the calculated group value corresponding to that statistical data point and the mean and the four positional intervals.

[0206] In one optional implementation, the personality prediction module 450 is used to determine the user's standardized text description information through the following steps: Based on the target dialogue features, the updated dialogue logs, the dialogue summary information, and at least one predicted label of the user to be predicted, determine the dialogue description information corresponding to each preset text description category; The user-standardized text description information is obtained by aggregating the dialogue description information corresponding to each preset text description category.

[0207] In one optional implementation, the personality prediction result of the user to be predicted includes the alignment result of the user to be predicted on each personality dimension.

[0208] In an optional embodiment, the personality prediction device 400 further includes an auxiliary information output module (not shown in the figure), the auxiliary information output module being used for: The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted, and outputs dialogue assistance information for the user to be predicted, so that the agent can carry out interaction with the user to be predicted based on the dialogue assistance information. The dialogue assistance information includes at least one of the following: The historical personality information of the user to be predicted, the current emotional information of the user to be predicted, and the dialogue direction indication information.

[0209] The personality prediction device based on voice dialogue provided in this application acquires voice dialogue information between a user to be predicted and an intelligent agent; extracts target dialogue features of the user to be predicted from the voice dialogue information, and extracts dialogue logs from the voice dialogue information; wherein, the target dialogue features include text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; updates the dialogue logs and dialogue summary information of the user to be predicted based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information; obtains at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information; and inputs the user's standardized text description information obtained based on the fusion of at least one predicted label into a pre-trained large language model, so that the large language model outputs the personality prediction result for the user to be predicted. Thus, after acquiring the voice dialogue information between the user to be predicted and the intelligent agent, the target dialogue features of the user to be predicted can be extracted based on the voice dialogue information and historical dialogue extraction information. The target dialogue features include not only text dialogue features but also acoustic dialogue features and behavioral features. This allows for the determination of the predicted label of the user to be predicted from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps improve the accuracy of personality prediction for the user to be predicted. Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0210] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530, causing the processor 510 to execute the following instructions during operation: Acquire the voice dialogue information between the user to be predicted and the intelligent agent; Extracting target dialogue features of the user to be predicted from the dialogue voice information, and extracting dialogue logs from the dialogue voice information; wherein, the target dialogue features include text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained. The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

[0211] In one optional implementation, the instructions executed by the processor 510, wherein extracting the target dialogue features of the user to be predicted from the dialogue voice information, includes: Based on the dialogue voice information and the historical dialogue extraction information of the user to be predicted, the target dialogue features of the user to be predicted are extracted from the dialogue voice information.

[0212] In one optional implementation, the instructions executed by the processor 510, wherein extracting the dialogue log from the dialogue voice information, includes: The dialogue voice information is input into the automatic speech recognition system, the dialogue voice information is converted into text dialogue information, and each word contained in the text dialogue information and the word association information corresponding to each word are determined. Based on the text dialogue information and the timestamp information corresponding to each word, at least one response unit in the dialogue voice information is determined. The dialogue log is obtained based on the at least one response unit, each word, and the word association information corresponding to each word.

[0213] In one optional implementation, the instructions executed by the processor 510, wherein determining at least one response unit in the dialogue voice information based on the text dialogue information and the timestamp information corresponding to each word, includes: Based on the text dialogue information and the timestamp information corresponding to each word, determine the time interval and semantic continuity between at least two consecutive words; Based on the time interval between at least two consecutive words and semantic continuity, at least two consecutive words are merged to obtain the response unit.

[0214] In one optional implementation, the instructions executed by the processor 510, which involve merging at least two consecutive words based on the time interval between them and semantic continuity to obtain the response unit, include: If the time interval between at least two consecutive words is greater than the first preset time interval, merge at least two consecutive words to obtain the first merged word group; Detect whether there is semantic continuity between at least two words in the first merged word group. If there is semantic continuity between at least two words, determine the first merged word group as a response unit. If there is no semantic continuity between at least two words, and the time interval between at least two consecutive words is greater than a second preset time interval, at least two consecutive words are merged to obtain a second merged word group, and the second merged word group is determined as a response unit.

[0215] In one optional implementation, the instructions executed by the processor 510 extract the text dialogue features from the dialogue voice information through the following steps: The dialogue log is input into a pre-trained sentiment classifier so that the sentiment classifier outputs the sentiment label and sentiment intensity information corresponding to each response unit in the dialogue speech information; The dialogue log is input into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label corresponding to each response unit in the dialogue voice information. The dialogue log is input into a pre-trained feedback classifier so that the feedback classifier outputs a feedback label corresponding to each response unit in the dialogue voice information.

[0216] In one optional implementation, the processor 510 executes instructions to extract the acoustic dialogue features from the dialogue voice information through the following steps: The dialogue speech information is input into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event. The dialogue speech information is input into a pre-trained speech emotion detection model so that the speech emotion detection model outputs the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information; The dialogue speech information is input into a pre-trained speech style detection model so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker attribute detection model so that the speaker attribute detection model outputs the speaker attribute information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker detection model so that the speaker detection model outputs a speaker log. The dialogue speech information is input into a pre-trained acoustic event detection model so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

[0217] In one optional implementation, the processor 510 executes instructions that extract the behavioral features from the dialogue voice information through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, target behavior detection is performed on the user to be predicted. Determine at least one target behavioral feature of the user to be predicted during the dialogue process; The target behavior includes at least one of the following: Interrupting conversations, disrupting dialogues, and engaging in conversations with other than the stated agent.

[0218] In one optional implementation, the instructions executed by the processor 510 determine the target behavioral feature of the user to be predicted during the dialogue as the interruption behavior through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, determine the overlap type of the dialogue speech between the user to be predicted and other speakers during the dialogue process; If the overlap type of the dialogue between the user to be predicted and other speakers is determined to be a partial overlap type, then the target behavioral feature of the user to be predicted during the dialogue is determined to be the interruption behavior.

[0219] In one optional implementation, the instructions executed by the processor 510 further include: The textual dialogue features, the acoustic dialogue features, and the behavioral features are fused to obtain the fused target dialogue features.

[0220] In an optional implementation, the instructions executed by the processor 510, wherein fusing the textual dialogue features, acoustic dialogue features, and behavioral features to obtain the fused target dialogue features, include: For each response unit in the dialogue speech information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are used as the speech feature labels corresponding to the response unit. For each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the duplicate labels are fused together to determine the speech feature label corresponding to the response unit. By combining the speech feature labels corresponding to each response unit, the fused target dialogue features are obtained.

[0221] In an optional implementation, the instructions executed by the processor 510, specifically regarding each response unit in the dialogue speech information, if the text dialogue features, acoustic dialogue features, and behavioral features of the response unit have duplicate labels, and after fusing the duplicate labels, determine the speech feature label corresponding to the response unit, include: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are consistent, the corresponding prediction results are used as the speech feature labels of the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is greater than the preset confidence threshold, the prediction result with the highest confidence level among each repeated label shall be used as the speech feature label corresponding to the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is not greater than a preset confidence threshold, the speech feature label corresponding to the response unit is determined based on the historical speech feature labels extracted from the historical dialogue information.

[0222] In one optional implementation, the processor 510 updates the dialogue log through the following steps in the instructions executed: Based on the target dialogue features, the historical dialogue logs corresponding to the user to be predicted are updated and supplemented to obtain the updated dialogue logs.

[0223] In an optional implementation, the instructions executed by the processor 510 include a prediction label comprising a numerical prediction label and a descriptive summary prediction label; when the prediction label includes a numerical prediction label, obtaining at least one prediction label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Based on the fused target dialogue features, determine the label categories and percentage values ​​of each label category in the target dialogue features, and determine the first numerical category predicted label; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one statistical data item corresponding to the user to be predicted and a grouping label corresponding to each statistical data item are determined, and a second numerical class prediction label is determined.

[0224] In an optional implementation, when the predicted label includes a descriptive summary predicted label, the step of obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Extract at least one dialogue analysis sample from the dialogue log; For each dialogue analysis sample, at least one non-data class label is extracted based on the target dialogue features corresponding to the dialogue analysis sample. For each dialogue analysis sample, at least one important node label is determined based on the dialogue summary information of that dialogue analysis sample.

[0225] In one optional implementation, the instructions executed by the processor 510 determine the grouping label corresponding to each statistical data item through the following steps: For each statistical data point, the mean of all speakers corresponding to that statistical data point and the four positional distances are calculated. For each statistical data point, the group label corresponding to that statistical data point is determined based on the relationship between the calculated group value corresponding to that statistical data point and the mean and the four positional intervals.

[0226] In one optional implementation, the instructions executed by the processor 510 determine the user-standardized text description information through the following steps: Based on the target dialogue features, the updated dialogue logs, the dialogue summary information, and at least one predicted label of the user to be predicted, determine the dialogue description information corresponding to each preset text description category; The user-standardized text description information is obtained by aggregating the dialogue description information corresponding to each preset text description category.

[0227] In one optional implementation, the personality prediction result of the user to be predicted includes the alignment result of the user to be predicted on each personality dimension.

[0228] In one optional implementation, the instructions executed by the processor 510 further include: The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted, and outputs dialogue assistance information for the user to be predicted, so that the agent can carry out interaction with the user to be predicted based on the dialogue assistance information. The dialogue assistance information includes at least one of the following: The historical personality information of the user to be predicted, the current emotional information of the user to be predicted, and the dialogue direction indication information.

[0229] By using the above method, after obtaining the dialogue voice information between the user to be predicted and the agent, the target dialogue features of the user to be predicted can be extracted based on the dialogue voice information and historical dialogue extraction information. The target dialogue features include not only text dialogue features, but also acoustic dialogue features and behavioral features. In this way, the prediction label of the user to be predicted can be determined from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps to improve the accuracy of personality prediction. Furthermore, in order to ensure that the agent can communicate more accurately with the user to be predicted, the large language model can also output dialogue auxiliary information while outputting the personality prediction result, thereby assisting the agent to demonstrate reasonable and accurate interaction with the user to be predicted.

[0230] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following instructions: Acquire the voice dialogue information between the user to be predicted and the intelligent agent; Extracting target dialogue features of the user to be predicted from the dialogue voice information, and extracting dialogue logs from the dialogue voice information; wherein, the target dialogue features include text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained. The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

[0231] In one optional implementation, the instructions executed by the computer-readable storage medium, wherein extracting the target dialogue features of the user to be predicted from the dialogue voice information, includes: Based on the dialogue voice information and the historical dialogue extraction information of the user to be predicted, the target dialogue features of the user to be predicted are extracted from the dialogue voice information.

[0232] In one optional implementation, the instructions executed by the computer-readable storage medium, wherein extracting the dialogue log from the dialogue voice information, includes: The dialogue voice information is input into the automatic speech recognition system, the dialogue voice information is converted into text dialogue information, and each word contained in the text dialogue information and the word association information corresponding to each word are determined. Based on the text dialogue information and the timestamp information corresponding to each word, at least one response unit in the dialogue voice information is determined. The dialogue log is obtained based on the at least one response unit, each word, and the word association information corresponding to each word.

[0233] In one optional implementation, the instructions executed by the computer-readable storage medium, wherein determining at least one response unit in the dialogue voice information based on the text dialogue information and the timestamp information corresponding to each word, includes: Based on the text dialogue information and the timestamp information corresponding to each word, determine the time interval and semantic continuity between at least two consecutive words; Based on the time interval between at least two consecutive words and semantic continuity, at least two consecutive words are merged to obtain the response unit.

[0234] In one optional implementation, the instructions executed by the computer-readable storage medium, wherein merging at least two consecutive words based on the time interval between at least two consecutive words and semantic continuity to obtain the response unit, includes: If the time interval between at least two consecutive words is greater than the first preset time interval, merge at least two consecutive words to obtain the first merged word group; Detect whether there is semantic continuity between at least two words in the first merged word group. If there is semantic continuity between at least two words, determine the first merged word group as a response unit. If there is no semantic continuity between at least two words, and the time interval between at least two consecutive words is greater than a second preset time interval, at least two consecutive words are merged to obtain a second merged word group, and the second merged word group is determined as a response unit.

[0235] In one optional implementation, the instructions executed by the computer-readable storage medium extract the text dialogue features from the dialogue voice information through the following steps: The dialogue log is input into a pre-trained sentiment classifier so that the sentiment classifier outputs the sentiment label and sentiment intensity information corresponding to each response unit in the dialogue speech information; The dialogue log is input into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label corresponding to each response unit in the dialogue voice information. The dialogue log is input into a pre-trained feedback classifier so that the feedback classifier outputs a feedback label corresponding to each response unit in the dialogue voice information.

[0236] In one optional implementation, the instructions executed by the computer-readable storage medium extract the acoustic dialogue features from the dialogue voice information through the following steps: The dialogue speech information is input into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event. The dialogue speech information is input into a pre-trained speech emotion detection model so that the speech emotion detection model outputs the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information; The dialogue speech information is input into a pre-trained speech style detection model so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker attribute detection model so that the speaker attribute detection model outputs the speaker attribute information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker detection model so that the speaker detection model outputs a speaker log. The dialogue speech information is input into a pre-trained acoustic event detection model so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

[0237] In one optional implementation, the instructions executed by the computer-readable storage medium extract the behavioral features from the dialogue voice information through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, target behavior detection is performed on the user to be predicted. Determine at least one target behavioral feature of the user to be predicted during the dialogue process; The target behavior includes at least one of the following: Interrupting conversations, disrupting dialogues, and engaging in conversations with other than the stated agent.

[0238] In one optional implementation, the instructions executed by the computer-readable storage medium determine the target behavioral feature of the user to be predicted during the dialogue as the interruption behavior through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, determine the overlap type of the dialogue speech between the user to be predicted and other speakers during the dialogue process; If the overlap type of the dialogue between the user to be predicted and other speakers is determined to be a partial overlap type, then the target behavioral feature of the user to be predicted during the dialogue is determined to be the interruption behavior.

[0239] In one optional implementation, the instructions executed by the computer-readable storage medium further include: The textual dialogue features, the acoustic dialogue features, and the behavioral features are fused to obtain the fused target dialogue features.

[0240] In one optional implementation, the instructions executed by the computer-readable storage medium, wherein fusing the textual dialogue features, acoustic dialogue features, and behavioral features to obtain the fused target dialogue features, include: For each response unit in the dialogue speech information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are used as the speech feature labels corresponding to the response unit. For each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the duplicate labels are fused together to determine the speech feature label corresponding to the response unit. By combining the speech feature labels corresponding to each response unit, the fused target dialogue features are obtained.

[0241] In an optional implementation, in the instructions executed by the computer-readable storage medium, the step of determining the speech feature label corresponding to each response unit in the dialogue speech information, if the text dialogue features, acoustic dialogue features, and behavioral features of the response unit have duplicate labels, involves fusing the duplicate labels and then determining the speech feature label corresponding to the response unit, including: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are consistent, the corresponding prediction results are used as the speech feature labels of the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is greater than the preset confidence threshold, the prediction result with the highest confidence level among each repeated label shall be used as the speech feature label corresponding to the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is not greater than a preset confidence threshold, the speech feature label corresponding to the response unit is determined based on the historical speech feature labels extracted from the historical dialogue information.

[0242] In one optional implementation, the instructions executed by the computer-readable storage medium update the dialogue log through the following steps: Based on the target dialogue features, the historical dialogue logs corresponding to the user to be predicted are updated and supplemented to obtain the updated dialogue logs.

[0243] In one optional implementation, in the instructions executable by the computer-readable storage medium, the prediction label includes numerical prediction labels and descriptive summary type prediction labels; when the prediction label includes numerical prediction labels, obtaining at least one prediction label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Based on the fused target dialogue features, determine the label categories and percentage values ​​of each label category in the target dialogue features, and determine the first numerical category predicted label; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one statistical data item corresponding to the user to be predicted and a grouping label corresponding to each statistical data item are determined, and a second numerical class prediction label is determined.

[0244] In one optional implementation, in the instructions executable by the computer-readable storage medium, when the predicted label includes a descriptive summary predicted label, obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Extract at least one dialogue analysis sample from the dialogue log; For each dialogue analysis sample, at least one non-data class label is extracted based on the target dialogue features corresponding to the dialogue analysis sample. For each dialogue analysis sample, at least one important node label is determined based on the dialogue summary information of that dialogue analysis sample.

[0245] In one optional implementation, the instructions executed by the computer-readable storage medium determine the grouping label corresponding to each statistical data item through the following steps: For each statistical data point, the mean of all speakers corresponding to that statistical data point and the four positional distances are calculated. For each statistical data point, the group label corresponding to that statistical data point is determined based on the relationship between the calculated group value corresponding to that statistical data point and the mean and the four positional intervals.

[0246] In one optional implementation, the instructions executed by the computer-readable storage medium determine the user-standardized text description information through the following steps: Based on the target dialogue features, the updated dialogue logs, the dialogue summary information, and at least one predicted label of the user to be predicted, determine the dialogue description information corresponding to each preset text description category; The user-standardized text description information is obtained by aggregating the dialogue description information corresponding to each preset text description category.

[0247] In one optional implementation, the personality prediction result of the user to be predicted includes the alignment result of the user to be predicted on each personality dimension.

[0248] In one optional implementation, the instructions executed by the computer-readable storage medium further include: The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted, and outputs dialogue assistance information for the user to be predicted, so that the agent can carry out interaction with the user to be predicted based on the dialogue assistance information. The dialogue assistance information includes at least one of the following: The historical personality information of the user to be predicted, the current emotional information of the user to be predicted, and the dialogue direction indication information.

[0249] By using the above method, after obtaining the dialogue voice information between the user to be predicted and the agent, the target dialogue features of the user to be predicted can be extracted based on the dialogue voice information and historical dialogue extraction information. The target dialogue features include not only text dialogue features, but also acoustic dialogue features and behavioral features. In this way, the prediction label of the user to be predicted can be determined from multiple dimensions, thereby determining the personality prediction result of the user to be predicted, which helps to improve the accuracy of personality prediction. Furthermore, in order to ensure that the agent can communicate more accurately with the user to be predicted, the large language model can also output dialogue auxiliary information while outputting the personality prediction result, thereby assisting the agent to demonstrate reasonable and accurate interaction with the user to be predicted.

[0250] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0251] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0252] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0253] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0254] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0255] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A personality prediction method based on voice dialogue, characterized in that, The personality prediction method includes: Acquire the voice dialogue information between the user to be predicted and the intelligent agent; Extracting target dialogue features of the user to be predicted from the dialogue voice information, and extracting dialogue logs from the dialogue voice information; wherein, the target dialogue features include text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; Update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one predicted label for the user to be predicted is obtained. The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model so that the large language model outputs the personality prediction result of the user to be predicted.

2. The personality prediction method according to claim 1, characterized in that, Extracting the target dialogue features of the user to be predicted from the dialogue voice information includes: Based on the dialogue voice information and the historical dialogue extraction information of the user to be predicted, the target dialogue features of the user to be predicted are extracted from the dialogue voice information.

3. The personality prediction method according to claim 1, characterized in that, Extracting the dialogue log from the dialogue voice information includes: The dialogue voice information is input into the automatic speech recognition system, the dialogue voice information is converted into text dialogue information, and each word contained in the text dialogue information and the word association information corresponding to each word are determined. Based on the text dialogue information and the timestamp information corresponding to each word, at least one response unit in the dialogue voice information is determined. The dialogue log is obtained based on the at least one response unit, each word, and the word association information corresponding to each word.

4. The personality prediction method according to claim 3, characterized in that, The step of determining at least one response unit in the dialogue voice information based on the text dialogue information and the timestamp information corresponding to each word includes: Based on the text dialogue information and the timestamp information corresponding to each word, determine the time interval and semantic continuity between at least two consecutive words; Based on the time interval between at least two consecutive words and semantic continuity, at least two consecutive words are merged to obtain the response unit.

5. The personality prediction method according to claim 4, characterized in that, The method of merging at least two consecutive words based on the time interval and semantic continuity to obtain the response unit includes: If the time interval between at least two consecutive words is greater than the first preset time interval, merge at least two consecutive words to obtain the first merged word group; Detect whether there is semantic continuity between at least two words in the first merged word group. If there is semantic continuity between at least two words, determine the first merged word group as a response unit. If there is no semantic continuity between at least two words, and the time interval between at least two consecutive words is greater than a second preset time interval, at least two consecutive words are merged to obtain a second merged word group, and the second merged word group is determined as a response unit.

6. The personality prediction method according to claim 3, characterized in that, The text dialogue features are extracted from the dialogue voice information through the following steps: The dialogue log is input into a pre-trained sentiment classifier so that the sentiment classifier outputs the sentiment label and sentiment intensity information corresponding to each response unit in the dialogue speech information; The dialogue log is input into a pre-trained emotion classifier so that the emotion classifier outputs the emotion label corresponding to each response unit in the dialogue voice information. The dialogue log is input into a pre-trained feedback classifier so that the feedback classifier outputs a feedback label corresponding to each response unit in the dialogue voice information.

7. The personality prediction method according to claim 3, characterized in that, The acoustic dialogue features are extracted from the dialogue speech information through the following steps: The dialogue speech information is input into a pre-trained speech event detection model so that the speech event detection model outputs at least one speech event contained in the dialogue speech information and the time information corresponding to each speech event. The dialogue speech information is input into a pre-trained speech emotion detection model so that the speech emotion detection model outputs the emotion classification label and emotion intensity information corresponding to each response unit in the dialogue speech information; The dialogue speech information is input into a pre-trained speech style detection model so that the speech style detection model outputs the speech style classification information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker attribute detection model so that the speaker attribute detection model outputs the speaker attribute information corresponding to the user to be predicted. The dialogue voice information is input into a pre-trained speaker detection model so that the speaker detection model outputs a speaker log. The dialogue speech information is input into a pre-trained acoustic event detection model so that the acoustic event detection model outputs at least one acoustic event contained in the dialogue speech information.

8. The personality prediction method according to claim 1, characterized in that, The behavioral features are extracted from the dialogue voice information through the following steps: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, target behavior detection is performed on the user to be predicted. Determine at least one target behavioral feature of the user to be predicted during the dialogue process; The target behavior includes at least one of the following: Interrupting conversations, disrupting dialogues, and engaging in conversations with other than the stated agent.

9. The personality prediction method according to claim 8, characterized in that, The following steps are used to determine the target behavioral feature of the user to be predicted during the dialogue as the interruption behavior: Based on the text dialogue features, the acoustic dialogue features, and the dialogue log, determine the overlap type of the dialogue speech between the user to be predicted and other speakers during the dialogue process; If the overlap type of the dialogue between the user to be predicted and other speakers is determined to be a partial overlap type, then the target behavioral feature of the user to be predicted during the dialogue is determined to be the interruption behavior.

10. The personality prediction method according to claim 2, characterized in that, The personality prediction method also includes: The textual dialogue features, the acoustic dialogue features, and the behavioral features are fused to obtain the fused target dialogue features.

11. The personality prediction method according to claim 10, characterized in that, The fusion of the textual dialogue features, acoustic dialogue features, and behavioral features to obtain the fused target dialogue features includes: For each response unit in the dialogue speech information, the non-repeating labels in the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are used as the speech feature labels corresponding to the response unit. For each response unit in the dialogue speech information, if there are duplicate labels in the text dialogue features, acoustic dialogue features and behavioral features of the response unit, the duplicate labels are fused together to determine the speech feature label corresponding to the response unit. By combining the speech feature labels corresponding to each response unit, the fused target dialogue features are obtained.

12. The personality prediction method according to claim 11, characterized in that, For each response unit in the dialogue speech information, if the text dialogue features, acoustic dialogue features, and behavioral features of the response unit have duplicate labels, the duplicate labels are fused to determine the speech feature label corresponding to the response unit, including: For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are consistent, the corresponding prediction results are used as the speech feature labels of the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is greater than the preset confidence threshold, the prediction result with the highest confidence level among each repeated label shall be used as the speech feature label corresponding to the response unit. For each response unit in the dialogue speech information, if the prediction results corresponding to the repeated labels of the text dialogue features, acoustic dialogue features, and behavioral features of the response unit are inconsistent, and if the difference between the confidence levels of each repeated label is not greater than a preset confidence threshold, the speech feature label corresponding to the response unit is determined based on the historical speech feature labels extracted from the historical dialogue information.

13. The personality prediction method according to claim 1, characterized in that, Update the conversation log using the following steps: Based on the target dialogue features, the historical dialogue logs corresponding to the user to be predicted are updated and supplemented to obtain the updated dialogue logs.

14. The personality prediction method according to claim 1, characterized in that, The prediction labels include numerical prediction labels and descriptive summary prediction labels; When the predicted label includes a numerical predicted label, obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Based on the fused target dialogue features, determine the label categories and percentage values ​​of each label category in the target dialogue features, and determine the first numerical category predicted label; Based on the fused target dialogue features, the updated dialogue logs, and the dialogue summary information, at least one statistical data item corresponding to the user to be predicted and a grouping label corresponding to each statistical data item are determined, and a second numerical class prediction label is determined.

15. The personality prediction method according to claim 14, characterized in that, When the predicted label includes a descriptive summary predicted label, obtaining at least one predicted label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information includes: Extract at least one dialogue analysis sample from the dialogue log; For each dialogue analysis sample, at least one non-data class label is extracted based on the target dialogue features corresponding to the dialogue analysis sample. For each dialogue analysis sample, at least one important node label is determined based on the dialogue summary information of that dialogue analysis sample.

16. The personality prediction method according to claim 14, characterized in that, The following steps are used to determine the group label corresponding to each statistical data item: For each statistical data point, the mean of all speakers corresponding to that statistical data point and the four positional distances are calculated. For each statistical data point, the group label corresponding to that statistical data point is determined based on the relationship between the calculated group value corresponding to that statistical data point and the mean and the four positional intervals.

17. The personality prediction method according to claim 1, characterized in that, The user-standardized text description information is determined through the following steps: Based on the target dialogue features, the updated dialogue logs, the dialogue summary information, and at least one predicted label of the user to be predicted, determine the dialogue description information corresponding to each preset text description category; The user-standardized text description information is obtained by aggregating the dialogue description information corresponding to each preset text description category.

18. The personality prediction method according to claim 1, characterized in that, The personality prediction results for the user to be predicted include the alignment results of the user on each personality dimension.

19. The personality prediction method according to claim 1, characterized in that, The personality prediction method also includes: The user-standardized text description information obtained by fusing the at least one prediction label is input into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted, and outputs dialogue assistance information for the user to be predicted, so that the agent can carry out interaction with the user to be predicted based on the dialogue assistance information. The dialogue assistance information includes at least one of the following: The historical personality information of the user to be predicted, the current emotional information of the user to be predicted, and the dialogue direction indication information.

20. A personality prediction device based on voice dialogue, characterized in that, The personality prediction device includes: The dialogue voice information acquisition module is used to acquire the dialogue voice information between the user to be predicted and the intelligent agent; The dialogue feature extraction module is used to extract the target dialogue features of the user to be predicted from the dialogue voice information, and to extract the dialogue log from the dialogue voice information; wherein, the target dialogue features include the text dialogue features, acoustic dialogue features, and behavioral features of the user to be predicted; The information update module is used to update the dialogue log and dialogue summary information of the user to be predicted based on the dialogue log and the target dialogue features. The prediction label determination module is used to obtain at least one prediction label for the user to be predicted based on the fused target dialogue features, the updated dialogue log, and the dialogue summary information. The personality prediction module is used to input standardized text description information of users obtained by fusing the at least one prediction label into a pre-trained large language model, so that the large language model outputs the personality prediction result of the user to be predicted.

21. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the personality prediction method based on voice dialogue as described in any one of claims 1 to 19.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the personality prediction method based on voice dialogue as described in any one of claims 1 to 19.

Images