Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium. The method comprises the steps that N first historical session features corresponding to N historical time periods of a first session object are acquired, and N second historical session features corresponding to N historical time periods of a second session object are acquired; and performing emotional feature completion processing on the N first historical session features to obtain N time sequence context session features. And according to the N second historical session features, performing emotional feature completion processing on the N first historical session features to obtain N interactive context session features. And according to the N time sequence context session features and the N interaction context session features, predicting a session emotion feature of the first session object at the current moment, and according to the session emotion feature, generating a session content of the second session object at the current moment. Through the method and the device, the prediction accuracy of the session emotion features can be improved, and the accuracy of the generated session content can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, sentiment prediction technology is also constantly improving. Sentiment prediction technology can be applied in various scenarios, such as intelligent emotional companionship, social dialogue, and game interaction. For example, in intelligent emotional companionship, sentiment prediction technology can predict the emotional characteristics of the person to be accompanied (i.e., the first conversational partner) at the current moment based on the conversational characteristics of that person. Based on the predicted emotional characteristics, the dialogue content for the intelligent emotional companion (i.e., the second conversational partner) at the current moment can be generated, and emotional companionship can be provided to the person to be accompanied through the generated dialogue content.

[0003] In practice, it has been found that current sentiment prediction methods rely on relatively limited information, resulting in low accuracy in predicting conversational sentiment features. Consequently, the generated dialogue content is not closely related to the user's actual emotions, leading to low accuracy in the generated dialogue content. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium to improve the prediction accuracy of conversational sentiment features and the accuracy of generated dialogue content.

[0005] One embodiment of this application provides a data processing method, including:

[0006] Obtain N first historical session features corresponding to the first session object in N historical time periods, and obtain N second historical session features corresponding to the second session object in the aforementioned N historical time periods; N is an integer greater than 1; the aforementioned N historical time periods are the time periods during which the aforementioned first session object and the aforementioned second session object had a conversation;

[0007] Sentiment feature completion processing is performed on the above N first historical conversation features to obtain N temporal context conversation features of the above first conversation object;

[0008] Based on the above N second historical conversation features, sentiment feature completion processing is performed on the above N first historical conversation features to obtain N interactive context conversation features between the above first conversation object and the above second conversation object.

[0009] Based on the aforementioned N temporal context conversation features and the aforementioned N interactive context conversation features, the conversation sentiment features of the first conversation object at the current moment are predicted, and the dialogue content of the second conversation object at the current moment is generated based on the aforementioned conversation sentiment features; the aforementioned current moment is the time after the aforementioned N historical time periods.

[0010] One embodiment of this application provides a data processing apparatus, including:

[0011] The acquisition module is used to acquire N first historical session features corresponding to N historical time periods for the first session object, and to acquire N second historical session features corresponding to the second session object in the aforementioned N historical time periods; N is an integer greater than 1; the aforementioned N historical time periods are the time periods during which the aforementioned first session object and the aforementioned second session object conduct a session;

[0012] The first processing module is used to perform sentiment feature completion processing on the above N first historical conversation features to obtain N temporal context conversation features of the above first conversation object.

[0013] The second processing module is used to perform sentiment feature completion processing on the N first historical conversation features based on the N second historical conversation features to obtain N interaction context conversation features between the first conversation object and the second conversation object.

[0014] The prediction module is used to predict the conversation sentiment characteristics of the first conversation object at the current moment based on the above N temporal context conversation characteristics and the above N interactive context conversation characteristics, and to generate the dialogue content of the second conversation object at the above current moment based on the above conversation sentiment characteristics; the above current moment is the time after the above N historical time periods.

[0015] One embodiment of this application provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0016] One embodiment of this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0017] In this application, by performing sentiment feature completion processing on N first historical conversation features of the first conversation object, N temporal context conversation features of the first conversation object are obtained. This involves completing the sentiment features of the first conversation object when it is not speaking, establishing a complete and continuous temporal context conversation sentiment feature for the first conversation object, thus overcoming the sentiment gap problem caused by traditional methods that rely solely on speaking time information. By performing sentiment feature completion processing on the N first historical conversation features based on the aforementioned N second historical conversation features, N interactive context conversation features between the first and second conversation objects are obtained. This not only completes the conversation sentiment features of the first conversation object when it is not speaking but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path. Furthermore, predicting the conversation sentiment features of the first conversation object at the current moment based on the N temporal context conversation features and the N interactive context conversation features improves the prediction accuracy of the conversation sentiment features. Predicting the dialogue content of the second conversation object based on the conversation sentiment features at the current moment improves the accuracy of the dialogue content. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0019] Figure 1 This is a schematic diagram of the structure of a data processing system provided in this application;

[0020] Figure 2 This is a flowchart illustrating a data processing method provided in this application. Figure 1 ;

[0021] Figure 3 This is a schematic diagram illustrating the process of dividing a historical time period as provided in this application;

[0022] Figure 4 This is a flowchart illustrating a data processing method provided in this application. Figure 2 ;

[0023] Figure 5 This is a schematic diagram illustrating a scenario for generating historical session features as provided in this application;

[0024] Figure 6 This is a flowchart illustrating a method for completing a modal feature set, as provided in this application.

[0025] Figure 7 This is a schematic diagram of a process for generating temporal context session features and interaction context session features provided in this application;

[0026] Figure 8 This is a flowchart illustrating a sentiment prediction process provided in this application;

[0027] Figure 9a This application provides a convergence analysis of a sentiment prediction network. Figure 1 ;

[0028] Figure 9b This application provides a convergence analysis of a sentiment prediction network. Figure 2 ;

[0029] Figure 9c This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 1 ;

[0030] Figure 9d This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 2 ;

[0031] Figure 9e This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 3 ;

[0032] Figure 9f This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 4 ;

[0033] Figure 10 This is a flowchart illustrating a data processing method provided in this application. Figure 3 ;

[0034] Figure 11 This is a schematic diagram of the structure of a data processing device provided in this application;

[0035] Figure 12 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0036] The embodiments of this application are described below with reference to the accompanying drawings.

[0037] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used only to distinguish different objects and not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0038] The term "embodiment" as used in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] It should be understood that in this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0040] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0041] First, let's introduce Figure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided in this application, such as... Figure 1 As shown in the diagram, the data processing system's structure generally includes a server 10 and a terminal device cluster, which may include multiple terminal devices. Figure 1As shown, taking a terminal device cluster containing three terminal devices as an example, the terminal device cluster can specifically include terminal device 11, terminal device 12, and terminal device 13. It can be understood that terminal device 11, terminal device 12, and terminal device 13 can all connect to server 10 via the network so that each terminal device can interact with server 10 through the network connection.

[0042] In this context, server 10 can refer to a backend device used to provide session services. For example, server 10 can provide a session platform to terminal devices. Specifically, server 10 can be used to authenticate objects logged into the session platform and provide session services to legitimate objects. This session platform can refer to a session application, a session webpage, a session applet, etc. All objects corresponding to terminal devices in the terminal device cluster can use the session services provided by the session platform.

[0043] The session service can include session services in real session scenarios and session services in virtual session scenarios.

[0044] It should be noted that a real-world conversation scenario refers to a conversation between two real conversation objects; a real conversation object can be an object corresponding to a terminal device. Specifically, after any two terminal devices in a terminal device cluster log into the conversation platform, the objects corresponding to those two terminal devices can engage in a conversation through the conversation services provided by the platform. Real-world conversation scenarios can include social dialogue scenarios, etc.

[0045] It should be noted that a virtual conversation scenario can refer to a situation where real conversation objects and virtual conversation objects engage in conversation. Virtual conversation objects can be conversation objects generated by the conversation platform based on Artificial Intelligence (AI) technology, and can include virtual assistants (such as intelligent question-and-answer robots), virtual characters (such as virtual idols, virtual customer service personnel, etc.), etc. Specifically, after any terminal device in the terminal device cluster logs into the conversation platform, the corresponding object can engage in conversation with virtual conversation objects on the platform through the conversation services provided by the platform. Virtual conversation scenarios can include intelligent emotional companionship scenarios, game interaction scenarios, etc.

[0046] Specifically, server 10 can predict the conversation sentiment characteristics of the first conversation object at the current moment based on N first historical conversation features corresponding to N historical time periods obtained during the conversation, and N second historical conversation features corresponding to the second conversation object at the same N historical time periods. Furthermore, server 10 can generate the dialogue content of the second conversation object at the current moment based on the predicted conversation sentiment characteristics.

[0047] In a real-world session scenario, the first session object can be an object corresponding to any terminal device in the terminal device cluster, and the second session object can be an object corresponding to any terminal device in the terminal device cluster; in particular, the terminal device corresponding to the first session object and the terminal device corresponding to the second session object are not the same terminal device.

[0048] In the virtual session scenario, the first session object can be an object corresponding to any terminal device in the terminal device cluster, and the second session object can be a virtual session object.

[0049] It should be noted that a server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0050] It should be noted that terminal equipment is also called terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal equipment can be smart home appliances, handheld devices with wireless communication capabilities (such as smartphones and tablets), computing devices (such as personal computers (PCs)), in-vehicle terminals, smart voice interaction devices, wearable devices, or other smart devices, but is not limited to these.

[0051] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in this application. Figure 1 .like Figure 2 As shown, this method can be derived from... Figure 1 It can be executed by any terminal device in the system, or by... Figure 1 Server 10 in the middle can be used to execute it, and it can also be executed by Figure 1The terminal device and server 10 work together to execute the data processing method. The device used to execute this data processing method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0052] S101. Obtain N first historical session features corresponding to the first session object in N historical time periods, and obtain N second historical session features corresponding to the second session object in N historical time periods.

[0053] In this application, a computer device can obtain, through local storage or network download, N first historical session features corresponding to N historical time periods for the first session object and N second historical session features corresponding to the second session object during the session between the first session object and the second session object.

[0054] It should be noted that the aforementioned N historical time periods can be the time periods during which the first session object and the second session object conduct their conversation; that is, the aforementioned N historical time periods can be obtained by dividing the process of the conversation between the first session object and the second session object into time periods. Each historical time period is a time period preceding the current moment, and N is an integer greater than 1.

[0055] In one embodiment, the computer device can divide the conversation process between the first conversation object and the second conversation object into time periods based on the output process of the dialogue content corresponding to the first conversation object and the second conversation object respectively, to obtain N historical time periods; that is, the output process of one dialogue content corresponds to one historical time period.

[0056] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the process of dividing historical time periods provided in this application. Figure 3 The system includes a conversation interface 31, which serves as the interface for the conversation between the first and second conversation objects. The conversation interface 31 includes three dialogue contents: dialogue contents 31a and 31c for the first conversation object, and dialogue content 31b for the second conversation object. The computer device can divide the conversation between the first and second conversation objects into three historical time periods based on the output processes corresponding to dialogue contents 31a, 31b, and 31c. Specifically, the output process of dialogue content 31a corresponds to the first historical time period, the output process of dialogue content 31b corresponds to the second historical time period, and the output process of dialogue content 31c corresponds to the third historical time period.

[0057] It should be noted that a historical time period corresponds to a first historical session feature, and a historical time period corresponds to a second historical session feature; specifically, the i-th historical time period corresponds to the first historical session feature i, and the i-th historical time period corresponds to the second historical session feature i; where i is a positive integer less than or equal to N.

[0058] The conversation process typically involves consecutive speaking sessions between the first and second conversation objects. This means the conversation process includes periods of speaking by the conversation objects (i.e., the first and second conversation objects) and periods of not speaking. Therefore, the N historical time periods can include both the speaking period and the non-speaking period of the first conversation object. In addition, the N historical time periods can also include periods of missing data for the first conversation object; when a device malfunction occurs during a certain historical time period, it may cause the first conversation object to miss one or more modal features during that period. In this application, this historical time period can be referred to as the data missing period.

[0059] In one embodiment, when a certain historical time period is the speaking time period of the first conversation object, the first historical conversation features of the first conversation object during that historical time period may include the video modal features, audio modal features, and text modal features of the first conversation object during that historical time period. In this case, based on the video modal features, audio modal features, and text modal features of the first conversation object during that historical time period, the conversational sentiment characteristics of the first conversation object during that historical time period can be accurately reflected. That is, when a certain historical time period is the speaking time period of the first conversation object, the first historical conversation features of the first conversation object during that historical time period can accurately reflect the conversational sentiment characteristics of the first conversation object during that historical time period.

[0060] Next, when a historical time period is a period during which the first conversation subject did not speak, or when a historical time period is a period of missing data for the first conversation subject, it will result in the first conversation subject lacking one or more modal features during that historical time period. In this case, it is necessary to fill in the missing modal features. The first historical conversation feature of the first conversation subject during that historical time period can include the first modal feature and the filled-in second modal feature. The filled-in second modal feature can be obtained by dimensionality completion of the first modal feature, that is, the dimension of the filled-in second modal feature is the same as the dimension of the first modal feature. At this time, based on the first modal feature and the filled-in second modal feature of the first conversation subject during that historical time period, it is not possible to accurately reflect the conversation sentiment feature of the first conversation subject during that historical time period. That is, when a historical time period is a period during which the first conversation subject did not speak, or when a historical time period is a period of missing data for the first conversation subject, the first historical conversation feature of the first conversation subject during that historical time period cannot accurately reflect the conversation sentiment feature of the first conversation subject during that historical time period.

[0061] Next, when a certain historical time period is the period during which the first conversation object did not speak, the first modal feature can refer to video modal features, and the second modal feature can refer to audio modal features and text modal features. When a certain historical time period is the period during which the first conversation object's data is missing, the first modal feature can refer to any one or any two of video modal features, audio modal features, and text modal features, and the second modal feature can refer to modal features other than the first modal feature among video modal features, audio modal features, and text modal features.

[0062] Similarly, it can be inferred that when a certain historical time period is the speaking time period of the second conversation object, the second historical conversation feature of the second conversation object in that historical time period can accurately reflect the conversation sentiment feature of the second conversation object in that historical time period. When a certain historical time period is the time period in which the second conversation object did not speak, or when a certain historical time period is the time period in which the data of the second conversation object is missing, the second historical conversation feature of the second conversation object in that historical time period cannot accurately reflect the conversation sentiment feature of the second conversation object in that historical time period.

[0063] Specifically, when a certain historical time period is the speaking time period of the second conversation object, the second historical conversation features of the second conversation object during that historical time period may include the video modal features, audio modal features, and text modal features of the second conversation object during that historical time period. When a certain historical time period is the non-speaking time period of the second conversation object, or when a certain historical time period is the data missing time period of the second conversation object, the second historical conversation features of the second conversation object during that historical time period may include the first modal feature and the supplemented second modal feature.

[0064] It should be noted that video modal features can reflect at least one of the following: facial expressions, body movements, clothing, and environment of the corresponding conversational object (i.e., the first conversational object and the second conversational object); audio modal features can reflect at least one of the following: voice timbre, voice tone, and voice content of the corresponding conversational object; and text modal features can reflect the dialogue content of the corresponding conversational object.

[0065] In real-world conversation scenarios, when a historical time period corresponds to the speaking time of a conversation participant, the video modal characteristics of that participant during that period can be extracted from video data captured by a video capture component deployed on the corresponding terminal device; this video capture component can be a camera. The audio modal characteristics of that participant during that historical time period can be extracted from audio data recorded by an audio capture component deployed on the corresponding terminal device; this audio capture component can be a microphone. The text modal characteristics of that participant during that historical time period can be obtained by extracting text features from the dialogue content in the conversation interface.

[0066] In a virtual conversation scenario, when a certain historical time period is the speaking time period of the first conversation object, the methods for obtaining the video modal features, audio modal features, and text modal features of the first conversation object during that historical time period are the same as those for obtaining the video modal features, audio modal features, and text modal features of the first conversation object during that historical time period in a real conversation scenario, and will not be repeated here. When a certain historical time period is the speaking time period of the second conversation object (i.e., the virtual conversation object), the video modal features of the second conversation object during that historical time period can be extracted from the video data of the virtual conversation object generated based on AI technology. The audio modal features of the second conversation object during that historical time period can be obtained by extracting text features from the dialogue content of the second conversation object in the conversation interface, which is generated based on AI technology.

[0067] In one embodiment, when a historical time period is the speaking time period of the first conversation object, the first historical conversation feature of the first conversation object during that historical time period may include the video modal feature, audio modal feature, and text modal feature of the first conversation object during that historical time period. When a historical time period is the non-speaking time period of the first conversation object, the first historical conversation feature of the first conversation object during that historical time period may include the video modal feature of the first conversation object during that historical time period. When a historical time period is the data missing time period of the first conversation object, the first historical conversation feature of the first conversation object during that historical time period may include any one or any two of the video modal feature, audio modal feature, and text modal feature of the first conversation object during that historical time period.

[0068] In one embodiment, when a historical time period corresponds to the speaking time period of the second conversation object, the second historical conversation feature of the second conversation object during that historical time period may include video modal features, audio modal features, and text modal features of the second conversation object during that historical time period. When a historical time period corresponds to the non-speaking time period of the second conversation object, the second historical conversation feature of the second conversation object during that historical time period may include video modal features of the second conversation object during that historical time period. When a historical time period corresponds to the data missing time period of the second conversation object, the second historical conversation feature of the second conversation object during that historical time period may include any one or any two of the video modal features, audio modal features, and text modal features of the second conversation object during that historical time period.

[0069] S102. Perform sentiment feature completion processing on the N first historical conversation features to obtain the N temporal context conversation features of the first conversation object.

[0070] As the above analysis shows, in this application, when the historical time period is a time when the first conversational subject is not speaking, or when the historical time period is a data-missing period, the first historical conversational features corresponding to that historical time period cannot accurately reflect the conversational sentiment characteristics of the first conversational subject during that historical time period. Therefore, the computer device can perform sentiment feature completion processing on N first historical conversational features to obtain N temporal context conversational features of the first conversational subject. That is, sentiment feature completion processing refers to completing the conversational sentiment characteristics of the first conversational subject during the time period when they are not speaking, or during the data-missing period. This is beneficial for establishing complete and continuous temporal context conversational sentiment characteristics for the first conversational subject pair, making up for the sentiment gap problem caused by traditional methods that only rely on speaking time information, and thus improving the prediction accuracy of conversational sentiment features.

[0071] It should be noted that the emotional categories of conversational sentiment features can include positive sentiment categories, negative sentiment categories, and neutral sentiment categories. Positive sentiment features can include happiness, joy, excitement, etc., negative sentiment features can include sadness, grief, frustration, etc., and neutral sentiment features can include calmness, indifference, neutrality, etc.

[0072] It should be noted that each temporal context conversation feature can be used to reflect the conversational sentiment characteristics of the first conversation object in the corresponding historical time period. The temporal context conversation feature of the i-th historical time period can be obtained by supplementing the sentiment features based on the first historical conversation feature of the i-th historical time period and the first historical conversation feature with adjacent relationships.

[0073] The first historical session feature that is adjacent to it may include at least one of the first historical session features whose corresponding historical time period is before the i-th historical time period and the first historical session features whose corresponding historical time period is after the i-th historical time period.

[0074] It should be noted that the first historical conversation feature that is adjacent to it can be called the first context conversation feature of the temporal context conversation feature of the i-th historical time period. Therefore, the temporal context conversation feature of the i-th historical time period is obtained by supplementing the sentiment features based on the first historical conversation feature and the first context conversation feature of the i-th historical time period. The temporal context conversation feature obtained in this way can more accurately reflect the conversation sentiment features of the first conversation object in each historical time period, making the conversation sentiment features of each historical time period continuous in time, and smoother and more realistic.

[0075] S103. Based on the N second historical conversation features, perform sentiment feature completion processing on the N first historical conversation features to obtain N interactive context conversation features between the first conversation object and the second conversation object.

[0076] In this application, the conversational sentiment characteristics of the first conversational subject during a certain historical time period are the result of dynamic evolution and interaction among multiple parties. That is, during the conversation, the conversational sentiment characteristics of the first conversational subject are influenced by the conversational sentiment characteristics of the second conversational subject. Therefore, the computer device can perform sentiment feature completion processing on N first historical conversational features based on N second historical conversational features, thereby completing the conversational sentiment characteristics of the first conversational subject during periods when it is not speaking or during periods of missing data, thus obtaining N interactive context conversational features between the first and second conversational subjects. This not only completes the conversational sentiment characteristics of the first conversational subject when it is not speaking, but also further enriches the conversational sentiment characteristics of the first conversational subject when it is speaking, thereby constructing a more complete and coherent evolutionary path of conversational sentiment characteristics.

[0077] It should be noted that each interaction context session feature can be used to reflect the conversation sentiment characteristics of the first conversation object in the corresponding historical time period. The interaction context session feature of the i-th historical time period can be obtained by supplementing the sentiment features based on the first historical session feature of the i-th historical time period and the second historical session feature that has an adjacent relationship with it.

[0078] The second historical session feature that is adjacent to it may include at least one of the following: the second historical session feature whose corresponding historical time period is before the i-th historical time period, the second historical session feature whose corresponding historical time period is after the i-th historical time period.

[0079] It should be noted that the second historical conversation feature that is adjacent to it can be called the second context conversation feature of the interaction context conversation feature of the i-th historical time period. Therefore, the interaction context conversation feature of the i-th historical time period is obtained by supplementing the sentiment features based on the first historical conversation feature and the second context conversation feature of the i-th historical time period. The interaction context conversation feature obtained in this way can more accurately reflect the conversation sentiment features of the first conversation object in each historical time period.

[0080] S104. Based on N temporal context conversation features and N interactive context conversation features, predict the conversation sentiment features of the first conversation object at the current moment, and generate the dialogue content of the second conversation object at the current moment based on the conversation sentiment features.

[0081] In this application, a computer device can construct a complete contextual sentiment expression of a first conversational object during a conversation based on N temporal contextual conversational features and N interactive contextual conversational features. This allows for the prediction of the first conversational object's sentiment characteristics at the current moment, improving the predictability of the sentiment characteristics. Furthermore, the computer device can generate dialogue content for a second conversational object at the current moment that matches the sentiment characteristics of the first conversational object. The dialogue content generated using highly accurate prediction of sentiment characteristics is more relevant to the user's actual emotions, meaning the generated dialogue content is highly accurate and can effectively provide emotional support to the first conversational object. Here, "current moment" can refer to a time after the aforementioned N historical time periods.

[0082] It should be noted that the dialogue content of the second conversational partner at the current moment may include quick hint text, which can be used to help the second conversational partner quickly understand the emotional characteristics of the first conversational partner at the current moment.

[0083] If the second conversation object is a real conversation object, the computer device can send the conversation content of the second conversation object at the current moment to the terminal device corresponding to the second conversation object. If the second conversation object is a virtual conversation object, the computer device can directly generate the video data, audio data, and text data of the second conversation object at the current moment based on the shortcut prompt text.

[0084] In this application, by performing sentiment feature completion processing on N first historical conversation features of the first conversation object, N temporal context conversation features of the first conversation object are obtained. This involves completing the sentiment features of the first conversation object when it is not speaking, establishing a complete and continuous temporal context conversation sentiment feature for the first conversation object, thus overcoming the sentiment gap problem caused by traditional methods that rely solely on speaking time information. By performing sentiment feature completion processing on the N first historical conversation features based on the aforementioned N second historical conversation features, N interactive context conversation features between the first and second conversation objects are obtained. This not only completes the conversation sentiment features of the first conversation object when it is not speaking but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path. Furthermore, predicting the conversation sentiment features of the first conversation object at the current moment based on the N temporal context conversation features and the N interactive context conversation features improves the prediction accuracy of the conversation sentiment features. Predicting the dialogue content of the second conversation object based on the conversation sentiment features at the current moment improves the accuracy of the dialogue content.

[0085] Please see Figure 4 , Figure 4This is a flowchart illustrating a data processing method provided in this application. Figure 2 .like Figure 4 As shown, this method can be derived from... Figure 1 It can be executed by any terminal device in the system, or by... Figure 1 Server 10 in the middle can be used to execute it, and it can also be executed by Figure 1 The terminal device and server 10 work together to execute the data processing method. The device used to execute this data processing method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0086] S201. Obtain N first historical session features corresponding to the first session object in N historical time periods, and obtain N second historical session features corresponding to the second session object in the N historical time periods.

[0087] In one embodiment, obtaining N first historical session features corresponding to N historical time periods for the first session object includes: the computer device can obtain N modal feature sets corresponding to the first session object in the N historical time periods; the computer device can fuse the modal features in each modal feature set of the first session object to obtain the first historical session features corresponding to the first session object in each of the aforementioned historical time periods. Please refer to... Figure 5 , Figure 5 This is a schematic diagram illustrating a scenario for generating historical session features, as provided in this application. Figure 5 As shown, Figure 5 The dataset includes video data 51a, audio data 52a, and text data 53a (i.e., "I don't like my classmates at the new school; they all ignore me.") of the first conversation subject in the i-th historical time period. The computer device can extract the video modal feature 51b of the first conversation subject in the i-th historical time period from the video data 51a, the audio modal feature 52b of the first conversation subject in the i-th historical time period from the audio data 52a, and the text modal feature 53b of the first conversation subject in the i-th historical time period from the text data 53a. The computer device can input the video modal feature 51b, audio modal feature 52b, and text modal feature 53b into the feature fusion network 50 to obtain the first historical conversation feature 54 of the first conversation subject in the i-th historical time period.

[0088] Specifically, when a historical time period corresponds to the speaking time of the first conversation object, the modal feature set of the first conversation object during that historical time period may include video modal features, audio modal features, and text modal features of the first conversation object during that historical time period. Conversely, when a historical time period is a period during which the first conversation object does not speak, or when a historical time period is a period of missing data for the first conversation object, one or more modal features of the first conversation object will be missing from the modal feature set of that historical time period.

[0089] In one embodiment, after obtaining N modal feature sets corresponding to N historical time periods for the first session object, the computer device can perform modal feature completion processing on the N modal feature sets of the first session object to obtain N completed feature sets for the first session object. Then, the computer device can fuse the modal features in each of the completed feature sets of the first session object to obtain the first historical session features corresponding to each of the aforementioned historical time periods, i.e., obtain the N first historical session features corresponding to the first session object in the N historical time periods. By performing modal feature completion processing, the modal features of the first session object in the periods of no speaking and periods of missing data are dynamically completed, which is beneficial for establishing complete and continuous temporal context features for the user.

[0090] In one embodiment, the process of generating the N complete feature sets of the first session object may include: a computer device performing feature integrity detection on each of the N modal feature sets of the first session object to obtain a detection result for the first session object; when the detection result of the first session object indicates that there is a target feature set in the N modal feature sets that does not have complete features, the computer device can use a feature completion network to perform feature completion on the target feature set in the N modal feature sets that does not have complete features according to the detection result of the first session object, to obtain the completed target feature set corresponding to the first session object; the remaining feature set of the first session object and the completed target feature set corresponding to the first session object are determined as the N complete feature sets of the first session object. Feature integrity detection helps to accurately detect modal feature sets in the N modal feature sets of the first session object that do not have complete features, avoids invalid completion of complete feature sets, and prevents omission of modal feature sets that need to be completed.

[0091] The remaining feature set of the first session object is the set of modal features other than the target feature set among the N modal feature sets of the first session object.

[0092] It should be noted that the lack of feature completeness in the modal feature set of the first session object can include any of the following situations: 1. The modal feature set of the first session object is empty, that is, the modal feature set of the first session object does not include any modal feature; 2. The modal feature set of the first session object includes any one or any two of video modal features, audio modal features, and text modal features.

[0093] It should be noted that the reason why the modal feature set of the first session object does not have feature completeness may include any of the following: 1. The corresponding historical time period is the time period during which the first session object did not speak; 2. At least one of the video acquisition component and audio acquisition component malfunctions during the corresponding historical time period.

[0094] In one embodiment, the specific process for detecting the feature integrity of N sets of first modal features may include: a computer device may download or obtain feature integrity conditions from a network or local storage; the feature integrity conditions include k modal features, where k is a positive integer. For each set of modal features of the first session object, when the computer device detects that the modal feature set does not have at least one of the k modal features, it can determine that the modal feature set does not have feature integrity; when the computer device detects that the modal feature set has the k modal features, it can determine that the modal feature set has feature integrity. For the N sets of modal features of the first session object, the above operations are performed sequentially. When the computer device detects that there is a modal feature set in the N sets of modal features of the first session object that does not have at least one of the k modal features, the modal feature set in the N sets of modal features of the first session object that does not have at least one of the k modal features is determined as the target feature set of the first session object; a detection result is generated to indicate that there is a target feature set in the N sets of modal features of the first session object that does not have feature integrity. By providing objective and accurate detection standards for feature integrity conditions, we can avoid the problems of missed detections and false detections caused by inconsistent detection standards, and further improve the efficiency and quality of the entire feature completion process.

[0095] It should be noted that feature integrity conditions can be default values ​​in computer equipment or manually set.

[0096] In one embodiment, the feature completeness of any modal feature set of the first session object means that the modal feature set includes both first modal features and second modal features.

[0097] The aforementioned feature completion process for the target feature set may include: when the detection result of the first session object still indicates that the target feature set contains a first modal feature but not a second modal feature, the computer device can input the first modal feature in the target feature set into a feature completion network. The feature completion network determines the target feature dimension of the first modal feature in the target feature set and the modality category of the second modal feature. The computer device can search for feature completion parameters for completing the second modal feature in the feature completion parameter set based on the modality category of the second modal feature. Based on the feature completion parameters, a second modal feature with the same feature dimension as the target feature is generated. The computer device can then determine the completed target feature set of the first session object based on the first modal feature in the target feature set and the generated second modal feature. Modal features with the same dimension do not require additional dimension adjustment in subsequent feature fusion processing, which helps reduce the time consumed by feature dimension adjustment.

[0098] It should be noted that modal features include video modal features, audio modal features, and text modal features. Modal features of the video modal feature category can be called video modal features, modal features of the audio modal feature category can be called audio modal features, and modal features of the text modal feature category can be called text modal features.

[0099] In one embodiment, when the detection result of the first session object still indicates that the target feature set does not contain the first modal feature and the second modal feature, the computer device can obtain a preset feature dimension, and according to the modal categories corresponding to the first modal feature and the second modal feature, search in the feature completion parameter set for completing the first modal feature and for completing the second modal feature. The computer device can input the preset feature dimension, the feature completion parameters for completing the first modal feature, and the feature completion parameters for completing the second modal feature into the feature completion network. The computer device can generate a first modal feature with the preset feature dimension according to the feature completion parameters for completing the first modal feature; and generate a second modal feature with the preset feature dimension according to the feature completion parameters for completing the second modal feature. The computer device can determine the completed target feature set of the first session object according to the first modal feature and the second modal feature with the preset feature dimension.

[0100] In one embodiment, the feature completion network described above is learnable, meaning that the network parameters (such as feature completion parameters) in the feature completion network are learnable. When training the initial feature completion network, the network parameters in the initial feature completion network can be adjusted to train the initial feature completion network, thereby obtaining the final feature completion network.

[0101] In one embodiment, the explanation of the process for obtaining N second historical session features of the second session object can be found in the explanation of the process for obtaining N first historical session features of the first session object above, and will not be repeated here.

[0102] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating a method for completing a modal feature set, as provided in this application. Figure 6 As shown, the above process includes the following steps:

[0103] S601, Obtain the first modality feature set and the second modality feature set.

[0104] In this application, Figure 6 The system includes a conversation interface 61, which serves as the interface for a conversation between a first conversation object and a second conversation object. The first conversation object is a real conversation object, and the second conversation object is a virtual conversation object. The conversation interface 61 includes six dialogue contents: dialogue contents 61a, 61d, and 61f for the first conversation object, and dialogue contents 61b, 61c, and 61e for the second conversation object. The computer device can divide the conversation between the first and second conversation objects into six historical time periods based on the output processes corresponding to each of the six dialogue contents. Specifically, the output process of dialogue content 61a corresponds to the first historical time period, the output process of dialogue content 61b corresponds to the second historical time period, the output process of dialogue content 61c corresponds to the third historical time period, the output process of dialogue content 61d corresponds to the fourth historical time period, the output process of dialogue content 61e corresponds to the fifth historical time period, and the output process of dialogue content 61f corresponds to the sixth historical time period.

[0105] Next, the computer device can extract video modal features A11 from the video data of the first conversational object captured by the video capture component within the first historical time period; the computer device can extract audio modal features B11 from the audio data of the first conversational object captured by the audio capture component within the first historical time period; the computer device can extract the text content (i.e., text content of the dialogue content 61a) from the audio data of the first conversational object captured by the audio capture component within the first historical time period. Figure 6The text modality C11 is obtained by extracting features from the sentence "I don't like my classmates at the new school, they all ignore me." Based on the video modality A11, audio modality B11, and text modality C11, the first modality feature set corresponding to the first historical time period is determined. The computer device can extract video modality A21 from the video data of the second conversation object generated through AI technology within the first historical time period; based on the video modality feature A21, the second modality feature set corresponding to the first historical time period is determined. Here, the first modality feature set is the modality feature set of the first conversation object, and the second modality feature set is the modality feature set of the second conversation object.

[0106] The computer device can extract video modal features A12 from the video data of the first conversation object captured by the video capture component during the second historical time period; based on the video modal features A12, it determines the first modal feature set corresponding to the second historical time period. The computer device can extract video modal features A22 from the video data of the second conversation object generated by AI technology during the second historical time period; the computer device can extract audio modal features B22 from the audio data of the second conversation object generated by AI technology during the second historical time period; the computer device can also extract audio modal features B22 from the text content (i.e., the text content of the dialogue content 61b) of the dialogue content 61b. Figure 6 Feature extraction is performed on the phrase "It sounds like you feel lonely" to obtain text modality feature C22. Based on video modality feature A22, audio modality feature B22, and text modality feature C22, the second modality feature set corresponding to the second historical time period is determined. For the third, fourth, fifth, and sixth historical time periods, the above operations are performed sequentially to obtain the first and second modality feature sets corresponding to the third, fourth, fifth, and sixth historical time periods, respectively.

[0107] Specifically, the first modal feature set corresponding to the third historical time period includes audio modal feature A13, and the second modal feature set corresponding to the third historical time period includes audio modal feature A23, audio modal feature B23, and text modal feature C23. The first modal feature set corresponding to the fourth historical time period includes audio modal feature A14, audio modal feature B14, and text modal feature C14, and the second modal feature set corresponding to the fourth historical time period includes audio modal feature A24. The first modal feature set corresponding to the fifth historical time period includes audio modal feature A15, and the second modal feature set corresponding to the fifth historical time period includes audio modal feature A25, audio modal feature B25, and text modal feature C25. The first modal feature set corresponding to the sixth historical time period includes audio modal feature A16, audio modal feature B16, and text modal feature C16, and the second modal feature set corresponding to the sixth historical time period includes audio modal feature A26. The display effect of the first modality feature set and the second modality feature set corresponding to the six historical time periods acquired by the computer device, such as... Figure 6 The display interface is shown in 62.

[0108] S602, Generate the first and second complete feature sets.

[0109] In this application, a computer device can perform feature completion processing on the first modal feature sets corresponding to six historical time periods respectively, to obtain six first completed feature sets; and perform feature completion processing on the second modal feature sets corresponding to the six historical time periods respectively, to obtain six second completed feature sets. The first completed feature sets are the completed feature sets of the first session object, and the second completed feature sets are the completed feature sets of the second session object.

[0110] Specifically, if the feature integrity condition indicates that any first modal feature set has feature integrity, it means that the first modal feature set includes video modal features, audio modal features, and text modal features; and if any second modal feature set has feature integrity, it means that the second modal feature set includes video modal features, audio modal features, and text modal features. Since the first modal feature sets corresponding to the 1st historical time period, the 4th historical time period, and the 6th historical time period all include video modal features, audio modal features, and text modal features, the computer device can determine that the first modal feature sets corresponding to the 1st historical time period, the 4th historical time period, and the 65th historical time period have feature integrity. The computer device can determine the first modal feature sets corresponding to the 1st historical time period, the 4th historical time period, and the 6th historical time period as the first complete feature sets corresponding to the 1st historical time period, the 4th historical time period, and the 6th historical time period, respectively.

[0111] Since the first modal feature set corresponding to the second historical time period includes video modal feature A12 but does not include audio modal features and text modal features, the computer device can determine that the first modal feature set corresponding to the second historical time period lacks feature completeness. The computer device can input the video modal feature A12 into a feature completion network, using the feature completion network to determine the target feature dimension of the video modal feature A12 and the modal categories (i.e., audio modal category and text modal category) corresponding to the audio and text modal features, respectively. The computer device can search for feature completion parameters for completing the audio modal category and for completing the text modal category from the feature completion parameter set; based on the feature completion parameters for completing the audio modal feature, it generates an audio modal feature B12 with the same feature dimension as the target feature dimension; and based on the feature completion parameters for completing the text modal feature, it generates a text modal feature C12 with the same feature dimension as the target feature dimension. The computer device can use the video modal feature A12, audio modal feature B12, and text modal feature C12, as well as the first complete feature set corresponding to the second historical time period.

[0112] Performing the above operations on the first modal feature sets corresponding to the 3rd and 5th historical time periods respectively yields the first completion feature sets corresponding to the 3rd and 5th historical time periods respectively. The first completion feature set corresponding to the 3rd historical time period includes video modal feature A13, audio modal feature B13, and text modal feature C13; the first completion feature set corresponding to the 5th historical time period includes video modal feature A15, audio modal feature B15, and text modal feature C15.

[0113] Since the second modal feature sets corresponding to the second, third, and fifth historical time periods each include video modal features, audio modal features, and text modal features, the second modal feature sets corresponding to the second, third, and fifth historical time periods can be determined as the second complement feature sets corresponding to the second, third, and fifth historical time periods, respectively.

[0114] Similarly, performing the above operations on the second modal feature sets corresponding to the 1st, 4th, and 6th historical time periods respectively yields the second completion feature sets corresponding to the 1st, 4th, and 6th historical time periods respectively. Specifically, the second completion feature set corresponding to the 1st historical time period includes video modal feature A21, audio modal feature B21, and text modal feature C21; the second completion feature set corresponding to the 4th historical time period includes video modal feature A24, audio modal feature B24, and text modal feature C24; and the second completion feature set corresponding to the 6th historical time period includes video modal feature A26, audio modal feature B26, and text modal feature C26. The display effect of the first and second completion feature sets corresponding to the six historical time periods generated by the computer device is as follows: Figure 6 The display interface is shown in 63.

[0115] In this application, in order to distinguish between the completion feature set of the first session object and the completion feature set of the second session object, the completion feature set of the first session object can be referred to as the first completion feature set, and the completion feature set of the second session object can be referred to as the second completion feature set.

[0116] In one embodiment, each first complement feature set includes k modal features; k is a positive integer.

[0117] The above-mentioned fusion of modal features from the aforementioned N first complement feature sets to obtain N first historical session features corresponding to the aforementioned N historical time periods for the aforementioned first session object includes: The computer device can obtain the modal categories corresponding to k modal features in the first complement feature set i through a feature projection network; wherein, the aforementioned first complement feature set i is the first complement feature set corresponding to the i-th historical time period; i is a positive integer less than or equal to N. The computer device can obtain the feature projection parameters corresponding to the k modal features from the feature projection parameter set according to the modal categories corresponding to the k modal features. The computer device can project the k modal features to a feature space of the same feature dimension according to the feature projection parameters corresponding to the k modal features, that is, adjust the feature dimensions corresponding to the k modal features according to the feature projection parameters corresponding to the k modal features so that the feature dimensions corresponding to the k modal features are all the same, thus obtaining k modal projection features. The computer device can fuse the aforementioned k modal projection features to obtain the first historical session features corresponding to the aforementioned first session object in the aforementioned i-th historical time period. By sequentially performing the above operations on N first historical session features, the computer device can obtain N first historical session features corresponding to the first session object in the above N historical time periods.

[0118] In one embodiment, the above-mentioned fusion of the k modal projection features to obtain the first historical conversation feature corresponding to the first conversation object in the i-th historical time period includes: the computer device obtaining the feature weights corresponding to the k modal projection features according to the modal categories corresponding to the k modal projection features respectively; performing a weighted summation of the k modal projection features according to the feature weights corresponding to the k modal projection features respectively, that is, multiplying each modal projection feature and its feature weight to obtain the product result corresponding to each modal projection feature; summing the product results corresponding to the N modal projection features respectively to obtain the first historical conversation feature corresponding to the first conversation object in the i-th historical time period. By dynamically allocating and adjusting the contribution of different modal projection features through feature weights, the first historical conversation feature can more accurately reflect the conversational sentiment characteristics of the first conversation object, which is beneficial to improving the accuracy of the first historical conversation feature.

[0119] It should be noted that the modal projection features of the video modal category can be called video modal projection features, the modal projection features of the audio modal category can be called audio modal projection features, and the modal projection features of the text modal category can be called text modal projection features.

[0120] It should be noted that feature weights can be used to reflect the contribution of corresponding modality features in the fusion process. A larger feature weight indicates a greater contribution, meaning the corresponding modality feature is more important; a smaller feature weight indicates a smaller contribution, meaning the corresponding modality feature is less important. Specifically, the more important a modality feature is, the more accurate the conversational sentiment characteristics it reflects; conversely, the less important a modality feature is, the less accurate the conversational sentiment characteristics it reflects.

[0121] In one embodiment, k modal projection features corresponding to the first complement feature set i can be calculated based on k modal features in the first complement feature set i using formula (1). In another embodiment, f modal projection features corresponding to the second complement feature set i can be calculated based on f modal features in the second complement feature set i using formula (1). The formula (1) is shown below.

[0122]

[0123] in, For the input modal features, The 's' in the input modality feature is used to indicate the session object corresponding to the input modality feature. In this context, 'm' indicates the modality category corresponding to the input modal feature. In this context, 'i' indicates the historical time period corresponding to the input modal feature. In this context, s can be either a or b. When s is a, it indicates that the session object corresponding to the input modal feature is the first session object; When s is b, it indicates that the session object corresponding to the input modal feature is the second session object.

[0124] in, In this context, m can be a, l, or v. When m is v, it indicates that the modal category corresponding to the input modal feature is the video modal category; When m is a, it indicates that the modal category corresponding to the input modal feature is the audio modal category; When m is l, it indicates that the modal category corresponding to the input modal feature is the text modal category. For example, This indicates that the input modal features are the audio modal features of the first conversation object in the second historical time period; This indicates that the input modal features are the video modal features of the second session object in the first historical time period.

[0125] in, The output modal projection features, The 's' in the output modal projection features is used to indicate the session object. In this context, 'm' indicates the modal category corresponding to the output modal projection feature. In this context, 'i' indicates the historical time period corresponding to the output modal projection features. In this context, s can be either a or b. When s is a, it indicates that the session object corresponding to the output modal projection feature is the first session object; When s is b, it indicates that the session object corresponding to the output modal projection feature is the second session object.

[0126] in, In this context, m can be a, l, or v. When m is v, it indicates that the modal category corresponding to the output modal projection feature is the video modal category; When m is a, it indicates that the modal category corresponding to the output modal projection feature is the audio modal category; When m is l, it indicates that the modal category corresponding to the output modal projection feature is the text modal category. For example, This indicates that the output modal projection features are the audio modal projection features of the first session object in the second historical time period; This indicates that the output modal projection features are the video modal projection features of the second session object in the first historical time period.

[0127] Where, θ m For the feature projection parameters, θ m In this context, m can be a, l, or v, and θ m When m is v, it indicates that the feature projection parameter is the feature projection parameter of the video modality category; θ m When m is a, it indicates that the feature projection parameter is the feature projection parameter of the audio modality category; θ m When m is l, it indicates that the feature projection parameter is the feature projection parameter of the text modality category. Wherein, FFN m It is a fully connected neural network that can be used to project modal features.

[0128] Specifically, the computer device can input the k modal features in the first completed feature set i, and the feature projection parameters corresponding to the k modal features respectively, into formula (1) to calculate the k modal projection features corresponding to the first completed feature set i. The computer device can input the f modal features in the second completed feature set i, and the feature projection parameters corresponding to the f modal features respectively, into formula (1) to calculate the f modal projection features corresponding to the second completed feature set i.

[0129] In one embodiment, the feature weights corresponding to the k modal projection features corresponding to the first complement feature set i can be calculated using formula (2). In another embodiment, the feature weights corresponding to the f modal projection features corresponding to the second complement feature set i can be calculated using formula (2). Formula (2) is shown below.

[0130]

[0131] in, The input is the audio modal projection feature. The input text modal projection features, The input video modal projection features. as well as The 's' in both cases are used to indicate the corresponding session object; For example, In this context, s can be either a or b. When s is a, it indicates that the corresponding session object is the first session object; When s is b, it indicates that the corresponding session object is the second session object.

[0132] in, The feature weights are the output audio modal projection features. The feature weights for the output text modal projection features. The feature weights are the feature weights for the output video modal projection features. Wherein, as well as The 's' in both cases are used to indicate the corresponding session object; For example, In this context, s can be either a or b. When s is a, it indicates that the corresponding session object is the first session object; When s is b, it indicates that the corresponding session object is the second session object.

[0133] Among them, FFN w This is a fully connected neural network that can be used to determine the initial feature weights of modal projection features, i.e., the feature projection network in this application; θ w For FFN w The network parameters in the code. Softmax() is a normalization function that can be used to normalize the initial feature weights to obtain the feature weights.

[0134] Specifically, the computer device can input the k modal projection features corresponding to the first complementary feature set i into formula (2) to calculate the feature weights corresponding to the k modal projection features corresponding to the first complementary feature set i. The computer device can also input the f modal projection features corresponding to the second complementary feature set i into formula (2) to calculate the feature weights corresponding to the f modal projection features corresponding to the second complementary feature set i.

[0135] In one embodiment, the feature projection network described above is learnable, meaning that the network parameters in the feature projection network are learnable. When training the initial feature projection network, the network parameters in the initial feature projection network can be adjusted to train the initial feature projection network, thereby obtaining the final feature projection network.

[0136] In one embodiment, the first historical session feature corresponding to the first session object in the i-th historical time period can be obtained by weighting and summing the k modal projection features corresponding to the first complement feature set i according to the feature weights corresponding to the k modal projection features corresponding to the first complement feature set i using formula (3). In another embodiment, the second historical session feature corresponding to the second session object in the i-th historical time period can be obtained by weighting and summing the f modal projection features corresponding to the second complement feature set i according to the feature weights corresponding to the f modal projection features corresponding to the second complement feature set i using formula (3). The formula (3) is shown below.

[0137]

[0138] Among them, targeting as well as For further explanation, please refer to the section above regarding formula (2). as well as The explanation will not be repeated here. Regarding... as well as For further explanation, please refer to the section above regarding formula (2). as well as The explanation will not be repeated here. For the output of historical session features, The 's' in the output indicates the session object corresponding to the historical session features. The 'i' in the output is used to indicate that the historical time period corresponding to the historical session feature is the i-th historical time period.

[0139] in, In this context, s can be either a or b. When s is a, it indicates that the session object corresponding to the output historical session features is the first session object; When s is b, it indicates that the session object corresponding to the output historical session features is the second session object. In formula (3), Indicates to as well as To perform splicing, in formula (3) Indicates first to as well as The concatenated features are obtained by splicing them together, and then the concatenated features are vector transposed.

[0140] In one embodiment, the explanation of the fusion process for each modal feature in the N second complement feature sets can be found in the explanation of the fusion process for each modal feature in the N first complement feature sets above, and will not be repeated here.

[0141] S202. From N first historical session features, obtain the first historical session feature that has an adjacent relationship with the first historical session feature i, and use it as the first reference session feature of the first historical session feature i.

[0142] It should be noted that the first historical session feature that is adjacent to the first historical session feature i may include at least one of the first historical session features whose corresponding historical time period is before the i-th historical time period and the first historical session features whose corresponding historical time period is after the i-th historical time period.

[0143] The first reference session feature of the first historical session feature i may include one or more first historical session features that are adjacent to it.

[0144] In this application, a computer device can obtain, from N first historical session features, the first historical session feature whose corresponding historical time period is before the i-th historical time period, and determine the obtained first historical session feature as the first historical session feature that has an adjacent relationship with the first historical session feature i. The computer device can use one or more of the first historical session features that have an adjacent relationship with the first historical session feature i as the first reference session feature of the first historical session feature i. By performing the above operation sequentially for N first historical session features, the first reference session features corresponding to the N first historical session features can be obtained.

[0145] In one embodiment, step S202 includes: the computer device can acquire a number of reference features P; where P is a positive integer. The computer device can filter first historical session features whose corresponding historical time periods are before the historical time period corresponding to first historical session feature i from the N first historical session features. The number of features of all filtered first historical session features is determined. When this number of features is greater than the number of reference features P, the computer device can use P first historical session features that have a continuous adjacent relationship with the first historical session feature i as the first reference session features of the first historical session feature i. Specifically, the first reference session feature corresponding to the first first historical session feature is the first first historical session feature, which is the first historical session feature corresponding to the earliest historical time period. The earliest historical time period can refer to the historical time period that is first in chronological order among the N historical time periods.

[0146] It should be noted that the P first historical session features that have a continuous adjacent relationship with the first historical session feature i can refer to the P first historical session features corresponding to consecutive historical time periods, and the latest session time in the historical time periods corresponding to the P first historical session features corresponding to consecutive historical time periods being consecutive with the historical time period corresponding to the first historical session feature i.

[0147] For example, the historical time period corresponding to the first historical session feature i is the 6th historical time period, and the historical time periods corresponding to the P first historical session features are the 3rd, 4th and 5th historical time periods respectively. The 3rd, 4th and 5th historical time periods are consecutive, and the 5th historical time period is the latest session time among the historical time periods corresponding to the P first historical session features. The 5th historical time period is also consecutive to the 6th historical time period.

[0148] In one embodiment, after determining the number of features of all the first historical session features selected, when the number of features is less than or equal to the number of reference features, the computer device may use all the selected first historical session features as the first reference session features of the first historical session feature i.

[0149] S203. Based on the first reference conversation features corresponding to the N first historical conversation features, perform sentiment feature completion processing on the N first historical conversation features to obtain the N temporal context conversation features of the first conversation object.

[0150] In this application, for a first historical conversation feature i, the computer device can perform sentiment feature completion processing on the first historical conversation feature i based on the first reference conversation feature corresponding to the first historical conversation feature i. That is, it adjusts the first historical conversation feature i so that the conversation sentiment feature reflected by the first historical conversation feature i maintains coherence with the conversation sentiment feature reflected by the first reference conversation feature corresponding to the first historical conversation feature i, thus obtaining the i-th temporal context conversation feature of the first conversation object. By performing the above operation on N first historical conversation features respectively, N temporal context conversation features of the first conversation object can be obtained. By utilizing the contextual information of the first conversation object itself during the conversation process, the missing modal features of the first conversation object mentioned above are dynamically completed. This is beneficial for establishing complete and continuous temporal context conversation sentiment features for the first conversation object pair, making up for the sentiment discontinuity problem caused by traditional methods that only rely on speaking time information, and thus improving the prediction accuracy of conversation sentiment features.

[0151] In one embodiment, the number of first reference session features of the first historical session feature i is S; S is a positive integer. Step S203 includes: the computer device determines S contextual session relationships between the S first reference session features and the first historical session feature i based on the chronological order between the historical time periods corresponding to the S first reference session features and the historical time period corresponding to the first historical session feature i. The computer device determines sentiment feature completion parameters corresponding to each first reference session feature based on the contextual session relationship corresponding to each first reference session feature, and determines the sentiment feature completion parameters corresponding to the S first reference session features as sentiment feature completion parameters (i.e., first sentiment feature completion parameters) for the first historical session feature i. The computer device adjusts the first historical session feature i based on the sentiment feature completion parameters (i.e., first sentiment feature completion parameters) to obtain the i-th temporal contextual session feature of the first session object.

[0152] It should be noted that contextual session relationships include past, present, and future.

[0153] Specifically, when the chronological order of the historical time period corresponding to the first reference session feature precedes the chronological order of the historical time period corresponding to the first historical session feature i, the contextual session relationship between the first reference session feature and the first historical session feature i is a preceding session relationship. When the historical time period corresponding to the first reference session feature is the same as the historical time period corresponding to the first historical session feature i, the contextual session relationship between the first reference session feature and the first historical session feature i is a current session relationship. When the chronological order of the historical time period corresponding to the first reference session feature is after the chronological order of the historical time period corresponding to the first historical session feature i, the contextual session relationship between the first reference session feature and the first historical session feature i is a following session relationship.

[0154] In one embodiment, the process of determining the first sentiment feature completion parameter may include: the computer device determining the sentiment features reflected by the S first reference session features based on the S context session relationships, and the degree of influence of the sentiment features reflected by the first historical session feature i, i.e., determining the sentiment completion weights corresponding to the S first reference session features respectively. The computer device may then perform a weighted average operation on the S first reference session features based on the sentiment completion weights corresponding to the S first reference session features respectively, to obtain the sentiment feature completion parameter (i.e., the first sentiment feature completion parameter) for the first historical session feature i.

[0155] In one embodiment, N temporal context session features of the first session object can be calculated based on formula (4) using the temporal network within the session object. Formula (4) is shown below.

[0156]

[0157] in, For the i-th temporal context session feature, In the context, 's' is used to indicate the session object corresponding to the i-th temporal context session feature. In this context, 'i' indicates that the historical time period corresponding to the 'i'-th temporal context session feature is the 'i'-th historical time period. In this context, s can be a. When s is a, it indicates that the session object corresponding to the i-th temporal context session feature is the first session object.

[0158] in, In this context, r indicates the context relationship. Since r∈α, it means that the context relationship r is a context session relationship. In this context, 'i' indicates the i-th historical time period. The 's' in the text refers to the session object. In this context, s can be a. When s is a, it indicates that the session object is the first session object. This represents the first reference session feature corresponding to the historical session feature (i.e., the first historical session feature i) of the first session object in the i-th historical time period. The number of first reference session features corresponding to the first historical session feature i.

[0159] in, W represents the j-th first reference session feature corresponding to the first historical session feature i. r These are network parameters relating to contextual interaction relationships within the temporal network of the session object.

[0160] In one embodiment, the aforementioned intra-session temporal network of the session object is learnable, that is, the network parameters in the intra-session temporal network of the session object (i.e., W in formula (4)) are learnable. r It is learnable. When training the internal temporal network of the initial session object, the internal temporal network of the initial session object can be trained by adjusting the network parameters in the internal temporal network of the initial session object, thereby obtaining the internal temporal network of the session object.

[0161] S204. Based on the N second historical conversation features, perform sentiment feature completion processing on the N first historical conversation features to obtain N interactive context conversation features between the first conversation object and the second conversation object.

[0162] In one embodiment, step S204 may include: the computer device can obtain, from N second historical session features, a second historical session feature whose corresponding historical time period is before the i-th historical time period, and determine the obtained second historical session feature as a second historical session feature that has an adjacent relationship with the first historical session feature i. The computer device may use one or more of the second historical session features that have an adjacent relationship with the first historical session feature i as second reference session features of the first historical session feature i. By sequentially performing the above operations on the N first historical session features, the second reference session features corresponding to the N first historical session features can be obtained.

[0163] Next, the computer device can perform sentiment feature completion processing on the N first historical conversation features based on the second reference conversation features corresponding to each of the N first historical conversation features. This involves adjusting the first historical conversation features to ensure consistency between the conversation sentiment features reflected by the first historical conversation feature i and the conversation sentiment features reflected by the second reference conversation features corresponding to the first historical conversation feature i, resulting in N interactive context conversation features between the first and second conversation objects. Combining the interactive conversation information between the first and second conversation objects within a historical time period, the missing modal features of the first conversation object mentioned above are dynamically completed. This not only completes the conversation sentiment features of the first conversation object when it is not speaking but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path.

[0164] It should be noted that the second historical session feature that is adjacent to the first historical session feature i may include at least one of the following: the second historical session feature whose corresponding historical time period is before the i-th historical time period, the second historical session feature corresponding to the i-th historical time period, and the second historical session feature whose corresponding historical time period is after the i-th historical time period.

[0165] The second reference session feature of the first historical session feature i may include one or more second historical session features that are adjacent to it.

[0166] In one embodiment, the computer device can obtain a number of reference features P, where P is a positive integer. The computer device can filter from the aforementioned N second historical session features all second historical session features whose corresponding historical time periods are prior to the historical time period corresponding to the first historical session feature i. The number of all filtered second historical session features is determined. When this number is greater than the number of reference features P, the computer device can use P second historical session features that are consecutively adjacent to the first historical session feature i as the second reference session features of the first historical session feature i. Specifically, the second reference session feature corresponding to the first first historical session feature is the aforementioned first second historical session feature, which is the second historical session feature corresponding to the earliest historical time period.

[0167] It should be noted that the P second historical session features that have a continuous adjacent relationship with the first historical session feature i can refer to the P second historical session features corresponding to consecutive historical time periods, and the latest session time in the historical time periods corresponding to the P second historical session features corresponding to consecutive historical time periods being consecutive with the historical time period corresponding to the first historical session feature i.

[0168] For example, the historical time period corresponding to the first historical session feature i is the fourth historical time period, and the historical time periods corresponding to the P second historical session features are the second and third historical time periods, respectively. The second and third historical time periods are consecutive, and the third historical time period is the latest session time among the historical time periods corresponding to the P second historical session features, and the third and fourth historical time periods are consecutive.

[0169] In one embodiment, after determining the number of features of all the filtered second historical session features, when the number of features is less than or equal to the number of reference features, the computer device may use all the filtered second historical session features as the second reference session features of the first historical session feature i.

[0170] In one embodiment, the number of second reference session features of the first historical session feature i is Q; Q is a positive integer. The computer device can determine Q contextual interaction relationships between the Q second reference session features and the first historical session feature i based on the chronological order between the historical time periods corresponding to the Q second reference session features and the historical time period corresponding to the first historical session feature i. Based on the contextual interaction relationship corresponding to each second reference session feature, the computer device determines the sentiment feature completion parameter (i.e., the second sentiment feature completion parameter) corresponding to each second reference session feature. The sentiment feature completion parameters (i.e., the second sentiment feature completion parameters) corresponding to the Q second reference session features are determined as the sentiment feature completion parameters (i.e., the second sentiment feature completion parameters) for the first historical session feature i. The computer device can adjust the first historical session feature i based on the sentiment feature completion parameters (i.e., the second sentiment feature completion parameters) to obtain the i-th interactive contextual session feature of the first session object.

[0171] It should be noted that contextual interaction relationships include preceding contextual interaction relationships, current session interaction relationships, and following contextual interaction relationships.

[0172] In one embodiment, the process of determining the second sentiment feature completion parameter may include: the computer device determining the conversation sentiment features reflected by the Q second reference conversation features based on the Q context interaction relationships, and the degree of influence of these features on the conversation sentiment features reflected by the first historical conversation feature i, i.e., determining the sentiment completion weights corresponding to the Q second reference conversation features respectively. The computer device may then perform a weighted average operation on the Q second reference conversation features based on their respective sentiment completion weights to obtain the sentiment feature completion parameter (i.e., the second sentiment feature completion parameter) for the first historical conversation feature i.

[0173] In one embodiment, N interactive context session features between the first session object and the second session object can be calculated based on formula (5) through the session object interaction network. Formula (5) is shown below.

[0174]

[0175] in, For the i-th interaction context session feature, middle , s is used to indicate the session object corresponding to the i-th interaction context session feature. In this context, 'i' indicates that the historical time period corresponding to the 'i'th interaction context session feature is the 'i'th historical time period. In this context, r indicates a contextual relationship. Since r∈β, it means that the contextual relationship r is a contextual interaction relationship. In this context, 'i' indicates the i-th historical time period. The 's' in the text refers to the session object. In this context, s can be a. When s is a, it indicates that the session object is the first session object. This represents the second reference session feature corresponding to the historical session feature of the first session object in the i-th historical time period (i.e., the first historical session feature i). This represents the number of second reference session features corresponding to the first historical session feature i. Wherein, W represents the j-th second reference session feature corresponding to the first historical session feature i. r These are network parameters relating to the contextual session relationship within the temporal network of the session object.

[0176] In one embodiment, the above-mentioned inter-session object interaction network is learnable, that is, the network parameters in the inter-session object interaction network (i.e., W in formula (5)) are learnable. r It is learnable. When training the initial interaction network between conversational objects, the initial interaction network between conversational objects can be trained by adjusting the network parameters in the initial interaction network between conversational objects, thereby obtaining the interaction network between conversational objects.

[0177] Please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating a process for generating temporal context session features and interaction context session features, as provided in this application. Figure 7 As shown, Figure 7It includes the first and second complete feature sets corresponding to the first historical time period, the first and second complete feature sets corresponding to the second historical time period, the first and second complete feature sets corresponding to the third historical time period, and the first and second complete feature sets corresponding to the fourth historical time period.

[0178] Specifically, the first complete feature set for the first historical time period includes video modal feature A11, audio modal feature B11, and text modal feature C11; the first complete feature set for the second historical time period includes video modal feature A12, audio modal feature B12, and text modal feature C12; the first complete feature set for the third historical time period includes video modal feature A13, audio modal feature B13, and text modal feature C13; and the first complete feature set for the fourth historical time period includes video modal feature A14, audio modal feature B14, and text modal feature C14.

[0179] Specifically, the second complement feature set for the first historical time period includes video modal feature A21, audio modal feature B21, and text modal feature C21; the second complement feature set for the second historical time period includes video modal feature A22, audio modal feature B22, and text modal feature C22; the second complement feature set for the third historical time period includes video modal feature A23, audio modal feature B23, and text modal feature C23; and the second complement feature set for the fourth historical time period includes video modal feature A24, audio modal feature B24, and text modal feature C24.

[0180] Specifically, the computer device can input video modal feature A11, audio modal feature B11, and text modal feature C11 into the feature fusion network 71 to obtain the first historical session feature D11 of the first session object in the first historical time period. Repeating the above operation sequentially, the computer device can obtain four first historical session features corresponding to the first session object in four historical time periods, namely, first historical session feature D11, first historical session feature D12, first historical session feature D13, and first historical session feature D14. Similarly, the computer device can obtain four second historical session features corresponding to the second session object in four historical time periods based on the feature fusion network 72, namely, second historical session feature D21, first historical session feature D22, second historical session feature D23, and second historical session feature D24.

[0181] Next, taking a reference feature count of 3 as an example, the first reference session feature corresponding to the first historical session feature D11 can be the first historical session feature D11; the second reference session feature corresponding to the first historical session feature D11 can be the first historical session feature D21. The first reference session feature corresponding to the first historical session feature D12 can be the first historical session feature D11; the second reference session feature corresponding to the first historical session feature D12 can be the first historical session feature D21. The first reference session feature corresponding to the first historical session feature D13 can be the first historical session feature D11 and the first historical session feature D12; the second reference session feature corresponding to the first historical session feature D13 can be the first historical session feature D21 and the first historical session feature D22. The first reference session feature corresponding to the first historical session feature D14 can be the first historical session feature D11, the first historical session feature D12, and the first historical session feature D13; the first reference session feature corresponding to the first historical session feature D14 can be the first historical session feature D21, the second historical session feature D22, and the first historical session feature D23.

[0182] The computer device can adjust the first historical session feature D14 based on the first historical session feature D11, the first historical session feature D12, and the first historical session feature D13 to obtain the temporal context session feature G14 corresponding to the fourth historical time period. The computer device can also adjust the first historical session feature D14 based on the first historical session feature D21, the second historical session feature D22, and the first historical session feature D23 to obtain the interaction context session feature Z14 corresponding to the fourth historical time period. By sequentially performing the above operations for the four historical time periods, the following can be obtained: the temporal context session feature G11 and the interaction context session feature Z11 corresponding to the first historical time period; the temporal context session feature G12 and the interaction context session feature Z12 corresponding to the second historical time period; the temporal context session feature G13 and the interaction context session feature Z13 corresponding to the first historical time period; and the temporal context session feature G14 and the interaction context session feature Z14 corresponding to the first historical time period.

[0183] S205. Based on N temporal context conversation features and N interactive context conversation features, predict the conversation sentiment features of the first conversation object at the current moment, and generate the dialogue content of the second conversation object at the current moment based on the conversation sentiment features.

[0184] In one embodiment, step S205 includes: the computer device concatenates the temporal context conversation features and interaction context conversation features corresponding to each historical time period to obtain concatenated context conversation features corresponding to each historical time period. Using a sentiment prediction network, key features reflecting conversational sentiment characteristics are extracted from the concatenated context conversation features corresponding to the first historical time period. Based on the extracted key features, the conversational sentiment characteristics of the first conversational object in the first historical time period are predicted. The computer device can use a sentiment prediction network to extract key features reflecting conversational sentiment characteristics from the concatenated context conversation features corresponding to the second historical time period. Based on the extracted key features, the conversational sentiment characteristics of the first conversational object in the first historical time period are adjusted to obtain the conversational sentiment characteristics of the first conversational object in the second historical time period. By sequentially performing the above operations on the temporal context conversation features corresponding to N historical time periods, the conversational sentiment characteristics of the first conversational object corresponding to the N historical time periods can be obtained. The computer device can determine the conversational sentiment characteristics of the first conversational object corresponding to the Nth historical time period as the conversational sentiment characteristics of the first conversational object at the current moment; wherein the Nth historical time period is the latest historical time period among the N historical time periods.

[0185] It should be noted that the feature dimension of the spliced ​​context session feature corresponding to each historical time period is consistent with the total feature dimension between the feature dimension of the temporal context session feature and the feature dimension of the interaction context session feature corresponding to each historical time period.

[0186] In one embodiment, N spliced ​​context conversation features can be calculated using formula (6); then, using a sentiment prediction network, the conversation sentiment features of the first conversation object at the current moment can be obtained based on formula (7). Formulas (6) and (7) are shown below.

[0187]

[0188] in, For the i-th temporal context session feature, For the i-th interaction context session feature, Let i be the i-th splicing context session feature.

[0189] Among them, F s Representing N concatenated contextual conversation features, LSTM can be a type of Long Short-Term Memory network used to predict conversation sentiment features, i.e., the sentiment prediction network in this application, θ q These are the network parameters in the sentiment prediction network. Let be the conversation sentiment features for the i-th historical time period.

[0190] In one embodiment, the sentiment prediction network described above is learnable, that is, the network parameters in the sentiment prediction network (i.e., θ in formula (7)) are learnable. q The sentiment prediction method is learnable. Specifically, the computer device can acquire R first-sample historical conversation features corresponding to R sample historical time periods for the first sample conversation object, R second-sample historical conversation features corresponding to R sample historical time periods for the second sample conversation object, and the labeled conversation sentiment features of the first sample conversation object at the current moment. Here, R is an integer greater than 1. The computer device can perform sentiment feature imputation processing on the R first-sample historical conversation features to obtain R sample temporal context conversation features; based on the R second-sample historical conversation features, it can perform sentiment feature imputation processing on the R first-sample historical conversation features to obtain R sample interaction context conversation features. The computer device can use an initial sentiment prediction network to predict the sample conversation sentiment features of the first sample conversation object at the current moment based on the R sample temporal context conversation features and the R sample interaction context conversation features. Using a loss function, based on the sample conversation sentiment features and the labeled conversation sentiment features, the prediction loss of the initial sentiment prediction network is determined. Based on this prediction loss, the network parameters in the initial sentiment prediction network are adjusted to obtain the sentiment prediction network. The loss function can include cross-entropy loss, binary entropy loss, etc.

[0191] Please refer to Figure 8 , Figure 8This is a schematic diagram of a sentiment prediction process provided in this application. The process includes the following steps: S81, generating concatenated contextual conversation features. In this application, the computer device can concatenate the temporal contextual conversation feature G11 and the interaction contextual conversation feature Z11 corresponding to the first historical time period to obtain the concatenated contextual conversation feature F1 corresponding to the first historical time period; concatenate the temporal contextual conversation feature G12 and the interaction contextual conversation feature Z12 corresponding to the second historical time period to obtain the concatenated contextual conversation feature F2 corresponding to the first historical time period; and concatenate the temporal contextual conversation feature G13 and the interaction contextual conversation feature Z13 corresponding to the third historical time period to obtain the concatenated contextual conversation feature F3 corresponding to the first historical time period. S82, predicting conversation sentiment features based on the concatenated contextual conversation features. In this application, a computer device can input the spliced ​​context conversation feature F1 into the sentiment prediction network 80, and through the sentiment prediction network 80, predict the conversation sentiment feature H1 corresponding to the first historical time period; input the spliced ​​context conversation feature F2 into the sentiment prediction network 80, and through the sentiment prediction network 80, predict the conversation sentiment feature H2 corresponding to the first historical time period based on the conversation sentiment feature H1 and the spliced ​​context conversation feature F2; input the spliced ​​context conversation feature F3 into the sentiment prediction network 80, and through the sentiment prediction network 80, predict the conversation sentiment feature H3 corresponding to the first historical time period based on the conversation sentiment feature H2 and the spliced ​​context conversation feature F2.

[0192] Please refer to Figure 9a and Figure 9b , Figure 9a This application provides a convergence analysis of a sentiment prediction network. Figure 1 , Figure 9b This application provides a convergence analysis of a sentiment prediction network. Figure 2 .

[0193] This application validates the convergence of the sentiment prediction network in this application on two dialogue datasets. The two datasets are the M3ED dataset (a dataset specifically designed for multimodal sentiment recognition tasks in sentiment computing and dialogue) and the IEMOCAP dataset (a multimodal sentiment recognition dataset). Figure 9a This is a convergence analysis diagram of the sentiment prediction network in this application on the M3ED dataset. Figure 9b This is a convergence analysis diagram of the sentiment prediction network in this application on the IEMOCAP dataset.

[0194] It should be noted that the loss function is a metric that measures the difference between the prediction results of a sentiment prediction network and the actual results. Generally, the smaller the prediction loss obtained from the loss function, the better, indicating that the sentiment prediction network has better performance and better convergence.

[0195] in, Figure 9a The x-axis parameter in the figure represents the number of training epochs. Figure 9a The vertical axis parameter in the table represents the prediction loss; for example... Figure 9a As shown, the sentiment prediction network obtained three training curves on three subsets of the M3ED dataset. The three subsets of the M3ED dataset are the M3ED training set, the M3ED validation set, and the M3ED test set; the three training curves are the training curve of the M3ED training set (i.e., the train curve in Figure 9a), the training curve of the M3ED validation set (i.e., the valid curve in Figure 9a), and the training curve of the M3ED test set (i.e., the test curve in Figure 9a). Based on Figure 9a It can be seen that as the number of training rounds increases, the loss values ​​of the three training curves all decrease significantly, and the loss values ​​tend to stabilize after the 35th round of training, indicating that the convergence of the sentiment prediction network is in a good state after the 35th round of training.

[0196] in, Figure 9b The x-axis parameter in the figure represents the number of training epochs. Figure 9b The vertical axis parameter in the table represents the prediction loss; for example... Figure 9a As shown, the sentiment prediction network obtained two training curves on two subsets of the IEMOCAP dataset. The two subsets are the IEMOCAP training set and the IEMOCAP validation set; the two training curves are the training curve on the IEMOCAP training set (i.e., the train curve in Figure 9a) and the training curve on the IEMOCAP validation set (i.e., the valid curve in Figure 9b). Based on Figure 9b As can be seen, with the increase of the number of training rounds, the loss values ​​of both training curves decreased significantly and then tended to stabilize, indicating that the convergence of the sentiment prediction network has reached a good state.

[0197] Therefore, it can be seen that the sentiment prediction network in this application has high prediction accuracy, that is, the prediction accuracy of the conversation sentiment features predicted by the sentiment prediction network is high.

[0198] Please refer to Figure 9c , Figure 9d , Figure 9e and Figure 9f , Figure 9c This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 1 , Figure 9d This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 2 , Figure 9e This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 3 ,as well as Figure 9f This is a visualization of the training results of a sentiment prediction network provided in this application. Figure 4 .

[0199] In this application, Figure 9c , Figure 9d , Figure 9e and Figure 9f The visualization results show the hidden layer representations of different sentiment categories in the sentiment prediction network at different training rounds.

[0200] It should be noted that as the training epochs of the sentiment prediction network increase, the stronger the aggregation between the hidden layer representations corresponding to the same sentiment category identified by the sentiment prediction network, and the higher the discrimination between the hidden layer representations corresponding to different sentiment categories identified by the sentiment prediction network, the higher the prediction accuracy of the sentiment prediction network.

[0201] Specifically, Figure 9c This is a visualization of the training results of the sentiment prediction network after 10 training epochs. Figure 9d This is a visualization of the training results of the sentiment prediction network after 20 training epochs. Figure 9e A visualization of the training results of the sentiment prediction network after 30 training epochs, and Figure 9f This is a visualization of the training results of the sentiment prediction network after 43 training epochs.

[0202] It should be noted that, Figure 9c , Figure 9d , Figure 9e and Figure 9f In a network, figures of the same shape represent the hidden layer representations of the same emotion category. That is, if two figures have different shapes, then the corresponding hidden layer representations represent different emotion categories. Therefore, from... Figure 9c , Figure 9d , Figure 9e and Figure 9f Analysis shows that as the training rounds of the sentiment prediction network in this application increase, the aggregation degree between the hidden layer representations corresponding to the same sentiment category identified by the sentiment prediction network in this application gradually increases, and the discrimination degree between the hidden layer representations corresponding to different sentiment categories identified by the sentiment prediction network in this application gradually increases, indicating that the prediction accuracy of the sentiment prediction network in this application is high.

[0203] In addition, this application designs different experiments on the IEMOCAP dataset to verify the functionality of the feature fusion network, the intra-session temporal network, and the inter-session interaction network provided in this application, so as to reveal the necessity and benefits of each network provided in this application.

[0204] First, a basic sentiment prediction network is designed. This basic sentiment prediction network does not use the feature fusion network, intra-conversation temporal network, and inter-conversation interaction network proposed in this application. Then, different networks are gradually added to the basic network to facilitate observation of their contributions to improving the recognition results. Specifically, during the use of the basic sentiment prediction network, the modal features of the first conversational subject at the time they are not speaking are still initialized. Feature concatenation is used instead of multimodal fusion based on the feature fusion network. The fused features are not processed for sentiment feature completion; instead, they are directly input into the basic sentiment prediction network to predict the conversational sentiment features.

[0205] Table 1

[0206]

[0207] As shown in Table 1, on the IEMOCAP dataset, using only the basic sentiment prediction network without the feature fusion network, intra-conversational temporal network, and inter-conversational interaction network, the accuracy of the predicted conversational sentiment features is 64.62%, and the weighted F1 score is 64.79%. Using both the basic sentiment prediction network and the feature fusion network, but without the intra-conversational temporal network and the inter-conversational interaction network, the accuracy of the predicted conversational sentiment features is 65.21%, and the weighted F1 score is 65.32%. Using the basic sentiment prediction network, intra-conversational temporal network, and inter-conversational interaction network, but without the feature fusion network, the accuracy of the predicted conversational sentiment features is 70.60%, and the weighted F1 score is 70.69%. Using the basic sentiment prediction network, feature fusion network, intra-conversational temporal network, and inter-conversational interaction network, the accuracy of the predicted conversational sentiment features is 71.35%, and the weighted F1 score is 71.30%.

[0208] The weighted F1 score reflects the prediction accuracy of conversational sentiment features for each sentiment category in the IEMOCAP dataset.

[0209] Analysis shows that adding any of the networks provided in this application to the basic sentiment prediction network can improve the prediction accuracy of conversation sentiment features. Specifically, adding the feature fusion network improves prediction accuracy and weighted F1 score by 0.59% and 0.53%, respectively. Adding the intra-conversational temporal network and the inter-conversational interaction network further improves prediction accuracy and weighted F1 score by 5.98% and 5.90%, respectively. Therefore, the data processing method provided in this application can provide high accuracy in predicting conversation sentiment features.

[0210] In this application, by using first historical conversation features that are adjacent to each of the N first historical conversation features, sentiment feature completion processing is applied to the N first historical conversation features, which helps improve the accuracy of sentiment completion processing, resulting in N temporal context conversation features. By applying sentiment feature completion processing to the N first historical conversation features based on the aforementioned N second historical conversation features, N interactive context conversation features between the aforementioned first conversation object and the aforementioned second conversation object are obtained. This not only completes the conversation sentiment features of the first conversation object when it is not speaking, but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path. Furthermore, based on the N temporal context conversation features and the N interactive context conversation features, predicting the conversation sentiment features of the first conversation object at the current moment can improve the prediction accuracy of conversation sentiment features; and based on the conversation sentiment features at the current moment, predicting the dialogue content of the second conversation object can improve the accuracy of dialogue content prediction.

[0211] Please refer to Figure 10 , Figure 10 This is a flowchart illustrating a data processing method provided in this application. Figure 3 The method includes the following steps:

[0212] S91. Data Acquisition; In this application, the computer device can acquire at least one of the modal data (such as video data, audio data, text data, etc.) corresponding to the first session object and the second session object respectively during the session between the first session object and the second session object, so as to obtain the modal data sets corresponding to the first session object and the second session object respectively.

[0213] S92. Generate a modal feature set; In this application, the computer device can divide the conversation process of the first session object and the second session object into time periods to obtain N historical time periods; based on the N historical time periods, the modal data in the modal data set of the first session object is divided into time periods to obtain a modal data subset corresponding to the first session object in the N historical time periods; based on the N historical time periods, the modal data in the modal data set of the second session object is divided into time periods to obtain a modal data subset corresponding to the second session object in the N historical time periods. Wherein, the modal data subset corresponding to the first session object in the i-th historical time period includes at least one of video data, audio data, text data, etc., of the first session object within the i-th historical time period; the modal data subset corresponding to the second session object in the i-th historical time period includes at least one of video data, audio data, text data, etc., of the second session object within the i-th historical time period.

[0214] Next, the computer device can extract at least one modal feature of the first session object within the i-th historical time period from the modal data subset corresponding to the first session object in the i-th historical time period, and determine it as the modal feature set of the first session object in the i-th historical time period. For the modal data subsets corresponding to the first session object in N historical time periods and the modal data subsets corresponding to the second session object in N historical time periods, the above operation is performed sequentially to obtain N sets of first modal features corresponding to the first session object in N historical time periods and N sets of second modal features corresponding to the second session object in N historical time periods.

[0215] S93. Whether the modal feature set has feature integrity; In this application, the computer device can perform feature integrity detection on N first modal feature sets to obtain a first detection result, and perform feature integrity detection on N second modal feature sets to obtain a second detection result. For a detailed explanation of the feature integrity detection process, please refer to the explanation of the feature integrity detection process in step S201 above, which will not be repeated here. Next, when the first detection result indicates that all N first modal feature sets have feature integrity, and the second detection result indicates that all N second modal feature sets have feature integrity, step S95 is executed. When the first detection result indicates that there is a first modal feature set without feature integrity among the N first modal feature sets, or when the second detection result indicates that there is a second modal feature set without feature integrity among the N second modal feature sets, step S94 is executed.

[0216] S94. Modal feature set completion processing: In this application, when the first detection result indicates that there is a first modal feature set without feature completeness among the N first modal feature sets, the computer device can perform feature completion processing on the N first modal feature sets according to the first detection result to obtain N first completed feature sets. When the second detection result indicates that there is a second modal feature set without feature completeness among the N second modal feature sets, the computer device can perform feature completion processing on the N second modal feature sets according to the second detection result to obtain N second completed feature sets. For a detailed explanation of the feature completion process, please refer to the explanation of the feature completion process in step S201 above, which will not be repeated here.

[0217] S95. Fusing the complemented feature sets; In this application, the computer device can fuse the modal features in the N first complemented feature sets to obtain N first historical session features corresponding to N historical time periods; and fuse the modal features in the N second complemented feature sets to obtain N second historical session features corresponding to N historical time periods. The specific fusion process can be found above and will not be repeated here.

[0218] S96. Contextual Feature Completion: In this application, the computer device can perform intra-session temporal context completion processing on N first historical session features, that is, adjust the N first historical session features according to the N first historical session features to obtain N temporal context session features. The specific fusion processing process can be referred to in steps S202 to S203 above, and will not be repeated here. Next, the computer device can perform inter-session interaction context completion processing on the N first historical session features according to the N second historical session features, that is, adjust the N first historical session features according to the N second historical session features to obtain N interaction context session features. The specific fusion processing process can be referred to in step S204 above, and will not be repeated here.

[0219] S97. Context feature modeling; In this application, the computer device can construct the complete context of the first session object based on N temporal context session features and N interactive context session features; Specifically, the computer device can concatenate the temporal context session features and interactive context session features corresponding to each historical time period to obtain concatenated context session features corresponding to N historical time periods.

[0220] S98. Sentiment Prediction: In this application, a computer device can input the spliced ​​contextual conversation features corresponding to N historical time periods into a sentiment prediction network, and predict the conversation sentiment features of the first conversation object at the current moment through the sentiment prediction network.

[0221] In this application, by performing feature integrity detection and completion processing on the modal feature set, the missing modal features of the first conversation object during the conversation process are initially completed, resulting in N first historical conversation features. Then, through context feature completion (i.e., temporal context completion processing within the conversation object and interaction context completion processing between conversation objects), sentiment completion is further applied to the N first historical conversation features, resulting in N temporal context conversation features and N interaction context conversation features. Based on this, the conversation sentiment features of the first conversation object at the current moment can be predicted, which helps improve the prediction accuracy of the obtained conversation sentiment features.

[0222] Please see Figure 11 , Figure 11This is a schematic diagram of the structure of a data processing device provided in this application. Figure 11 As shown, the data processing apparatus may include:

[0223] The acquisition module 1001 is used to acquire N first historical session features corresponding to N historical time periods for the first session object, and to acquire N second historical session features corresponding to the second session object in the aforementioned N historical time periods; N is an integer greater than 1; the aforementioned N historical time periods are the time periods during which the aforementioned first session object and the aforementioned second session object conduct a session.

[0224] The first processing module 1002 is used to perform sentiment feature completion processing on the above N first historical conversation features to obtain N temporal context conversation features of the above first conversation object.

[0225] The second processing module 1003 is used to perform sentiment feature completion processing on the N first historical conversation features based on the N second historical conversation features to obtain N interactive context conversation features between the first conversation object and the second conversation object.

[0226] The prediction module 1004 is used to predict the conversation sentiment features of the first conversation object at the current moment based on the above N temporal context conversation features and the above N interactive context conversation features, and to generate the dialogue content of the second conversation object at the above current moment based on the above conversation sentiment features; the above current moment is the time after the above N historical time periods.

[0227] Optionally, the first processing module 1002 described above can also be used to perform the following operations:

[0228] From the above N first historical session features, the first historical session feature that is adjacent to the first historical session feature i is obtained and used as the first reference session feature of the first historical session feature i; i is a positive integer less than or equal to N;

[0229] Based on the first reference conversation features corresponding to the above N first historical conversation features, sentiment feature completion processing is performed on the above N first historical conversation features to obtain the above N temporal context conversation features of the first conversation object.

[0230] Optionally, the first processing module 1002 described above can also be used to perform the following operations:

[0231] Obtain the number of reference features P; P is a positive integer;

[0232] From the above N first historical session features, select the first historical session features whose corresponding historical time period is before the historical time period corresponding to the first historical session feature i; the first reference session feature corresponding to the first first historical session feature is the above first first historical session feature, and the above first first historical session feature is the first historical session feature corresponding to the earliest historical time period.

[0233] When the number of features of the first historical conversation feature obtained by filtering is greater than the number of reference features, the P first historical conversation features that have a continuous adjacent relationship with the first historical conversation feature i will be used as the first reference conversation feature of the first historical conversation feature i.

[0234] When the number of features of the first historical session feature obtained by filtering is less than or equal to the number of reference features mentioned above, the first historical session feature obtained by filtering will be used as the first reference session feature of the first historical session feature i mentioned above.

[0235] Optionally, the number of first reference session features of the first historical session feature i mentioned above is S; S is a positive integer;

[0236] The first processing module 1002 described above can also be used to perform the following operations:

[0237] Determine the S contextual session relationships between the S first reference session features and the aforementioned first historical session feature i;

[0238] Based on the above S first reference session features and the above S context session relationships, determine the sentiment feature completion parameters for the above first historical session feature i;

[0239] Based on the above-mentioned sentiment features, the parameters are supplemented and the first historical conversation feature i is adjusted to obtain the i-th temporal context conversation feature of the first conversation object.

[0240] Optionally, the first processing module 1002 described above can also be used to perform the following operations:

[0241] Based on the above S contextual conversation relationships, determine the sentiment completion weights corresponding to the above S first reference conversation features respectively;

[0242] Based on the sentiment completion weights corresponding to the S first reference conversation features, a weighted average operation is performed on the S first reference conversation features to obtain the sentiment feature completion parameters for the first historical conversation feature i.

[0243] Optionally, the second processing module 1003 described above can also be used to perform the following operations:

[0244] From the above N second historical session features, obtain the second historical session feature that is adjacent to the first historical session feature i, and use it as the second reference session feature of the first historical session feature i; i is a positive integer less than or equal to N;

[0245] Based on the second reference conversation features corresponding to the above N first historical conversation features, sentiment feature completion processing is performed on the above N first historical conversation features to obtain N interaction context conversation features between the above first conversation object and the above second conversation object.

[0246] Optionally, the prediction module 1004 described above can also be used to perform the following operations:

[0247] The N temporal context session features and the N interactive context session features mentioned above are concatenated to obtain the N concatenated context session features corresponding to the N historical time periods.

[0248] Using a sentiment prediction network, based on the sentiment features of the conversation corresponding to the (j-1)th historical time period and the concatenated conversation features corresponding to the jth historical time period, the sentiment features of the first conversation object in the jth historical time period are predicted; the sentiment features of the conversation corresponding to the first historical time period are determined based on the concatenated conversation features corresponding to the first conversation time; j is an integer greater than 1 and less than or equal to N.

[0249] Until the conversation sentiment features of the first conversation object are obtained for each of the above N historical time periods, the conversation sentiment feature of the first conversation object in the Nth historical time period is determined as the conversation sentiment feature of the first conversation object at the current moment; the Nth historical time period is the latest historical time period among the above N historical time periods.

[0250] Optionally, the acquisition module 1001 described above can also be used to perform the following operations:

[0251] Obtain the set of N modal features corresponding to the first session object in N historical time periods;

[0252] Modal feature completion processing is performed on the above N modal feature sets to obtain N completed feature sets;

[0253] By fusing the modal features in the above N complement feature sets, N first historical session features corresponding to the above N historical time periods of the above first session object are obtained.

[0254] Optionally, the acquisition module 1001 described above can also be used to perform the following operations:

[0255] When the target modal feature set in the above N modal feature sets includes the first modal feature but does not have the second modal feature, the target feature dimension of the first modal feature in the above target modal feature set is determined by the feature completion network.

[0256] Based on the feature completion parameters corresponding to the second modality features mentioned above, generate second modality features with the same feature dimension as the target feature dimension mentioned above;

[0257] The first modal feature and the generated second modal feature included in the above target modal feature set are determined as the completed target modal feature set;

[0258] When the above N modal feature sets also include the remaining modal feature sets of the first and second modal features, the remaining modal feature sets and the above-mentioned completed target modal feature sets are determined as N completed feature sets.

[0259] Optionally, each complement feature set includes k modal features; k is a positive integer;

[0260] The aforementioned acquisition module 1001 can also be used to perform the following operations:

[0261] Obtain the modal categories corresponding to the k modal features in the feature set i; the feature set i is the feature set corresponding to the i-th historical time period; i is a positive integer less than or equal to N;

[0262] Based on the modal categories corresponding to the above k modal features, determine the feature projection parameters corresponding to the above k modal features respectively;

[0263] Based on the feature projection parameters corresponding to the k modal features, feature projection is performed on the k modal features to obtain k modal projection features; the feature dimensions corresponding to the k modal projection features are all the same;

[0264] By fusing the above k modal projection features, the first historical session feature corresponding to the first session object in the above i-th historical time period is obtained.

[0265] Optionally, the acquisition module 1001 described above can also be used to perform the following operations:

[0266] Obtain the feature weights corresponding to the k modal projection features mentioned above;

[0267] Based on the feature weights corresponding to the k modal projection features, the k modal projection features are weighted and summed to obtain the first historical session feature corresponding to the first session object in the i-th historical time period.

[0268] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0269] In this application, by performing sentiment feature completion processing on N first historical conversation features of the first conversation object, N temporal context conversation features of the first conversation object are obtained. This involves completing the sentiment features of the first conversation object when it is not speaking, establishing a complete and continuous temporal context conversation sentiment feature for the first conversation object, thus overcoming the sentiment gap problem caused by traditional methods that rely solely on speaking time information. By performing sentiment feature completion processing on the N first historical conversation features based on the aforementioned N second historical conversation features, N interactive context conversation features between the first and second conversation objects are obtained. This not only completes the conversation sentiment features of the first conversation object when it is not speaking but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path. Furthermore, predicting the conversation sentiment features of the first conversation object at the current moment based on the N temporal context conversation features and the N interactive context conversation features improves the prediction accuracy of the conversation sentiment features. Predicting the dialogue content of the second conversation object based on the conversation sentiment features at the current moment improves the accuracy of the dialogue content.

[0270] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in this application. For example... Figure 12 As shown, the computer device may include:

[0271] The aforementioned computer device 1100 includes a processor 1101, a network interface 1104, and a memory 1105. Furthermore, the computer device 1100 may also include a user interface 1103 and at least one communication bus 1102. The communication bus 1102 is used to enable communication between these components. In some embodiments, the user interface 1103 may include a display screen and a keyboard; optionally, the user interface 1103 may also include a standard wired interface or a wireless interface. The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1105 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. The memory 1105 may also optionally be at least one storage device located remotely from the aforementioned processor 1101. Figure 12 As shown, the memory 1105 of a computer device 1100 may include an operating system, a network communication module, a user interface module, and computer programs.

[0272] exist Figure 12 In the computer device 1100 shown, the network interface 1104 provides network communication functionality; the user interface 1103 is mainly used to provide an input interface; and the processor 1101 can be used to call computer programs stored in the memory 1105 to execute:

[0273] Obtain N first historical session features corresponding to the first session object in N historical time periods, and obtain N second historical session features corresponding to the second session object in the aforementioned N historical time periods; N is an integer greater than 1; the aforementioned N historical time periods are the time periods during which the aforementioned first session object and the aforementioned second session object had a conversation;

[0274] Sentiment feature completion processing is performed on the above N first historical conversation features to obtain N temporal context conversation features of the above first conversation object;

[0275] Based on the above N second historical conversation features, sentiment feature completion processing is performed on the above N first historical conversation features to obtain N interactive context conversation features between the above first conversation object and the above second conversation object.

[0276] Based on the aforementioned N temporal context conversation features and the aforementioned N interactive context conversation features, the conversation sentiment features of the first conversation object at the current moment are predicted, and the dialogue content of the second conversation object at the current moment is generated based on the aforementioned conversation sentiment features; the aforementioned current moment is the time after the aforementioned N historical time periods.

[0277] In this application, by performing sentiment feature completion processing on N first historical conversation features of the first conversation object, N temporal context conversation features of the first conversation object are obtained. This involves completing the sentiment features of the first conversation object when it is not speaking, establishing a complete and continuous temporal context conversation sentiment feature for the first conversation object, thus overcoming the sentiment gap problem caused by traditional methods that rely solely on speaking time information. By performing sentiment feature completion processing on the N first historical conversation features based on the aforementioned N second historical conversation features, N interactive context conversation features between the first and second conversation objects are obtained. This not only completes the conversation sentiment features of the first conversation object when it is not speaking but also further enriches the conversation sentiment features of the first conversation object when it is speaking, thereby constructing a more complete and coherent conversation sentiment feature evolution path. Furthermore, predicting the conversation sentiment features of the first conversation object at the current moment based on the N temporal context conversation features and the N interactive context conversation features improves the prediction accuracy of the conversation sentiment features. Predicting the dialogue content of the second conversation object based on the conversation sentiment features at the current moment improves the accuracy of the dialogue content.

[0278] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program executed by the aforementioned data processing apparatus. This computer program includes program instructions, which, when executed by the processor, enable the execution of the data processing method described in the corresponding embodiments above. Therefore, further details will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0279] As an example, the above program instructions can be deployed and executed on a computer device, or deployed and executed on at least two computer devices in one location, or executed on at least two computer devices distributed in at least two locations and interconnected by a communication network. At least two computer devices distributed in at least two locations and interconnected by a communication network can form a blockchain network.

[0280] The aforementioned computer-readable storage medium may be a data processing apparatus provided in any of the foregoing embodiments or a central storage unit of the aforementioned computer device, such as a hard disk or central storage of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both the central storage unit and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0281] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish content in different media, rather than to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0282] In practice, the collection and processing of data in this application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the data subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the data subject.

[0283] This application also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the data processing method and decoding method described in the preceding embodiments, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0284] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0285] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions are provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable network-connected device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable network-connected device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable network-connected device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable network-connected device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0286] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining N first historical conversation features corresponding to a first conversation object in N historical time periods, and obtaining N second historical conversation features corresponding to a second conversation object in the N historical time periods; N is an integer greater than 1; the N historical time periods are time periods in which the first conversation object and the second conversation object have a conversation; performing emotion feature padding processing on the N first historical conversation features to obtain N time sequence context conversation features of the first conversation object; performing emotion feature padding processing on the N first historical conversation features according to the N second historical conversation features to obtain N interaction context conversation features between the first conversation object and the second conversation object; predicting a conversation emotion feature of the first conversation object at a current time according to the N time sequence context conversation features and the N interaction context conversation features, and generating a dialogue content of the second conversation object at the current time according to the conversation emotion feature; the current time is a time after the N historical time periods.

2. The method of claim 1, wherein, The emotion feature padding processing on the N first historical conversation features to obtain the N time sequence context conversation features of the first conversation object comprises the following steps: obtaining, from the N first historical conversation features, a first historical conversation feature having an adjacent relationship with a first historical conversation feature i as a first reference conversation feature of the first historical conversation feature i; i is a positive integer less than or equal to N; performing emotion feature padding processing on the N first historical conversation features according to the first reference conversation features corresponding to the N first historical conversation features to obtain the N time sequence context conversation features of the first conversation object.

3. The method of claim 2, wherein, The emotion feature padding processing on the N first historical conversation features to obtain the N time sequence context conversation features of the first conversation object comprises the following steps: obtaining a reference feature number P; P is a positive integer; obtaining, from the N first historical conversation features, a first historical conversation feature corresponding to a historical time period located before a historical time period corresponding to a first historical conversation feature i; a first reference conversation feature corresponding to a first historical conversation feature is the first historical conversation feature, and the first historical conversation feature is a first historical conversation feature corresponding to an earliest historical time period; when the number of the obtained first historical conversation features is greater than the reference feature number, taking P first historical conversation features having a continuous adjacent relationship with the first historical conversation feature i from the obtained first historical conversation features as the first reference conversation features of the first historical conversation feature i; when the number of the obtained first historical conversation features is less than or equal to the reference feature number, taking the obtained first historical conversation features as the first reference conversation features of the first historical conversation feature i.

4. The method of claim 2, wherein, The number of the first reference conversation features of the first historical conversation feature i is S; S is a positive integer; The emotion feature filling processing is performed on the N first historical conversation features according to the first reference conversation features corresponding to the N first historical conversation features respectively, to obtain N time sequence context conversation features of the first conversation object, including: S context conversation relationships corresponding to the first historical conversation feature i and the S first reference conversation features are determined; According to the S first reference conversation features and the S context conversation relationships, emotion feature filling parameters for the first historical conversation feature i are determined; The first historical conversation feature i is adjusted according to the emotion feature filling parameters, to obtain the i th time sequence context conversation feature of the first conversation object.

5. The method of claim 4, wherein, The emotion feature filling parameters for the first historical conversation feature i are determined according to the S first reference conversation features and the S context conversation relationships, including: According to the S context conversation relationships, emotion filling weights corresponding to the S first reference conversation features are determined; The S first reference conversation features are weighted and averaged according to the emotion filling weights corresponding to the S first reference conversation features, to obtain the emotion feature filling parameters for the first historical conversation feature i.

6. The method of claim 1, wherein, The emotion feature filling processing is performed on the N first historical conversation features according to the N second historical conversation features, to obtain N interaction context conversation features between the first conversation object and the second conversation object, including: From the N second historical conversation features, a second historical conversation feature having an adjacent relationship with the first historical conversation feature i is obtained as a second reference conversation feature of the first historical conversation feature i; i is a positive integer less than or equal to N; The emotion feature filling processing is performed on the N first historical conversation features according to the second reference conversation features corresponding to the N first historical conversation features respectively, to obtain N interaction context conversation features between the first conversation object and the second conversation object.

7. The method of claim 1, wherein, The conversation emotion feature of the first conversation object at the current time is predicted according to the N time sequence context conversation features and the N interaction context conversation features, including: The N time sequence context conversation features and the N N interaction context conversation features are feature spliced to obtain N spliced context conversation features corresponding to the N historical time periods; The conversation emotion feature of the first conversation object in the j th historical time period is predicted based on the conversation emotion feature corresponding to the j-1 th historical time period and the spliced context conversation feature corresponding to the j th historical time period through an emotion prediction network; the conversation emotion feature corresponding to the first historical time period is determined based on the spliced context conversation feature corresponding to the first conversation time; j is an integer greater than 1 and less than or equal to N; Until the conversation emotion feature of the first conversation object corresponding to the Nth historical time period is obtained, the conversation emotion feature of the first conversation object corresponding to the Nth historical time period is determined as the conversation emotion feature of the first conversation object at the current time; the Nth historical time period is the latest historical time period in the N historical time periods.

8. The method of claim 1, wherein, The obtaining of the N first historical conversation features of the first conversation object corresponding to the N historical time periods comprises: obtaining N sets of modal features of the first conversation object corresponding to the N historical time periods; performing modal feature padding processing on the N sets of modal features to obtain N sets of padded features; fusing each modal feature in the N sets of padded features to obtain the N first historical conversation features of the first conversation object corresponding to the N historical time periods.

9. The method of claim 8, wherein, The modal feature padding processing on the N sets of modal features to obtain the N sets of padded features comprises: when a target modal feature set in the N sets of modal features comprises a first modal feature and does not comprise a second modal feature, determining a target feature dimension of the first modal feature in the target modal feature set through a feature padding network; generating a second modal feature with the same feature dimension as the target feature dimension according to a feature padding parameter corresponding to the second modal feature; determining the first modal feature included in the target modal feature set and the generated second modal feature as a padded target modal feature set; when the N sets of modal features further comprise a remaining modal feature set including the first modal feature and the second modal feature, determining the remaining modal feature set and the padded target modal feature set as the N sets of padded features.

10. The method of claim 9, wherein, Each set of padded features comprises k modal features; k is a positive integer; The fusing of each modal feature in the N sets of padded features to obtain the N first historical conversation features of the first conversation object corresponding to the N historical time periods comprises: obtaining modal categories corresponding to k modal features in a set of padded features i; the set of padded features i is a set of padded features corresponding to an ith historical time period; i is a positive integer less than or equal to N; determining feature projection parameters corresponding to the k modal features according to the modal categories corresponding to the k modal features; performing feature projection on the k modal features according to the feature projection parameters corresponding to the k modal features to obtain k modal projection features; the feature dimensions corresponding to the k modal projection features are all the same; fusing the k modal projection features to obtain a first historical conversation feature of the first conversation object corresponding to the ith historical time period.

11. The method of claim 10, wherein, The fusing of the k modal projection features to obtain the first historical conversation feature of the first conversation object corresponding to the ith historical time period comprises: obtaining feature weights corresponding to the k modal projection features; According to the feature weights corresponding to the k modal projection features respectively, the k modal projection features are weighted and summed to obtain a first historical conversation feature corresponding to the i-th historical time period of the first conversation object.

12. A data processing apparatus, characterized by Comprise: An acquisition module is configured to acquire N first historical conversation features corresponding to a first conversation object in N historical time periods, and acquire N second historical conversation features corresponding to a second conversation object in the N historical time periods; N is an integer greater than 1; the N historical time periods are time periods in which the first conversation object and the second conversation object have a conversation; A first processing module is configured to perform emotion feature padding processing on the N first historical conversation features to obtain N time sequence context conversation features of the first conversation object; A second processing module is configured to perform emotion feature padding processing on the N first historical conversation features according to the N second historical conversation features to obtain N interaction context conversation features between the first conversation object and the second conversation object; A prediction module is configured to predict a conversation emotion feature of the first conversation object at a current time according to the N time sequence context conversation features and the N interaction context conversation features, and generate a dialogue content of the second conversation object at the current time according to the conversation emotion feature; the current time is a time after the N historical time periods.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 11.

14. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the method in any one of claims 1 to 11.

Images