Emotion prediction method and system based on multi-agent spatiotemporal distribution vibration images

Abstract

The present invention provides an emotion prediction method and system based on multi-subject spatiotemporal distribution vibration images, the method comprising: obtaining facial video sub-images and spatiotemporal information of multiple subjects; performing time difference estimation on each facial video sub-image respectively to obtain the vibration image of each subject; performing minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject; calculating the current emotion index of each subject according to the vibration image of each subject; constructing spatiotemporal slices at each moment according to the category of each subject; predicting future spatiotemporal slices according to the spatiotemporal slices; restoring and reconstructing the future spatiotemporal slices to obtain the future emotion index of each subject. This method is based on the fact that the emotion change of the subject is affected by other subjects, and performs emotion prediction according to the relationship between the emotion and time and space changes among multiple subjects, as well as the influence of vibration, thereby improving the accuracy of emotion prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of software algorithms, and in particular relates to an emotion prediction method and system based on multi-subject spatiotemporal distribution vibration images. Background Art

[0002] Subject emotion prediction refers to the estimation and judgment of the emotional state that a specific subject may have in a specific situation. Its purpose is to understand the emotional trend of the subject in advance so as to better cope with and deal with related situations, such as predicting and preventing some mental illnesses or depression in advance. The current subject emotion prediction methods mainly analyze the past emotional expressions of specific subjects, evaluate the current environment and experienced events, and use relevant psychological, statistical and technical means to make predictions. However, due to the complexity of emotions and subject differences, the existing subject emotion prediction methods have certain uncertainties and limitations, resulting in inaccurate prediction results. Summary of the invention

[0003] In view of the defects in the prior art, the present invention provides an emotion prediction method and system based on multi-subject spatiotemporal distribution vibration images, which improves the accuracy of emotion prediction.

[0004] In a first aspect, an emotion prediction method based on multi-subject spatiotemporal distribution vibration images comprises:

[0005] Obtain facial video sub-images and spatiotemporal information of multiple subjects;

[0006] Performing temporal difference estimation on each facial video sub-image respectively to obtain the shaking image of each subject;

[0007] Perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject;

[0008] Calculate the current emotion index of each subject according to the vibration image of each subject;

[0009] Constructing a space-time slice at each moment according to the categories of each subject;

[0010] Predict future space-time slices based on space-time slices;

[0011] The future space-time slices are restored and reconstructed to obtain the future emotional index of each subject.

[0012] Furthermore, obtaining facial video sub-images and spatiotemporal information of multiple subjects specifically includes:

[0013] Acquire facial video images;

[0014] Identify the subject of the facial video image, and define the position information and time information of the subject in the facial video image as spatiotemporal information;

[0015] The facial video images are cut according to the subjects to obtain facial video sub-images of each subject.

[0016] Furthermore, the minimum correlation spatiotemporal intersection clustering is performed on the spatiotemporal information of all subjects to obtain the categories of each subject, including:

[0017] Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection;

[0018] The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

[0019] Furthermore, calculating the current emotion index of each subject according to the vibration image of each subject specifically includes:

[0020] Creating a hierarchical index recognition model that includes the mapping relationship between vibration images and emotion indices;

[0021] The vibration image is input into the grade index recognition model to obtain the current emotion index.

[0022] Furthermore, constructing the time-space slices at each moment according to the categories of each subject specifically includes:

[0023] In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects;

[0024] Construct the mapping relationship between each subject and the space-time slice in chronological order;

[0025] Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

[0026] Furthermore, predicting future space-time slices based on the space-time slices specifically includes:

[0027] Build predictive models;

[0028] L time-continuous space-time slices are input into the prediction model to obtain future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

[0029] In the second aspect, an emotion prediction system based on multi-subject spatiotemporal distribution vibration images includes:

[0030] Spatiotemporal data acquisition unit: used to obtain facial video sub-images and spatiotemporal information of multiple subjects; perform time difference estimation on each facial video sub-image to obtain a vibration image of each subject;

[0031] Clustering unit: used to perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject;

[0032] Emotion index calculation unit: used to calculate the current emotion index of each subject according to the vibration image of each subject;

[0033] Space-time slice construction unit: used to construct space-time slices at each moment according to the categories of each subject;

[0034] A space-time slice prediction unit: used to predict future space-time slices based on the space-time slices;

[0035] Emotional index reconstruction unit: used to restore and reconstruct future space-time slices to obtain the future emotional index of each subject.

[0036] Furthermore, the clustering unit is specifically used for:

[0037] Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection;

[0038] The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

[0039] Furthermore, the space-time slice construction unit is specifically used for:

[0040] In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects;

[0041] Construct the mapping relationship between each subject and the space-time slice in chronological order;

[0042] Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

[0043] Furthermore, the spatiotemporal slice prediction unit is specifically used for:

[0044] Build predictive models;

[0045] L time-continuous space-time slices are input into the prediction model to obtain future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

[0046] It can be seen from the above technical scheme that the emotion prediction method and system based on multi-subject spatiotemporal distributed vibration images provided by the present invention is based on the fact that the emotional changes of the subject are affected by other subjects, and emotion prediction is performed according to the relationship between the emotions and time and space changes among multiple subjects, as well as the impact of vibration, thereby improving the accuracy of emotion prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0048] Figure 1 A flowchart of the emotion prediction method provided in an embodiment.

[0049] Figure 2 A block diagram of a prediction model provided in an embodiment. DETAILED DESCRIPTION

[0050] The embodiments of the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and the protection scope of the present invention cannot be limited thereto. It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should be the common meanings understood by those skilled in the art to which the present invention belongs.

[0051] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0054] Example:

[0055] A method for predicting emotions based on multi-agent spatiotemporal distribution of vibration images, see Figure 1 ,include:

[0056] Obtain facial video sub-images and spatiotemporal information of multiple subjects;

[0057] Performing temporal difference estimation on each facial video sub-image respectively to obtain the shaking image of each subject;

[0058] Perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject;

[0059] Calculate the current emotion index of each subject according to the vibration image of each subject;

[0060] Constructing a space-time slice at each moment according to the categories of each subject;

[0061] Predict future space-time slices based on space-time slices;

[0062] The future space-time slices are restored and reconstructed to obtain the future emotional index of each subject.

[0063] In this embodiment, the method first obtains facial video sub-images and spatiotemporal information of multiple subjects, wherein the spatiotemporal information is generally information attached to the facial video sub-images. Then, the temporal difference estimation is performed on the facial video sub-images of each subject to obtain the vibration image I of each subject. vi，r Among them, temporal difference estimation (TD) is a commonly used method in reinforcement learning. It combines the idea of ​​dynamic programming and the sampling method of Monte Carlo method, and is faster.

[0064] In this embodiment, the method then constructs a spatiotemporal slice that reflects the relationship between multiple subjects, that is, the spatiotemporal information of all subjects is clustered with the minimum correlation spatiotemporal intersection to obtain the category of each subject; the current emotion index of each subject is calculated based on the vibration image of each subject; the spatiotemporal slice of each moment is constructed according to the category of each subject. Finally, the future spatiotemporal slice is predicted based on the spatiotemporal slice; the future spatiotemporal slice is restored and reconstructed to obtain the future emotion index of each subject.

[0065] This method is based on the fact that the emotional changes of a subject are affected by other subjects. It predicts emotions based on the relationship between emotions and time and space changes among multiple subjects, as well as the impact of vibration, thereby improving the accuracy of emotion prediction.

[0066] Furthermore, in some embodiments, obtaining facial video sub-images and spatiotemporal information of multiple subjects specifically includes:

[0067] Acquire facial video images;

[0068] Identify the subject of the facial video image, and define the position information and time information of the subject in the facial video image as spatiotemporal information;

[0069] The facial video images are cut according to the subjects to obtain facial video sub-images of each subject.

[0070] In this embodiment, the method first obtains a facial video image containing multiple subjects, and then uses a deep convolutional neural network to recognize the facial video image to identify all subjects in the facial video image, wherein subject recognition can be implemented using an existing face recognition method. The method can also set an identity number i for the subject; the subject's position information is Pos i ,Pos i =(x i ,y i ), x and y are two-dimensional position coordinate data; time information is t i，r , t i，r is the time of the i-th subject in the time series numbered r; finally, the facial video image is cut according to the subject to obtain the facial video sub-image I corresponding to each subject i，r , I i，r The facial video sub-image of the i-th subject numbered r in the time sequence.

[0071] In this embodiment, the method can also define a set P containing all subject record information, where P = {P1, P2, ..., P i , ..., P n}, where P i is the set of information recorded by the ith subject at different locations and times, n is the total number of subjects, P i =[P i，1 , P i，2 , ..., P i，r , ..., P i，T ], T is the length of the recorded time series, r is the time series number, P i，r =(t i，r ,Pos i，r , I vi，r ), I vi，r is the vibration image of the i-th subject in the time series numbered r.

[0072] Furthermore, in some embodiments, the minimum correlation space-time intersection clustering is performed on the space-time information of all subjects to obtain the categories of each subject, specifically including:

[0073] Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection;

[0074] The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

[0075] In this embodiment, the method performs minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections. For example, all space-time intersections can be represented by a set U, where U = {U1, U2, ..., U k , ..., U N},U k is the kth space-time intersection, k is the category of the space-time intersection; N is the total number of space-time intersections. Finally, the category of each subject is determined according to the space-time intersection into which each subject falls, and the category k is assigned to the corresponding subject.

[0076] Furthermore, in some embodiments, calculating the current emotion index of each subject according to the vibration image of each subject specifically includes:

[0077] Creating a hierarchical index recognition model that includes the mapping relationship between vibration images and emotion indices;

[0078] The vibration image is input into the grade index recognition model to obtain the current emotion index.

[0079] In this embodiment, the emotion index may include 12 categories, which are represented by a set E, where E = {E1, E2, ..., E j , ..., E 12}, E1-E 12 They represent aggression, stress, tension, suspicion, balance, depression, charm, vitality, self-discipline, neuroticism, and happiness respectively. Each category can also include ten levels. The level index recognition model is represented by g, so the recognition process of the emotional index can be described as: I vi，r →{I∩[1,10]} 12 , will vibrate the image I vi，r Mapped to a certain level under a certain category of the sentiment index. i，r =g(I vi，r ), e i，r ∈{I∩[1,10]} 12 The training method of the grade index recognition model g can be: using a conventional deep convolutional neural network to extract the characteristic parameters of the vibration image, labeling the characteristic parameters with the emotion index, inputting the characteristic parameters and the corresponding emotion index into the feedforward neural network, training the feedforward neural network, and obtaining the grade index recognition model g. This method can be used with the new set Q ir represents the data of the i-th subject record, Q ir =(t i，r ,Pos i，r , e i，r ).

[0080] Furthermore, in some embodiments, constructing a spatiotemporal slice at each moment according to the category of each subject specifically includes:

[0081] In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects;

[0082] Construct the mapping relationship between each subject and the space-time slice in chronological order;

[0083] Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

[0084] In this embodiment, the method constructs a spatiotemporal slice at each moment based on the spatiotemporal intersection obtained by clustering and the categories of each subject. M k，r is the space-time slice of the kth category in the time series numbered r. The space-time slice can be represented by a matrix; R is a set of real numbers. That is, the space-time slice is constructed according to the subject of each space-time intersection at each moment. The space-time slice can reflect the relationship between the subjects at that moment. The specific calculation steps include: first, according to the space-time intersection U k The spatiotemporal information of the internal subject, calculate the association relationship a between subject i and subject j ij，r :

[0085]

[0086] Wherein, L is the time window, which is a positive integer greater than 0, and is usually taken to correspond to a duration of three days. For example, assuming that the data collection cycle is δ hours, then L=72 / δ.

[0087] Then the method constructs the mapping relationship f k (i), map subject i to the space-time intersection U k , for example, convert i into the category k of the space-time intersection to complete the mapping. Finally, fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship, that is, M k，r (f k (i), f k (j)) = a ij，r .

[0088] Further, in some embodiments, predicting future space-time slices according to the space-time slices specifically includes:

[0089] Build predictive models;

[0090] L time-continuous space-time slices are input into the prediction model to obtain future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

[0091] In this embodiment, see Figure 2The prediction model may include a graph encoding neural network, a Transformer sequence prediction neural network and a graph decoding neural network. The graph encoding neural network is used to extract the features of the spatiotemporal slices. The Transformer sequence prediction neural network outputs the predicted feature data according to the features of the spatiotemporal slices. The graph decoding neural network is used to output the predicted spatiotemporal slices according to the predicted feature data. k，r (r=1, ..., T) is input into the prediction model to predict the future space-time slices. The space-time slices can be input into the prediction model in the form of a sliding time window, and the window size of the sliding time window is L, that is, one input {M k，r-L , M k，r-L+1 , ..., M k，r} to the prediction model, the prediction time length is also L, that is, the output {M k，r+1 , ..., M k，r+L}.

[0092] In this embodiment, after obtaining the above-mentioned future time-space slice, the method reversely restores and reconstructs the future time-space slice to obtain the subject's future emotion index, thereby realizing the subject's emotion prediction. If the change is steady and slow, the steps of restoration and reconstruction are:

[0093]

[0094] in, The predicted sentiment index for the ith subject in the future time series numbered s.

[0095] An emotion prediction system based on multi-agent spatiotemporal distribution vibration images, including

[0096] Spatiotemporal data acquisition unit: used to obtain facial video sub-images and spatiotemporal information of multiple subjects; perform time difference estimation on each facial video sub-image to obtain a vibration image of each subject;

[0097] Clustering unit: used to perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject;

[0098] Emotion index calculation unit: used to calculate the current emotion index of each subject according to the vibration image of each subject;

[0099] Space-time slice construction unit: used to construct space-time slices at each moment according to the categories of each subject;

[0100] A space-time slice prediction unit: used to predict future space-time slices based on the space-time slices;

[0101] Emotional index reconstruction unit: used to restore and reconstruct future space-time slices to obtain the future emotional index of each subject.

[0102] Furthermore, in some embodiments, the clustering unit is specifically used to:

[0103] Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection;

[0104] The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

[0105] Furthermore, in some embodiments, the spatiotemporal slice construction unit is specifically used to:

[0106] In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects;

[0107] Construct the mapping relationship between each subject and the space-time slice in chronological order;

[0108] Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

[0109] Further, in some embodiments, the spatiotemporal slice prediction unit is specifically used to:

[0110] Build predictive models;

[0111] L time-continuous space-time slices are input into the prediction model to obtain future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

[0112] The system provided in the embodiment of the present invention is briefly described. For matters not mentioned in the embodiment part, reference may be made to the corresponding contents in the aforementioned embodiment.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. An emotion prediction method based on multi-subject spatiotemporal distribution vibration images, characterized in that: include: Obtain facial video sub-images and spatiotemporal information of multiple subjects; Performing temporal difference estimation on each facial video sub-image respectively to obtain the shaking image of each subject; Perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject; Calculate the current emotion index of each subject according to the vibration image of each subject; Constructing a space-time slice at each moment according to the categories of each subject; predicting future space-time slices based on the space-time slices; Restoring and reconstructing the future space-time slices to obtain the future emotion index of each subject; The step of calculating the current emotion index of each subject according to the vibration image of each subject specifically includes: Creating a hierarchical index recognition model that includes the mapping relationship between vibration images and emotion indices; The vibration image is input into the level index recognition model to obtain the current emotion index.

2. The emotion prediction method based on multi-subject spatiotemporal distribution vibration images according to claim 1, characterized in that: The step of obtaining facial video sub-images and spatiotemporal information of multiple subjects specifically includes: Acquire facial video images; Identify the subject of the facial video image, and define the position information and time information of the subject in the facial video image as the spatiotemporal information; The facial video image is cut according to the subject to obtain the facial video sub-image of each subject.

3. The emotion prediction method based on multi-subject spatiotemporal distribution vibration images according to claim 1, characterized in that: The minimum correlation space-time intersection clustering of the space-time information of all subjects to obtain the categories of each subject specifically includes: Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection; The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

4. The emotion prediction method based on multi-subject spatiotemporal distribution vibration images according to claim 3 is characterized in that: The step of constructing a time-space slice at each moment according to the category of each subject specifically includes: In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects; Construct the mapping relationship between each subject and the space-time slice in chronological order; Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

5. The emotion prediction method based on multi-subject spatiotemporal distribution vibration images according to claim 4, characterized in that: Predicting future space-time slices according to the space-time slices specifically includes: Build predictive models; L time-continuous space-time slices are input into the prediction model to obtain the future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

6. An emotion prediction system based on multi-agent spatiotemporal distribution vibration images, characterized in that: It includes a spatiotemporal data acquisition unit: used to obtain facial video sub-images and spatiotemporal information of multiple subjects; perform time difference estimation on each facial video sub-image respectively to obtain a vibration image of each subject; Clustering unit: used to perform minimum correlation spatiotemporal intersection clustering on the spatiotemporal information of all subjects to obtain the category of each subject; Emotion index calculation unit: used to calculate the current emotion index of each subject according to the vibration image of each subject; Space-time slice construction unit: used to construct space-time slices at each moment according to the categories of each subject; A space-time slice prediction unit: used for predicting future space-time slices according to the space-time slices; Emotion index reconstruction unit: used for restoring and reconstructing the future space-time slices to obtain the future emotion index of each subject; The step of calculating the current emotion index of each subject according to the vibration image of each subject specifically includes: Creating a hierarchical index recognition model that includes the mapping relationship between vibration images and emotion indices; The vibration image is input into the level index recognition model to obtain the current emotion index.

7. The emotion prediction system based on multi-agent spatiotemporal distribution vibration images according to claim 6, characterized in that: The clustering unit is specifically used for: Perform minimum correlation space-time intersection clustering on the space-time information of all subjects to obtain multiple space-time intersections and the category of each space-time intersection; The category of each subject is determined according to the spatial and temporal intersection into which each subject falls.

8. The emotion prediction system based on multi-agent spatiotemporal distribution vibration images according to claim 7, characterized in that: The space-time slice construction unit is specifically used for: In each space-time intersection, the association relationship is calculated based on the space-time information of any two subjects; Construct the mapping relationship between each subject and the space-time slice in chronological order; Fill the association relationship of each subject into the mapped space-time slice according to the corresponding mapping relationship.

9. The emotion prediction system based on multi-agent spatiotemporal distribution vibration images according to claim 8, characterized in that: The spatiotemporal slice prediction unit is specifically used for: Build predictive models; L time-continuous space-time slices are input into the prediction model to obtain the future space-time slices; the prediction time length in the future space-time slices is L, where L is an integer.

Images