Method and device for evaluating attention level of target performance

Abstract

The present invention provides a method and device for evaluating the degree of attention of a target performance, and the method for evaluating the degree of attention of a target performance comprises: obtaining first brain wave information and first eye movement information of an audience watching a target performance; performing feature extraction on the first brain wave information to generate a target brain wave feature; performing feature extraction on the first eye movement information to generate a target eye movement feature; fusing the target brain wave feature and the target eye movement feature to generate target physiological information; based on the target physiological information, generating the audience's attention information on the target performance, the attention information being used to characterize the change in the audience's concentration on the target performance based on time. The method for evaluating the degree of attention of a target performance of the present invention can, on the one hand, convert abstract attention into quantitative information, and on the other hand, can significantly improve the pertinence and accuracy of the performance evaluation results.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method and device for evaluating the degree of attention of a target performance. Background Art

[0002] During a stage performance, it is often necessary to evaluate the performance content by analyzing the audience's attention to the performance. The existing methods for extracting audience attention are all based on obtaining data by having the subjects watch or listen to short-term videos or audios with guiding or emotional tendencies, which are less targeted and accurate. Summary of the invention

[0003] The present invention provides a method and device for evaluating the degree of attention of a target performance, so as to solve the defects of low pertinence and accuracy of performance evaluation results in the prior art and improve the pertinence and accuracy of performance evaluation results.

[0004] The present invention provides a method for evaluating the degree of attention of a target performance, comprising:

[0005] Obtain the first brain wave information and the first eye movement information of the audience watching the target performance;

[0006] Performing feature extraction on the first brain wave information to generate target brain wave features; performing feature extraction on the first eye movement information to generate target eye movement features;

[0007] fusing the target brain wave characteristics and the target eye movement characteristics to generate target physiological information;

[0008] Based on the target physiological information, the audience's attention information on the target performance is generated, and the attention information is used to characterize changes in the audience's concentration on the target performance based on time changes.

[0009] According to a target performance-oriented attention evaluation method provided by the present invention,

[0010] The step of extracting features from the first brain wave information to generate target brain wave features includes:

[0011] Preprocessing the first brain wave information to generate second brain wave information;

[0012] Performing feature extraction on the second brain wave information to generate the target brain wave feature;

[0013] and / or,

[0014] The extracting features of the first eye movement information to generate target eye movement features includes:

[0015] Preprocessing the first eye movement information to generate second eye movement information;

[0016] Perform feature extraction on the second eye movement information to generate the target eye movement feature.

[0017] According to a method for evaluating attention degree for a target performance provided by the present invention, extracting features from the second brain wave information to generate the target brain wave features includes:

[0018] Performing a time-domain-frequency-domain conversion on the second brain wave information to generate third brain wave information;

[0019] Based on the target frequency band, feature extraction is performed on the third brain wave information to generate first brain wave features corresponding to the target frequency band;

[0020] Connect multiple of the first brain wave features to generate the target brain wave feature.

[0021] According to a target performance-oriented attention evaluation method provided by the present invention, the feature extraction of the second eye movement information to generate the target eye movement feature includes:

[0022] Based on the second eye movement information, determining the center position of the eyeball;

[0023] Based on the eyeball center position point, an offset of the eyeball center position point and a time value corresponding to the offset are generated.

[0024] According to a method for evaluating the degree of attention of a target performance provided by the present invention, the step of fusing the target brain wave characteristics and the target eye movement characteristics to generate target physiological information includes:

[0025] Performing dimensionality reduction processing on the target brain wave feature to generate a second brain wave feature;

[0026] Temporally aligning the target eye movement feature with the second brain wave feature to generate a first eye movement feature;

[0027] The target physiological information is generated based on the correlation between the second brain wave feature and the first eye movement feature.

[0028] According to a method for evaluating the degree of attention of a target performance provided by the present invention, generating the audience's degree of attention information on the target performance based on the target physiological information includes:

[0029] Based on the target physiological information, the target performance is segmented and labeled based on a first target duration to generate the attention information.

[0030] The present invention also provides a target performance-oriented attention evaluation device, comprising:

[0031] An acquisition module, used to acquire the first brain wave information and the first eye movement information of the audience watching the target performance;

[0032] A feature extraction module, configured to extract features from the first brain wave information to generate target brain wave features; and extract features from the first eye movement information to generate target eye movement features;

[0033] A feature fusion module, used to fuse the target brain wave feature and the target eye movement feature to generate target physiological information;

[0034] A generating module is used to generate the audience's attention information to the target performance based on the target physiological information, wherein the attention information is used to characterize the change of the audience's concentration on the target performance based on time.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for evaluating the degree of attention of a target performance as described above is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for evaluating the degree of attention of target performances.

[0037] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for evaluating the degree of attention of target performance.

[0038] The target performance-oriented attention evaluation method and device provided by the present invention generate target physiological information by extracting and fusing the features of the first brain wave information and the first eye movement information, and mark the target performance based on the target physiological information to generate attention information. On the one hand, it can convert abstract attention into quantitative information, and on the other hand, by analyzing the physiological information of the audience as the evaluation information for evaluating the target performance, it can significantly improve the pertinence and accuracy of the performance evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is one of the flow charts of the target performance-oriented attention evaluation method provided by the present invention;

[0041] Figure 2 This is the second flow chart of the method for evaluating the degree of attention of target performance provided by the present invention;

[0042] Figure 3 This is the third flow chart of the target performance-oriented attention evaluation method provided by the present invention;

[0043] Figure 4 This is one of the principle schematic diagrams of the target performance-oriented attention evaluation method provided by the present invention;

[0044] Figure 5 This is the second principle schematic diagram of the target performance-oriented attention evaluation method provided by the present invention;

[0045] Figure 6 This is a fourth flow chart of the target performance-oriented attention evaluation method provided by the present invention;

[0046] Figure 7 This is the third principle schematic diagram of the target performance-oriented attention evaluation method provided by the present invention;

[0047] Figure 8 This is the fourth principle schematic diagram of the target performance-oriented attention evaluation method provided by the present invention;

[0048] Fig. 9 This is the fifth principle schematic diagram of the target performance-oriented attention evaluation method provided by the present invention;

[0049] Fig.10 It is a structural schematic diagram of a target performance-oriented attention evaluation device provided by the present invention;

[0050] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Combine the following Figures 1 to 9 The present invention describes the attention evaluation method for target performance.

[0053] It should be noted that the executor of the target performance-oriented attention evaluation method may be a target performance-oriented attention evaluation device; or a server; or a user's terminal, including but not limited to the user's mobile phone, computer, and tablet computer.

[0054] like Figure 1 As shown, the method for evaluating the degree of attention for target performance includes: step 110, step 120, step 130 and step 140.

[0055] Step 110, obtaining the first brain wave information and the first eye movement information of the audience watching the target performance;

[0056] In this step, the target performance is the performance that needs to be evaluated based on the attention level.

[0057] For example, the target performance may be a live stage performance, a virtual stage performance, a TV series, a movie, or other performance videos.

[0058] The target performance forms may include, but are not limited to, stage performances such as dramas and ballets, as well as other forms of performances such as opening ceremonies, trailers, and dances.

[0059] The proportion of each form of the target performance can be customized based on the user, such as setting the proportion of stage performances to 60% and the proportion of other forms of performances to 40%, which is not limited in the present invention.

[0060] Brain wave information is an electrical signal generated by the activity between brain neurons. It is used to characterize the changes in electrical waves during brain activity and is the overall reflection of the electrophysiological activity of brain nerve cells on the cerebral cortex or scalp surface.

[0061] Brain wave information can be collected through wearable EEG collection devices. For example, EMOTIVEpoc+ can be used to collect brain wave information. When the audience is watching the target performance, electrodes are set on the audience's scalp to detect brain wave signals, and then the EEG collection device is used to collect brain waves, thereby realizing the collection of brain wave signals of the audience while watching the target performance.

[0062] Eye movement information is used to characterize eye movements, which include pursuit movements and saccades.

[0063] Eye movement information can be collected by an eye tracker or sensor. For example, by using a camera to track the position change information of the audience's eyes when they watch a target performance, eye movement information is generated based on the position change information.

[0064] It should be noted that when using a camera to collect eye movement information, it is necessary to ensure that the camera can continuously and stably track the eye position and calibrate the camera before collection.

[0065] In the actual implementation process, calibration can be performed through the following steps:

[0066] 1) Taking the target performance as a video performance, adjust the screen height so that the audience's horizontal line of sight is at one quarter of the screen;

[0067] 2) Adjust the camera position so that the distance from the camera to the audience's eyes is between 40-70cm (ideally 55-60cm);

[0068] 3) Adjust the lens height so that the eyes are located in the center of the camera lens;

[0069] 4) Adjust the lens angle. For viewers who wear glasses, it is necessary to adjust the angle between the lens and the glasses according to the actual situation, so as to change the angle of reflected light and increase the light rate.

[0070] The lens position is adjusted and calibration is officially carried out.

[0071] Make sure the audience stares at the calibration point on the screen until the current point disappears and then looks for the next point, and keep staring at each point for 5 seconds. Finally, the calibration result is obtained, among which a good calibration shows the characteristics of "rows in rows and columns in columns".

[0072] The first brain wave information is the collected initial brain wave information without being processed, and the first eye movement information is the collected initial eye movement information without being processed.

[0073] The audience may be one audience or multiple audiences, and each audience has corresponding first brain wave information and first eye movement information.

[0074] It is understandable that for the same target performance, the first brain wave information and the first eye movement information corresponding to different audiences may be different.

[0075] During the preliminary training process, the audience can be a sample audience. When selecting a sample audience, audiences of different age groups, professional backgrounds, genders, and cultural backgrounds can be selected to collect physiological information to improve the authenticity and accuracy of the training results.

[0076] In the later application process, the audience can be the live audience watching the performance.

[0077] The following will take the preliminary training as an example, that is, the audience is a sample audience, and the target performance is a sample performance video as an example to illustrate this step.

[0078] like Figure 2 Shown is an environment for collecting first brain wave information and first eye movement information. Figure 3 A process time allocation is shown.

[0079] In the actual implementation process, Figure 3 As shown, before data collection began, each audience member was given an EEG cap and a VR headset and had a few minutes of adaptation time before data collection began.

[0080] After the data collection started, the audience was asked to keep their eyes open for one minute and look at a fixed point on the wall for EEG signal recording calibration, and then close their eyes and rest for one minute. The audience had ten minutes to familiarize themselves with the test environment.

[0081] Each viewer needs to go through two test stages and watch a total of 60 sample video clips (each video is about 1 to 5 minutes long), with a 60-second break in the three stages.

[0082] In the first test phase, a viewer was randomly selected to watch the first 20 sample videos in a fixed order. After the test, the viewer conducted a stress assessment with a score from 1 to 10 (1 = "no stress", 10 = "extremely stressful") and a SAM self-assessment report.

[0083] After the first round of testing, the audience kept wearing the headset normally and rested for 60 seconds.

[0084] In the second test phase, the audience watched the last 20 sample videos in a fixed order, and were evaluated again at the end of the test. They then kept the headset on normally, rested for 60 seconds, and then watched the third round of 20 videos.

[0085] It should be noted that during the data collection process, the environment must be kept constant, such as temperature, noise, air circulation, etc.

[0086] It should be noted that after obtaining the first brain wave information and the first eye movement information of the audience watching the target performance, the above information can be stored in a local or cloud database and retrieved when needed.

[0087] Step 120: extracting features from the first brain wave information to generate target brain wave features; extracting features from the first eye movement information to generate target eye movement features;

[0088] In this step, the target brain wave feature is the brain wave feature generated after preprocessing and feature extraction of the first brain wave information.

[0089] The target brain wave feature can be expressed as a feature vector of the brain wave signal within the target frequency band.

[0090] The target eye movement feature is an eye movement feature generated after preprocessing and feature extraction of the first eye movement information.

[0091] The target eye movement feature may be expressed as at least one of a blinking state, an eye gesture state, and an offset of an eyeball center position point.

[0092] The eye gesture state is used to characterize the direction of the eye's gaze, and the eye gesture state may include but is not limited to: gaze to the left, gaze to the right, and gaze upward.

[0093] Blinking states may include but are not limited to: single blink and double blink, etc.

[0094] The offset of the eyeball center position point is the offset between the actual coordinates of the eyeball center position point and the initial coordinates. The initial coordinates are the coordinates of the eyeball center position point of the viewer in a normal state.

[0095] Of course, in other embodiments, the target eye movement feature may also be expressed as an offset of other positions of the eyeball, which is not limited in the present invention.

[0096] In the actual implementation process, a neural network model can be used to extract features of the first brain wave information and the first eye movement information.

[0097] The specific implementation process will be described in detail in the subsequent embodiments and will not be elaborated here.

[0098] Step 130: Fusing target brain wave features and target eye movement features to generate target physiological information;

[0099] In this step, the target physiological information is a fusion feature used to simultaneously characterize changes in brain waves and changes in eye movements.

[0100] In the actual execution process, the target function between the target brain wave characteristics and the target eye movement characteristics can be constructed based on the changes in the target brain wave characteristics and the target eye movement characteristics, and the correlation coefficient between the target brain wave characteristics and the target eye movement characteristics can be generated based on the target function. The correlation coefficient is the target physiological information.

[0101] For example, the horizontal coordinate of the target function can be set to the target brain wave feature, the vertical coordinate of the target function can be set to the target eye movement feature, the target function can be generated, and the correlation coefficient can be generated based on the target function.

[0102] For another example, the ordinate of the target function may be set to the target brain wave feature, and the abscissa of the target function may be set to the target eye movement feature to generate a target function image, and a correlation coefficient may be generated based on the target function image.

[0103] Of course, in other embodiments, other fusion methods may be used to generate target physiological information for simultaneously characterizing target brain wave characteristics and target eye movement characteristics, such as establishing a data comparison table between target brain wave characteristics and target eye movement characteristics, etc., which is not limited by the present invention.

[0104] The implementation of step 130 is described in detail below.

[0105] In some embodiments, step 130 may include:

[0106] Perform dimensionality reduction processing on the target brain wave feature to generate a second brain wave feature;

[0107] Temporally aligning the target eye movement feature with the second brain wave feature to generate a first eye movement feature;

[0108] Based on the correlation between the second brain wave feature and the first eye movement feature, target physiological information is generated.

[0109] In this embodiment, the target eye movement feature is a high-dimensional feature, and the second brain wave feature is a low-dimensional feature.

[0110] In the actual implementation process, the principal component analysis PCA (Principal Component Analysis) can be used for dimensionality reduction. For example, PCA can be used to decentralize the target brain wave feature to determine the origin, determine the direction of maximum variance as the coordinate system, perform feature dimensionality reduction and standardization, and generate the second brain wave feature after dimensionality reduction.

[0111] PCA converts multiple indicators into a few and comprehensive indicators through coordinate transformation to achieve dimensionality reduction and obtain information related to the target EEG characteristics.

[0112] PCA can map high-dimensional features to a new coordinate system by rotation and stretching, thereby minimizing information loss.

[0113] like Figure 6 The specific implementation process is as follows:

[0114] 1) Decentralize the origin. The new coordinate origin is placed in the data center. Assuming the original data sample number of the target EEG feature is n, the feature number is m, and it is arranged into an m×n matrix X. The x sample set is generated, and each row is averaged to "0". The formula can be used:

[0115]

[0116] Perform averaging, where is the sample mean of the x sample set, n is the number of original data samples of the target EEG feature, i is the i-th sample, X i is the i-th sample value of the x sample set.

[0117] 2) Determine the direction with the largest variance. Perform a linear transformation (stretching and rotating) on ​​the original data of the target EEG feature to find the direction with the largest variance as the coordinate system. Determine the rotation direction based on the eigenvalue and eigenvector of the covariance matrix, such as through the formula:

[0118]

[0119] in,

[0120] Determine the rotation direction, where n is the number of original data samples of the target brain wave feature, x is the sample set corresponding to the x dimension, and y is the sample set corresponding to the y dimension. is the sample mean of the x sample set, is the sample mean of the y sample set, x i is the i-th sample value of the x sample set, y i is the i-th sample value of the y sample set, and i is the i-th sample.

[0121] 3) Feature dimensionality reduction processing: Calculate the eigenvalues ​​and eigenvectors based on the covariance matrix, sort them from large to small, and form the first k rows into a matrix P, Y = KP, which represents k-dimensional data.

[0122] 4) Standardization. Normalization is the process of limiting the feature value range to [0.1, 0.9]. The specific formula is:

[0123]

[0124] Normalization is performed, where X' is the sample value after normalization, X max is the maximum sample value in the sample set, X min is the minimum sample value in the sample set.

[0125] Through the above steps, the second brain wave feature can be generated.

[0126] The annotation file of the second EEG feature generated by the above steps is in txt format, and its content may include [stimulus material file name, segment start time, segment end time, attention characteristics]. The time unit can be set to / s, which can be accurate to 0.1s.

[0127] Of course, in other embodiments, the time unit or precision may also be set to other units or values, which is not limited in the present invention.

[0128] In this embodiment, after the target brain wave features are reduced in dimension by PCA, on the one hand, the dimensionality disaster can be avoided by discarding part of the information to reduce the data dimension, thereby solving the problem of too high dimension; on the other hand, effective noise reduction can be performed. Since EEG acquisition is susceptible to noise interference, the second noise usually corresponds to the minimum eigenvalue and its corresponding eigenvector. By discarding this information through the PCA method, noise reduction can be performed again on the basis of data preprocessing; on the other hand, the independence of features can be ensured.

[0129] After the second brain wave feature is generated, the target eye movement feature is time-aligned with the second brain wave feature to generate the first eye movement feature, thereby avoiding calculation errors caused by time difference and improving the accuracy of the calculation results.

[0130] In the actual implementation process, the target eye movement features can be aligned at the time level based on the second target duration.

[0131] The second target duration is a standard time interval used to align the second brain wave feature and the target eye movement feature at the time level.

[0132] The second target duration can be customized based on the audience, such as being set to 3s or 5s, etc., which is not limited in the present invention.

[0133] It should be noted that the target eye movement information may also be processed in a normalized manner, and the processing steps are the same as those in the above embodiment, which will not be described in detail here.

[0134] The following takes the target performance duration as 60 seconds and the second target duration as 3 seconds as an example to specifically illustrate this step.

[0135] In this case, the target performance can be divided into 20 segments of fixed length, each segment corresponds to a target performance segment, and the target performance segment corresponds to the second brain wave characteristics and the target eye movement characteristics, and the occurrence time of the two in the target performance segment should be consistent.

[0136] This embodiment is described by taking the Pearson correlation coefficient as the target eye movement feature, the offset of the eyeball center position point, and calculating the target eye movement feature within 3 seconds as an example.

[0137] The current target performance segment is recorded as n, n∈(2,n), the previous segment is numbered as n-1, and the transformation of the eyeball center position is recorded as X, x i ∈X,yi∈Y,

[0138] By formula:

[0139]

[0140] The first eye movement feature can be generated, where r n is the first eye movement feature, is the mean of X, is the mean of Y, s X is the variance of X, s Y is the variance of Y, n is the total number of segments, X is the transformation of the center position of the eyeball, and Y is the transformation of the center position on the y axis.

[0141] The first eye movement feature can be obtained through the above steps. The annotation format of the first eye movement feature is the same as that in the above embodiment, and will not be described herein.

[0142] After obtaining a plurality of first eye movement features, target physiological information may be generated based on the correlation between the second brain wave features and the first eye movement features.

[0143] Specifically, it can be manifested as:

[0144] 1) Data alignment. When calculating the fusion features of the two, it is necessary to ensure that the time of the two based on the target performance is aligned. Taking 3s as the second target duration as an example, the occurrence time of the second EEG feature and the first eye movement feature within the same 3s must be consistent;

[0145] 2) Correlation coefficient calculation. EEG (second brain wave feature) is used as the X-axis, and the longitudinal data change of ET (first eye movement feature) is used as the Y-axis. The Pearson correlation coefficient within the second target duration is calculated respectively. The specific calculation method is the same as the calculation of the first eye movement feature mentioned above, so it will not be repeated here.

[0146] Through the above steps, the target physiological information can be generated, such as Figure 8 As shown, the target physiological information fusion feature annotation format is the same as above and will not be repeated here.

[0147] Step 140: Generate audience attention information on the target performance based on the target physiological information, where the attention information is used to characterize how the audience's concentration on the target performance changes over time.

[0148] In this step, the attention information is used to characterize the audience's interest in the target performance.

[0149] It is understandable that when the same audience is watching the same target performance, their attention level changes in real time, and the attention level may be different at different times.

[0150] The target physiological information is a measure that converts the abstract degree of attention into quantitative data.

[0151] The target physiological information and the target physiological information annotation file have been generated by step 130 , and each target physiological information corresponds to a time value.

[0152] Based on this moment value, the target physiological information corresponding to each moment of the target performance can be marked respectively, thereby realizing the time-level marking of the target performance and generating a target performance marking file based on the target physiological information marking. The marking file is the audience's attention information to the target performance.

[0153] It should be noted that the attention information is used to characterize the change in the audience's attention to the entire target performance over time. The attention information is dynamic and changes in real time. For the same target performance, the attention at different times may be different.

[0154] During the research and development process, the inventors discovered that related technologies mainly rely on the subjects watching or listening to short-term videos or audios with guiding or emotional tendencies to obtain the audience's attention to the stage. The data obtained are abstract and rough, and not targeted.

[0155] In the present invention, the target physiological information of the audience for any entire target performance is obtained, and the audience's attention to the target performance at the minute-by-second level is marked based on the obtained target physiological information to generate target physiological information. This can not only convert abstract attention into quantitative information, but also significantly improve the precision and accuracy of data results.

[0156] In some embodiments, step 140 may include: based on the target physiological information, segmentally labeling the target performance based on the first target duration to generate attention information.

[0157] In this embodiment, the first target duration is a standard time interval used to align the target physiological information with the target performance at the time level.

[0158] The first target duration may be customized based on the audience, such as being set to 3s or 5s, etc., which is not limited in the present invention.

[0159] The first target duration may be equal to the second target duration.

[0160] Based on the target physiological information, the target performance is annotated in seconds to generate attention information, namely the Multimodal dataset annotation, such as Fig. 9 shown.

[0161] Of course, continue to refer to Fig. 9 In other embodiments, the target performance can be segmented and labeled at the second level based on the target brain wave characteristics or the target eye movement characteristics to generate a Single-EEG corresponding to the brain wave characteristics or a Single-ET corresponding to the eye movement characteristics.

[0162] According to the attention evaluation method for target performance provided by an embodiment of the present invention, target physiological information is generated by extracting and fusing features of the first brain wave information and the first eye movement information, and the target performance is labeled based on the target physiological information to generate attention information. On the one hand, abstract attention can be converted into quantitative information. On the other hand, by analyzing the physiological information of the audience as evaluation information for evaluating the target performance, the pertinence and accuracy of the performance evaluation results can be significantly improved.

[0163] The following describes step 120 of the embodiment of the present application from two different implementation perspectives.

[0164] 1. Feature extraction of the first brain wave information

[0165] In some embodiments, extracting features from the first brainwave information to generate target brainwave features in step 120 may include:

[0166] Preprocessing the first brain wave information to generate second brain wave information;

[0167] Feature extraction is performed on the second brain wave information to generate target brain wave features.

[0168] In this embodiment, the preprocessing process includes: importing data, lead positioning, removing useless leads, re-referencing, filtering, extracting segments, ICA, denoising, and removing segments.

[0169] The second brain wave signal is an electrical signal within a target frequency band, wherein the target frequency band is a frequency band required by the audience, and the target frequency band can be customized based on the audience.

[0170] For example, the target frequency band can be set to β, α, and θ frequency bands.

[0171] Among them, the beta signal frequency band (14-30Hz) has a small amplitude and a large frequency, and is emotionally related to anger, rage, fear and stress.

[0172] The waveform amplitude of the α signal frequency band (8-13.9Hz) is larger than that of β and the frequency is lower, which mainly represents the state of relaxation.

[0173] The waveform in the theta signal frequency band (4-7.9Hz) has a larger amplitude and a lower frequency, mainly representing a deep meditation unconscious state.

[0174] The σ signal frequency band (1-3.9 Hz) has the largest amplitude and the lowest frequency and is used to characterize the sleep state.

[0175] It can be understood that the collected first brain wave information includes signals of multiple frequency bands, and the second brain wave information is obtained by performing preprocessing such as noise reduction to remove unnecessary electrical signals and retaining signals of the β, α and θ frequency bands.

[0176] The following example uses EMOTIVEpoc+ to collect 14 first brain wave information of 7 electrodes symmetrical about the temporal lobe: IO1, IO2, AF3, AF4, F7, F3, F4, F8, FC5, FC6, T7, T8, P7 and P8. Figure 4 This step is described as shown.

[0177] In actual implementation, the EEGLABToolBox filter can be used to retain the β, α, and θ signal bands and remove eye movement interference caused by eye movement during signal acquisition.

[0178] When eye movements and EEG oscillations were comparable, runICA of EEGLAB was used to remove eye movement artifacts.

[0179] Finally, for all electrodes, the common average reference value is calculated and removed to generate the second brain wave information.

[0180] After obtaining the second brain wave information, feature extraction is performed on the second brain wave information to generate target brain wave features.

[0181] In the actual implementation process, feature extraction can be performed through a neural network model.

[0182] In this embodiment, the first brain wave information is preprocessed to remove interference signals, thereby reducing the amount of data, which helps to improve the rate of subsequent operations and the accuracy of calculation results.

[0183] In some embodiments, extracting features from the second brainwave information to generate target brainwave features may include:

[0184] Performing time-domain-frequency-domain conversion on the second brain wave information to generate third brain wave information;

[0185] Based on the target frequency band, feature extraction is performed on the third brain wave information to generate the first brain wave feature corresponding to the target frequency band;

[0186] Connect multiple first brain wave features to generate a target brain wave feature.

[0187] In this embodiment, it should be noted that the first brain wave information is a finite discrete time series data composed of a plurality of sample points, and the number of sample points is determined by the sampling rate. For example, the sampling rate can be set to 128 Hz.

[0188] The second brain wave information is also a time domain signal. Before feature extraction, it needs to be converted into a frequency domain signal to generate the third brain wave information.

[0189] In the actual implementation process, discrete Fourier transform (DFT) can be used to obtain spectrum information, which is then decomposed into four frequency bands β, α, θ, and σ related to psychological activities and emotional expressions, and then the power spectral density (PSD) is used to calculate the characteristics of each channel.

[0190] The PSD solution process is: S(f) is a mapping with frequency f as the independent variable, and S(f) is the power value of the frequency component f where the signal is located.

[0191] The specific formula is:

[0192]

[0193] Perform signal sampling, where X t0 (t) is the sample set of signal sampling, t is the time value, t0 is the truncation threshold. If the threshold is exceeded, signal sampling will no longer be performed.

[0194] The energy of this random process is defined as:

[0195]

[0196] Wherein, t0 is the cutoff threshold, and X(t) is the signal sample of the second brain wave information in the time domain.

[0197] The average power is obtained by derivation of the energy function as follows:

[0198]

[0199] Among them, P Xt0 is the average power of the second brain wave information, t0 is the cutoff threshold, and X(t) is the signal sample of the second brain wave information in the time domain.

[0200] According to Parseval's theorem, energy is equal from the perspective of time domain and frequency domain, X t0 (f) is X t0 (t) After DFT transformation, the overall average power is as follows:

[0201]

[0202] Among them, P Xt0 is the average power of the second brain wave information, t0 is the cutoff threshold, X(f) is the signal sample of the second brain wave information in the frequency domain, and f is the frequency.

[0203] When t0 approaches infinity, the average power of the original random process The formula can be:

[0204]

[0205] Get, where is the average power of the original random process, and t0 is the cutoff threshold.

[0206] By extracting the integrand alone, we can get the PSD function S(f), where:

[0207]

[0208] In the actual implementation process, the Welch method of classic spectrum estimation can be used, and the "pwelch function" in Matlab can be called to calculate and average the PSD power spectrum density of each segment of data to obtain the PSD of the entire signal. After EEG preprocessing, the EEG signal is divided into theta (4Hz~8Hz), alpha (8Hz~13Hz) and beta (13Hz~20Hz), and the features of the 14-channel EEG data extracted by EMOTIVEpoch+ are extracted respectively, and a total of 42 features are obtained, which are multiple first brain wave features.

[0209] After obtaining multiple first brain wave features, multiple first brain wave features are connected, such as connecting 42 features into the final feature vector FEEG: Define F iθ , F iα , F iβ represents the PSD of theta, alpha, beta of the ith electrode, i = 1, 2, ..., 14, and the eigenvector of EEG is F EEG =[F 1θ F 1α F 1β ...F 1θ F 14α F 14β ], the feature vector is the target brain wave feature.

[0210] The target brain wave features extracted by this method have high precision and accuracy.

[0211] 2. Feature extraction of the first eye movement information

[0212] In some embodiments, extracting features from the first eye movement information to generate target eye movement features in step 120 may include:

[0213] Preprocessing the first eye movement information to generate second eye movement information;

[0214] Feature extraction is performed on the second eye movement information to generate target eye movement features.

[0215] In this embodiment, the first eye movement information is preprocessed, including tracking the location of human eyes, acquiring eye movement information, and identifying behavior.

[0216] The following takes the use of a front camera to collect the first eye movement information as an example to illustrate this step.

[0217] In the actual implementation process, eye movement recognition based on LSTM can be used to integrate the gradient directional histogram (HOG) features, SVM model and LSTM neural network to complete face positioning and tracking, obtain the coordinates of the key points of the eye corners of both eyes, judge the eye state, and obtain the first eye movement information.

[0218] After obtaining the first eye movement information, the first eye movement information is preprocessed through human eye positioning tracking, eye movement information acquisition and behavior recognition, as shown in the following:

[0219] 1) Eye positioning. The first frame of the video stream uses HOG features to locate the face, and based on the regression tree method, completes face alignment and key point positioning, such as Figure 5 shown.

[0220] The four eye corners can be marked as "37, 40, 43, 46" from left to right, and the position coordinates of each eye corner are recorded as (x1, y1), (x2, y2), (x3, y3), and (x4, y4) respectively.

[0221] The binocular rectangular area is calculated by the formula:

[0222] width=(x2+x4-x1-x3) / 2

[0223] eyedist=(x3-x2) / 2

[0224] Calculate, where width is the average width of both eyes and eyedist is the distance between the two eyes.

[0225] 2) Eye tracking. Continue to refer to Figure 5 , the rectangular frame area containing both eyes, remember that the upper left and lower right coordinates of the rectangle are (X1, Y1) and (X2, Y2) respectively. Starting from the second frame, match the eye area frame by frame, and then expand the face area based on basic experience and the "three courts and five eyes" ratio rule, and repeat this, so as to use the formula:

[0226] X1=x1-eyedist

[0227] Y1=y1-width×2 / 3

[0228] X2=X4-eyedist

[0229] Y2=y2+width×2 / 3

[0230] You can perform eye tracking, where width is the average width of both eyes, eyedist is the distance between the two eyes, (X1, Y1) is the second eye movement information corresponding to the left eye, and (X2, Y2) is the second eye movement information corresponding to the right eye.

[0231] After obtaining the second eye movement information, feature extraction is performed on the second eye movement information to generate target eye movement features.

[0232] In some embodiments, extracting features from the second eye movement information to generate target eye movement features may include:

[0233] Determine the center position of the eyeball based on the second eye movement information;

[0234] Based on the eyeball center position point, an offset of the eyeball center position point and a time value corresponding to the offset are generated.

[0235] In this embodiment, the offset of the center position point of the eyeball is used to characterize the change of the eyeball over time.

[0236] It can be understood that when the center position of the eyeball shifts upward, it indicates that the viewer's line of sight is upward; when the center position of the eyeball shifts downward, it indicates that the viewer's line of sight is downward; when the center position of the eyeball shifts to the left, it indicates that the viewer's line of sight is to the left; when the center position of the eyeball shifts to the right, it indicates that the viewer's line of sight is to the right.

[0237] Each offset corresponds to a time value, which is used to represent the line of sight information at the current moment.

[0238] The following uses a single eye as an example to illustrate how to determine the center position of the eyeball.

[0239] 1) Calculate the eye center point and expand it to the left and right by 4h×3 pixels and to the top and bottom by 2 pixels according to the empirical rule of “three courts and five eyes”;

[0240] 2) Adjustment and enhancement of eye images: crop the eye images to 36×36 pixels, use edge-preserving filtering to remove noise, improve regional contrast and eliminate shadows;

[0241] 3) Determine the center position of the eyeball. Calculate the gradient direction and amplitude in the rectangular area of ​​both eyes, and select the point where the gradient directions intersect the most as the center of the eyeball, recorded as (Xc, Yc), where (Xc, Yc) is the coordinate of the center position of the eyeball.

[0242] By collecting the coordinate values ​​of (Xc, Yc) within the target period and the corresponding time values ​​of each (Xc, Yc), the offset of the center position of the eyeball and the time value corresponding to the offset can be generated, such as Figure 7 shown.

[0243] The inventors have found through multiple experiments that the y-axis coordinate variation of the eyeball center position shows a more complex and obvious variation than the x-axis, and therefore can better reflect the degree of attention paid to the sample.

[0244] In this embodiment, by selecting the offset of the center position of the eyeball as the target eye movement feature, higher precision and accuracy are achieved, and the targeting is better.

[0245] In other embodiments, the target eye movement feature may also be expressed as an eye gesture feature.

[0246] Before feature extraction, eye movement behavior needs to be defined.

[0247] For example, eye movement behavior can be divided into five states: blink judgment and eye gesture judgment: single blink, double blink, gaze to the left, gaze to the right, and gaze to the top. Among them, double blink is defined as unintentional eye gesture, and the rest are intentional eye gestures. The judgment method is as follows.

[0248] like Figure 5 As shown, according to the face positioning points, the key points 37 to 48 around the two eyes are obtained. Taking the left eye as an example, "37 to 42" are recorded in a clockwise direction, where "37 to 42" correspond to "p1 to p6" in sequence. According to the formula:

[0249]

[0250] The left eye aspect ratio can be calculated, where W is the left eye aspect ratio.

[0251] The calculation method for the right eye is similar and will not be described in detail here.

[0252] After obtaining the aspect ratio of the eyes, if the aspect ratio of one eye is less than 0.2, the eye is judged to be in a blinking state. When the difference between the aspect ratios of the two eyes is greater than 0.05, it is defined as an intentional blink.

[0253] Then by the formula:

[0254] t x =X c / 36

[0255] t y =Y c / 36

[0256] Determine the eye posture, where (Xc, Yc) is the coordinate of the center of the eyeball, t x is the horizontal eye posture, t y It is the vertical eye posture.

[0257] When tx<0.30, the gaze is to the left; when tx>0.60, the gaze is to the right; when ty>0.62, the gaze is upward. The proportion of unintentional eye gesture time to the total time in a unit segment is used to represent the degree of attention, and the target eye movement feature can be generated. The value range of the target eye movement feature is (0, 1), and the value is inversely proportional to the degree of attention PA.

[0258] The following is a description of the attention evaluation device for target performance provided by the present invention. The attention evaluation device for target performance described below and the attention evaluation method for target performance described above can be referenced to each other.

[0259] like Fig.10 As shown, the target performance-oriented attention evaluation device includes: an acquisition module 1010, a feature extraction module 1020, a feature fusion module 1030 and a generation module 1040.

[0260] An acquisition module 1010 is used to acquire first brain wave information and first eye movement information of the audience watching the target performance;

[0261] The feature extraction module 1020 is used to extract features from the first brain wave information to generate target brain wave features; extract features from the first eye movement information to generate target eye movement features;

[0262] A feature fusion module 1030 is used to fuse target brain wave features and target eye movement features to generate target physiological information;

[0263] The generating module 1040 is used to generate the audience's attention information to the target performance based on the target physiological information, where the attention information is used to characterize the change of the audience's concentration on the target performance based on time.

[0264] According to the attention evaluation device for target performance provided by the embodiment of the present invention, target physiological information is generated by extracting and fusing features of the first brain wave information and the first eye movement information, and the target performance is labeled based on the target physiological information to generate attention information. On the one hand, abstract attention can be converted into quantitative information. On the other hand, by analyzing the physiological information of the audience as evaluation information for evaluating the target performance, the pertinence and accuracy of the performance evaluation results can be significantly improved.

[0265] In some embodiments, the feature extraction module 1020 is further configured to:

[0266] Preprocessing the first brain wave information to generate second brain wave information;

[0267] Feature extraction is performed on the second brain wave information to generate target brain wave features.

[0268] In some embodiments, the feature extraction module 1020 is further configured to:

[0269] Preprocessing the first eye movement information to generate second eye movement information;

[0270] Feature extraction is performed on the second eye movement information to generate target eye movement features.

[0271] In some embodiments, the feature extraction module 1020 is further configured to:

[0272] Performing time-domain-frequency-domain conversion on the second brain wave information to generate third brain wave information;

[0273] Based on the target frequency band, feature extraction is performed on the third brain wave information to generate the first brain wave feature corresponding to the target frequency band;

[0274] Connect multiple first brain wave features to generate a target brain wave feature.

[0275] In some embodiments, the feature extraction module 1020 is further configured to:

[0276] Determine the center position of the eyeball based on the second eye movement information;

[0277] Based on the eyeball center position point, an offset of the eyeball center position point and a time value corresponding to the offset are generated.

[0278] In some embodiments, the feature fusion module 1030 is further used to:

[0279] Perform dimensionality reduction processing on the target brain wave feature to generate a second brain wave feature;

[0280] Temporally aligning the target eye movement feature with the second brain wave feature to generate a first eye movement feature;

[0281] Based on the correlation between the second brain wave feature and the first eye movement feature, target physiological information is generated.

[0282] In some embodiments, the generation module 1040 is further used to: based on the target physiological information, segmentally annotate the target performance based on the first target duration to generate attention information.

[0283] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the attention evaluation method for the target performance, the method comprising: obtaining the first brain wave information and the first eye movement information of the audience watching the target performance; extracting the features of the first brain wave information to generate the target brain wave features; extracting the features of the first eye movement information to generate the target eye movement features; fusing the target brain wave features and the target eye movement features to generate the target physiological information; based on the target physiological information, generating the audience's attention information to the target performance, the attention information is used to characterize the change of the audience's concentration on the target performance based on the time change.

[0284] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0285] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the attention evaluation method for a target performance provided by the above-mentioned methods, and the method includes: obtaining first brain wave information and first eye movement information of an audience watching a target performance; performing feature extraction on the first brain wave information to generate a target brain wave feature; performing feature extraction on the first eye movement information to generate a target eye movement feature; fusing the target brain wave feature and the target eye movement feature to generate target physiological information; based on the target physiological information, generating audience attention information on the target performance, the attention information being used to characterize how the audience's concentration on the target performance changes based on time.

[0286] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned methods for evaluating attention to target performances, the methods comprising: obtaining first brain wave information and first eye movement information of an audience watching a target performance; performing feature extraction on the first brain wave information to generate target brain wave features; performing feature extraction on the first eye movement information to generate target eye movement features; fusing the target brain wave features and the target eye movement features to generate target physiological information; based on the target physiological information, generating audience attention information on the target performance, the attention information being used to characterize changes in the audience's concentration on the target performance based on time.

[0287] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0288] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0289] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the attention level of a target performance, characterized in that: include: Obtain the first brain wave information and the first eye movement information of the audience watching the target performance; Extracting features from the first brain wave information to generate target brain wave features; Performing feature extraction on the first eye movement information to generate target eye movement features; fusing the target brain wave characteristics and the target eye movement characteristics to generate target physiological information; Based on the target physiological information, generating the audience's attention information to the target performance, the attention information being used to characterize the change of the audience's concentration on the target performance based on time; The fusing of the target brain wave feature and the target eye movement feature to generate target physiological information includes: Performing dimensionality reduction processing on the target brain wave feature to generate a second brain wave feature; Temporally aligning the target eye movement feature with the second brain wave feature to generate a first eye movement feature; The target physiological information is generated based on the correlation between the second brain wave feature and the first eye movement feature.

2. The method for evaluating the degree of attention of target performance according to claim 1, characterized in that: The step of extracting features from the first brain wave information to generate target brain wave features includes: Preprocessing the first brain wave information to generate second brain wave information; Performing feature extraction on the second brain wave information to generate the target brain wave feature; and / or, The extracting features of the first eye movement information to generate target eye movement features includes: Preprocessing the first eye movement information to generate second eye movement information; Perform feature extraction on the second eye movement information to generate the target eye movement feature.

3. The method for evaluating the degree of attention of target performance according to claim 2, characterized in that: The step of extracting features from the second brain wave information to generate the target brain wave features includes: Performing a time-domain-frequency-domain conversion on the second brain wave information to generate third brain wave information; Based on the target frequency band, feature extraction is performed on the third brain wave information to generate first brain wave features corresponding to the target frequency band; Connect multiple of the first brain wave features to generate the target brain wave feature.

4. The method for evaluating the degree of attention of target performance according to claim 2, characterized in that: The extracting features of the second eye movement information to generate the target eye movement features includes: Based on the second eye movement information, determining the center position of the eyeball; Based on the eyeball center position point, an offset of the eyeball center position point and a time value corresponding to the offset are generated.

5. The method for evaluating the degree of attention of target performance according to any one of claims 1 to 4, characterized in that: The step of generating the audience's attention information to the target performance based on the target physiological information includes: Based on the target physiological information, the target performance is segmented and labeled based on a first target duration to generate the attention information.

6. A target performance-oriented attention evaluation device, characterized in that: include: An acquisition module, used to acquire the first brain wave information and the first eye movement information of the audience watching the target performance; A feature extraction module, configured to extract features from the first brain wave information to generate target brain wave features; and extract features from the first eye movement information to generate target eye movement features; A feature fusion module, used to fuse the target brain wave feature and the target eye movement feature to generate target physiological information; A generating module, configured to generate, based on the target physiological information, information about the audience's attention to the target performance, wherein the information about the audience's attention is used to characterize changes in the audience's concentration on the target performance based on time; The fusing of the target brain wave feature and the target eye movement feature to generate target physiological information includes: Performing dimensionality reduction processing on the target brain wave feature to generate a second brain wave feature; Temporally aligning the target eye movement feature with the second brain wave feature to generate a first eye movement feature; The target physiological information is generated based on the correlation between the second brain wave feature and the first eye movement feature.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for evaluating the degree of attention of a target performance as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the degree of attention of a target performance as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for evaluating the degree of attention of a target performance as described in any one of claims 1 to 5 is implemented.

Images